Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Increase memory and use sort in bedtool_genomecov process when dealing with merged replicates #369

Merged
merged 6 commits into from
Jul 3, 2024
Merged

Increase memory and use sort in bedtool_genomecov process when dealing with merged replicates #369

merged 6 commits into from
Jul 3, 2024

Conversation

JoseEspinosa
Copy link
Member

@JoseEspinosa JoseEspinosa commented Jul 3, 2024 •
edited
Loading

As reported in #344, the bedtools sort command in the BEDTOOL_GENOMECOV process is memory-greedy when dealing with merged replicates. Here, the default memory is increased and sorting is performed using the bash sort command which should alleviate also the problem as it takes into account the memory available and uses temporary files, if needed.

PR checklist

  • This comment contains a description of changes (with reason).
  • Make sure your code lints (nf-core lint).
  • Ensure the test suite passes (nextflow run . -profile test,docker --outdir <OUTDIR>).
  • CHANGELOG.md is updated.

@JoseEspinosa JoseEspinosa changed the title Increase memory and use sort in bedtool_genomecov process Increase memory and use sort in bedtool_genomecov process when dealing with merge replicates Jul 3, 2024
@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 3, 2024 •
edited
Loading

nf-core lint overall result: Passed ✅ ⚠️

Posted for pipeline commit e53da35

+| ✅ 220 tests passed       |+
#| ❔   2 tests were ignored |#
!| ❗   5 tests had warnings |!

❗ Test warnings:

  • readme - README contains the placeholder zenodo.XXXXXXX. This should be replaced with the zenodo doi (after the first release).
  • pipeline_todos - TODO string in main.nf: Optionally add in-text citation tools to this list.
  • pipeline_todos - TODO string in main.nf: Optionally add bibliographic entries to this list.
  • pipeline_todos - TODO string in main.nf: Only uncomment below if logic in toolCitationText/toolBibliographyText has been filled!
  • pipeline_todos - TODO string in methods_description_template.yml: #Update the HTML below to your preferred methods description, e.g. add publication citation for this pipeline

❔ Tests ignored:

  • nextflow_config - Config default ignored: params.bamtools_filter_pe_config
  • nextflow_config - Config default ignored: params.bamtools_filter_se_config

✅ Tests passed:

Run details

  • nf-core/tools version 2.14.1
  • Run at 2024-07-03 15:03:05

@JoseEspinosa JoseEspinosa changed the title Increase memory and use sort in bedtool_genomecov process when dealing with merge replicates Increase memory and use sort in bedtool_genomecov process when dealing with merged replicates Jul 3, 2024
@JoseEspinosa
Copy link
Member Author

Maybe we could also implement --skip_merged_replicate_bigwig parameter? So that the subworkflow MERGED_REPLICATE_BAM_TO_BIGWIG can be skipped?

maxulysse
maxulysse reviewed Jul 3, 2024
View reviewed changes
conf/modules.config Outdated
@@ -793,6 +793,7 @@ if (!params.skip_merge_replicates) {
pattern: "*.txt"
]
]
memory = { check_max( 72.GB * task.attempt, 'memory' ) }
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would put that in base.config instead

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cf Sarek: https://github.com/nf-core/sarek/blob/dev/conf/base.config#L57-L103

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, thanks was looking for examples and found this one in differentialabundance, that is why I implemented it that way.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But fair enough, probably this way is more findable

@maxulysse
Copy link
Member

Maybe we could also implement --skip_merged_replicate_bigwig parameter? So that the subworkflow MERGED_REPLICATE_BAM_TO_BIGWIG can be skipped?

that sounds like a good idea too.

@JoseEspinosa JoseEspinosa merged commit 0623e2e into nf-core:dev Jul 3, 2024
11 checks passed
@JoseEspinosa JoseEspinosa mentioned this pull request Jul 3, 2024
Closed
@JoseEspinosa JoseEspinosa mentioned this pull request Jul 30, 2024
Merged
17 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@maxulysse maxulysse maxulysse approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@JoseEspinosa @maxulysse