Skip to content

Commit

Permalink
Merge pull request #193 from BU-ISCIII/hotfix
Browse files Browse the repository at this point in the history
New version 1.0.1
  • Loading branch information
saramonzon authored Feb 1, 2024
2 parents 615f139 + d38036e commit 3accca9
Show file tree
Hide file tree
Showing 21 changed files with 768 additions and 117 deletions.
72 changes: 72 additions & 0 deletions .github/CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
# buisciii-tools: Contributing Guidelines

## Contribution workflow

If you'd like to write or modify some code for buisciii-tools, the standard workflow is as follows:

1. Check that there isn't already an issue about your idea in the [buisciii-tools issues](https://github.com/BU-ISCIII/buisciii-tools/issues) to avoid duplicating work. **If there isn't one already, please create one so that others know you're working on this**.
2. [Fork](https://help.github.com/en/github/getting-started-with-github/fork-a-repo) the [buisciii-tools repository](https://github.com/BU-ISCIII/buisciii-tools/) to your GitHub account.
3. Make the necessary changes / additions within your forked repository following the [code style guidelines](#code-style-guidelines).
4. Modify the [`CHANGELOG`](../CHANGELOG.md) file according to your changes in the appropiate section ([X.X.Xhot] or [X.X.Xdev]), you should register your changes regarding:
1. Added enhancements
2. Template changes
3. Fixes
4. Removed stuff
5. Requirements added or version update
5. Update any documentation as needed.
6. [Submit a Pull Request](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request) against the `develop` or `hotfix` branch and send the url to the #pipelines-dev channel in slack (if you are not in the slack channel just wait fot the PR to be reviewed and rebased).

If you're not used to this workflow with git, you can start with:

- Some [docs in the bu-isciii wiki](https://github.com/BU-ISCIII/BU-ISCIII/wiki/Github--gitflow).
- [some slides](https://docs.google.com/presentation/d/1PruqGxPQVxtNcuEbOd86mylXorgYIU5a/edit?pli=1#slide=id.p1) (in spanish).
- some github generic docs [docs from GitHub](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests).
- even their [excellent `git` resources](https://try.github.io/).

### buisciii-tools repo branches

buisciii-tools repo works with a three branching scheme. Two regular branches `main` and `develop`, and a third one created for hot fixes `hotfix`. This last one is created for changes in the **services templates**.

- `main`: stable code only for releases.
- `develop`: new code development for the diferente modules.
- `hotfix`: bug fixing and/or templates addition/modification (bash scripts in: `templates` folder).

You need to submit your PR always against `develop` or `hotfix` depending on the nature of your changes. Once approbed, this changes must be **`rebased`** so we do not create empty unwanted merges.

## Tests

When you create a pull request with changes, [GitHub Actions](https://github.com/features/actions) will run automatic tests.
Typically, pull-requests are only fully reviewed when these tests are passing, though of course we can help out before then.

There are typically two types of tests that run:

### Lint tests

We use black and flake8 linting based on PEP8 guidelines for python coding. You can check more information [here](https://github.com/BU-ISCIII/BU-ISCIII/wiki/Python#linting).

### Code tests

TODO. NOT YET IMPLEMENTED.
Anyhow you should always submit locally tested code!!

### New version bumping and release

In order to create a new release you need to follow the next steps:

1. Set the new version according to [semantic versioning](https://semver.org/), in our particular case, changes in the `hotfix` branch will change the PATCH version (third one), and changes in develop will typicaly change the MINOR version, unless the developing team decides otherwise.
2. Create a PR bumping the new version against `hotfix` or `develop`. For bumping a new version just change [this line](https://github.com/BU-ISCIII/buisciii-tools/blob/615f1390d96cd6c8168acebc384289520a3cd728/setup.py#L5) with the new version.
3. Once that PR is merged, create via web another PR against `main` (origin `develop` or `hotfix` accordingly). This PR would need 2 approvals.
4. [Create a new release](https://docs.github.com/en/repositories/releasing-projects-on-github/managing-releases-in-a-repository) copying the appropiate notes from the `CHANGELOG`.
5. Once the release is approved and merged, you're all set!

PRs from one branch to another, like in a release should be **`merged`** not rebased, so we avoid conflicts and the branch merge is correctly visualize in the commits history.

> A new PR for `develop` branch will be automatically generated if the changes came from `hotfix` so everything is properly sync.
### Code style guidelines

We follow PEP8 conventions as code style guidelines, please check [here](https://github.com/BU-ISCIII/BU-ISCIII/wiki/Python#pep-8-guidelines-read-the-full-pep-8-documentation) for more detail.

## Getting help

For further information/help, please ask on the `#pipelines-dev` slack channel or write us an email! ([[email protected]](emailto:[email protected])).
21 changes: 21 additions & 0 deletions .github/PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
<!--
# bu-isciii tools pull request
Based on nf-core/viralrecon pull request template
Fill in the appropriate checklist below and delete whatever is not relevant.
PRs should be made against the develop of hotfix branch, unless you're preparing a software release.
-->

## PR checklist

- [ ] This comment contains a description of changes (with reason).
- [ ] Make sure your code lints (`black and flake8`).
- If a new tamplate was added make sure:
- [ ] Template's schema is added in `templates/services.json`.
- [ ] Template's pipeline's documentation in `assets/reports/md/template.md` is added.
- [ ] Results Documentation in `assets/reports/results/template.md` is updated.
- [ ] `CHANGELOG.md` is updated.
- [ ] `README.md` is updated (including new tool citations and authors/contributors).
- [ ] If you know a new user was added to the SFTP, make sure you added it to `templates/sftp_user.json`
30 changes: 30 additions & 0 deletions .github/workflows/update_branches.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
name: Create Pull Request from main to develop and/or hotfix

on:
push:
branches:
- main

env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} # setting GH_TOKEN for the entire workflow

jobs:
create-and-auto-merge-pr:
runs-on: ubuntu-latest

steps:
- name: Checkout code
uses: actions/checkout@v2

- name: Install GitHub CLI
run: |
sudo apt update
sudo apt install gh
- name: Create Pull Request to develop
run: |
gh pr create --base develop --head main --title "Merge changes from main into develop" --body "Automatically created pull request to merge changes from main into develop."
- name: Create Pull Request to hotfix
run: |
gh pr create --base hotfix --head main --title "Merge changes from main into hotfix" --body "Automatically created pull request to merge changes from main into hotfix."
126 changes: 126 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
# bu-isciii tools Changelog

All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]

## [2.0.0dev] - 2024-0X-0X : https://github.com/BU-ISCIII/buisciii-tools/releases/tag/2.0.0

### Credits

Code contributions to the release:

- [Sara Monzón](https://github.com/saramonzon)
- [Sarai Varona](https://github.com/svarona)
- [Pablo Mata](https://github.com/Shettland)
- [Guillermo Gorines](https://github.com/GuilleGorines)


### Template fixes and updates

- Added templates:
- freebayes


### Modules

#### Added enhancements

- Added credential parameters: --api_user, --api_password and --cred_file
- Make modules to create folder's paths automatically from DB
- Added finish module
- Added json files: sftp_user.json
- Added delivery jinja templates

#### Fixes

#### Changed

- Fixed API requests to fit in the new database format
- Updated README

#### Removed

### Requirements

- Added PyYAML

## [1.0.2hot] - 2024-0X-0X : https://github.com/BU-ISCIII/buisciii-tools/releases/tag/1.0.2

### Credits

Code contributions to the hotfix:

### Template fixes and updates

### Modules

#### Added enhancements

#### Fixes

#### Changed

#### Removed

### Requirements

## [1.0.1] - 2024-02-01 : https://github.com/BU-ISCIII/buisciii-tools/releases/tag/1.0.1

### Credits

Code contributions to the hotfix:

- [Pablo Mata](https://github.com/Shettland)
- [Jaime Ozaez](https://github.com/jaimeozaez)
- [Sara Monzón](https://github.com/saramonzon)
- [Sarai Varona](https://github.com/svarona)
- [Daniel Valle](https://github.com/Daniel-VM)

### Template fixes and updates

- Added new line in `buisciii_tools/bu_isciii/templates/viralrecon/ANALYSIS/lablog_viralrecon`, in order to automatically rename `ANALYSIS0X_MAG` directory with the current date.
- Introduced handling of flu-C in `buisciii_tools/bu_isciii/templates/IRMA/ANALYSIS/ANALYSIS01_FLU_IRMA/04-irma/` `lablog` and `create_irma_stats.sh`
- Small changes to `buisciii_tools/bu_isciii/templates/viralrecon/RESULTS/viralrecon_results` for blast and new excel_generator.py
- Introduced better error handling in excel_generator.py. Now it can also be used for single files
- Brought back `PASS_ONLY` to exometrio's `exomiser_configfile.yml`
- [#187](https://github.com/BU-ISCIII/buisciii-tools/pull/187) - Added new template for bacterial assembly. Allowing for short, long and hybrid assembly.
- [#190](https://github.com/BU-ISCIII/buisciii-tools/pull/190) - renamed some variables in create-summary_report from viralrecon template as their name was misleading and fixed a small typo in regex finding in excel_generator.py
- [#192](https://github.com/BU-ISCIII/buisciii-tools/pull/192) - Small changes in excel_generator.py to automatically merge pangolin/nextclade tables when more than 1 reference is found

### Modules

#### Added enhancements
- Added CHANGELOG
- Added template for Pull Request
- Added Contributing guidelines
- Added github action to sync branches

#### Fixes

#### Changed

#### Removed


### Requirements


## [1.0.0] - 2024-01-08 : https://github.com/BU-ISCIII/buisciii-tools/releases/tag/1.0.0

### Credits

Code contributions to the inital release:

- [Sara Monzón](https://github.com/saramonzon)
- [Saria Varona](https://github.com/svarona)
- [Guillermo Gorines](https://github.com/GuilleGorines)
- [Pablo Mata](https://github.com/Shettland)
- [Luis Chapado](https://github.com/luissian)
- [Erika Kvalem](https://github.com/ErikaKvalem)
- [Alberto Lema](https://github.com/Alema91)
- [Daniel Valle](https://github.com/Daniel-VM)
- [Fernando Gomez](https://github.com/FGomez-Aldecoa)

2 changes: 1 addition & 1 deletion bu_isciii/__main__.py
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ def run_bu_isciii():
)

# stderr.print("[green] `._,._,'\n", highlight=False)
__version__ = "1.0.0"
__version__ = "1.0.1"
stderr.print(
"[grey39] BU-ISCIII-tools version {}".format(__version__), highlight=False
)
Expand Down
6 changes: 3 additions & 3 deletions bu_isciii/autoclean_sftp.py
Original file line number Diff line number Diff line change
Expand Up @@ -121,9 +121,9 @@ def get_sftp_services(self):
# Get sftp-service last modification
service_finder = LastMofdificationFinder(sftp_service_fullPath)
service_last_modification = service_finder.find_last_modification()
self.sftp_services[
sftp_service_fullPath
] = service_last_modification
self.sftp_services[sftp_service_fullPath] = (
service_last_modification
)
if len(self.sftp_services) == 0:
sys.exit(f"No services found in {self.path}")

Expand Down
34 changes: 33 additions & 1 deletion bu_isciii/templates/IRMA/ANALYSIS/ANALYSIS01_FLU_IRMA/04-irma/create_irma_stats.sh
100644 → 100755
Original file line number Diff line number Diff line change
@@ -1 +1,33 @@
echo -e "sample_ID\tTotalReads\tMappedReads\tFlu_type\tReads_HA\tReads_MP\tReads_NA\tReads_NP\tReads_NS\tReads_PA\tReads_PB1\tReads_PB2" > irma_stats.txt; cat ../samples_id.txt | while read in; do paste <(echo ${in}) <(grep '1-initial' ${in}/tables/READ_COUNTS.txt | cut -f2) <(grep '3-match' ${in}/tables/READ_COUNTS.txt | cut -f2) <(paste <(grep '4-[A-B]_HA' ${in}/tables/READ_COUNTS.txt | cut -f1 | cut -d '_' -f1,3 | cut -d '-' -f2) <(grep '4-[A-B]_NA' ${in}/tables/READ_COUNTS.txt | cut -f1 | cut -d '_' -f3) | tr '\t' '_') <(grep '4-[A-B]_HA' ${in}/tables/READ_COUNTS.txt | cut -f2) <(grep '4-[A-B]_MP' ${in}/tables/READ_COUNTS.txt | cut -f2) <(grep '4-[A-B]_NA' ${in}/tables/READ_COUNTS.txt | cut -f2) <(grep '4-[A-B]_NP' ${in}/tables/READ_COUNTS.txt | cut -f2) <(grep '4-[A-B]_NS' ${in}/tables/READ_COUNTS.txt | cut -f2) <(grep '4-[A-B]_PA' ${in}/tables/READ_COUNTS.txt | cut -f2) <(grep '4-[A-B]_PB1' ${in}/tables/READ_COUNTS.txt | cut -f2) <(grep '4-[A-B]_PB2' ${in}/tables/READ_COUNTS.txt | cut -f2); done >> irma_stats.txt

echo -e "sample_ID\tTotalReads\tMappedReads\tFlu_type\tReads_HA\tReads_MP\tReads_NA\tReads_NP\tReads_NS\tReads_PA\tReads_PB1\tReads_PB2" > irma_stats.txt

cat ../samples_id.txt | while read in
do
SAMPLE_ID=$(echo ${in})
TOTAL_READS=$(grep '1-initial' ${in}/tables/READ_COUNTS.txt | cut -f2)
MAPPEDREADS=$(grep '3-match' ${in}/tables/READ_COUNTS.txt | cut -f2)
FLU_TYPE=$(paste <(grep '4-[A-C]_MP' ${in}/tables/READ_COUNTS.txt | cut -f1 | cut -d '_' -f1 | cut -d '-' -f2) <(grep '4-[A-B]_HA' ${in}/tables/READ_COUNTS.txt | cut -f1 | cut -d '_' -f3 | cut -d '-' -f2) <(grep '4-[A-B]_NA' ${in}/tables/READ_COUNTS.txt | cut -f1 | cut -d '_' -f3) | tr '\t' '_')
HA=$(grep '4-[A-C]_HA' ${in}/tables/READ_COUNTS.txt | cut -f2)
MP=$(grep '4-[A-C]_MP' ${in}/tables/READ_COUNTS.txt | cut -f2)
NA=$(grep '4-[A-C]_NA' ${in}/tables/READ_COUNTS.txt | cut -f2)
NP=$(grep '4-[A-C]_NP' ${in}/tables/READ_COUNTS.txt | cut -f2)
NS=$(grep '4-[A-C]_NS' ${in}/tables/READ_COUNTS.txt | cut -f2)
PA=$(grep '4-[A-C]_PA' ${in}/tables/READ_COUNTS.txt | cut -f2)
PB1=$(grep '4-[A-C]_PB1' ${in}/tables/READ_COUNTS.txt | cut -f2)
PB2=$(grep '4-[A-C]_PB2' ${in}/tables/READ_COUNTS.txt | cut -f2)
#In case of Influenza C in samples:
HE=$(grep '4-C_HE' ${in}/tables/READ_COUNTS.txt | cut -f2)
if [[ -n "$HE" ]]; then
LINE=$(paste <(echo $SAMPLE_ID) <(echo $TOTAL_READS) <(echo $MAPPEDREADS) <(echo $FLU_TYPE) <(echo $HA) <(echo $MP) <(echo $NA) <(echo $NP) <(echo $NS) <(echo $PA) <(echo $PB1) <(echo $PB2) <(echo $HE))
else
LINE=$(paste <(echo $SAMPLE_ID) <(echo $TOTAL_READS) <(echo $MAPPEDREADS) <(echo $FLU_TYPE) <(echo $HA) <(echo $MP) <(echo $NA) <(echo $NP) <(echo $NS) <(echo $PA) <(echo $PB1) <(echo $PB2))
fi

echo "$LINE" >> irma_stats.txt

done

ANY_C=$(grep "C_" irma_stats.txt)
if [[ -n "$ANY_C" ]]; then
sed -i 's/Reads_PB2/Reads_PB2\tReads_HE/g' irma_stats.txt
fi
15 changes: 12 additions & 3 deletions bu_isciii/templates/IRMA/ANALYSIS/ANALYSIS01_FLU_IRMA/04-irma/lablog
100644 → 100755
Original file line number Diff line number Diff line change
Expand Up @@ -15,12 +15,21 @@ echo "cat HA_types.txt | while read in; do mkdir \${in}; done" >> _03_post_proce

echo "mkdir B" >> _03_post_processing.sh

echo "mkdir C" >> _03_post_processing.sh

echo "ls */*.fasta | cut -d '/' -f2 | cut -d '.' -f1 | cut -d '_' -f1,2 | sort -u | grep 'A_' > A_fragment_list.txt" >> _03_post_processing.sh

echo "ls */*.fasta | cut -d '/' -f2 | cut -d '.' -f1 | cut -d '_' -f1,2 | sort -u | grep 'B_' > B_fragment_list.txt" >> _03_post_processing.sh

echo 'cat HA_types.txt | while read type; do grep ${type} irma_stats.txt | cut -f1 | while read sample; do cat A_fragment_list.txt | while read fragment; do if test -f ${sample}/${fragment}*.fasta; then cat ${sample}/${fragment}*.fasta | sed "s/^>/\>${sample}_/g" | sed 's/_H1//g' | sed 's/_H3//g' | sed 's/_N1//g' | sed 's/_N2//g'; fi >> ${type}/${fragment}.txt; done; done; done' >> _03_post_processing.sh
echo "ls */*.fasta | cut -d '/' -f2 | cut -d '.' -f1 | cut -d '_' -f1,2 | sort -u | grep 'C_' > C_fragment_list.txt" >> _03_post_processing.sh

echo 'cat HA_types.txt | while read type; do grep ${type} irma_stats.txt | cut -f1 | while read sample; do cat A_fragment_list.txt | while read fragment; do if test -f ${sample}/${fragment}*.fasta; then cat ${sample}/${fragment}*.fasta | sed "s/^>/\>${sample}_/g" | sed 's/_H1//g' | sed 's/_H3//g' | sed 's/_N1//g' | sed 's/_N2//g' | sed s@-@/@g | sed s/_A_/_/g ; fi >> ${type}/${fragment}.txt; done; done; done' >> _03_post_processing.sh

echo 'grep -w 'B__' irma_stats.txt | cut -f1 | while read sample; do cat B_fragment_list.txt | while read fragment; do if test -f ${sample}/${fragment}*.fasta; then cat ${sample}/${fragment}*.fasta | sed "s/^>/\>${sample}_/g" | sed s/_H1//g | sed s/_H3//g | sed s/_N1//g | sed s/_N2//g | sed s@-@/@g | sed s/_B_/_/g ; fi >> B/${fragment}.txt; done; done' >> _03_post_processing.sh

echo 'grep -w 'C__' irma_stats.txt | cut -f1 | while read sample; do cat C_fragment_list.txt | while read fragment; do if test -f ${sample}/${fragment}*.fasta; then cat ${sample}/${fragment}*.fasta | sed "s/^>/\>${sample}_/g" | sed s/_H1//g | sed s/_H3//g | sed s/_N1//g | sed s/_N2//g | sed s@-@/@g | sed s/_C_/_/g ; fi >> C/${fragment}.txt; done; done' >> _03_post_processing.sh

echo 'grep -w 'B_' irma_stats.txt | cut -f1 | while read sample; do cat B_fragment_list.txt | while read fragment; do if test -f ${sample}/${fragment}*.fasta; then cat ${sample}/${fragment}*.fasta | sed "s/^>/\>${sample}_/g" | sed s/_H1//g | sed s/_H3//g | sed s/_N1//g | sed s/_N2//g; fi >> B/${fragment}.txt; done; done' >> _03_post_processing.sh
echo 'cat ../samples_id.txt | while read in; do cat ${in}/*.fasta | sed "s/^>/\>${in}_/g" | sed 's/_H1//g' | sed 's/_H3//g' | sed 's/_N1//g' | sed 's/_N2//g' | sed 's@-@/@g' | 's/_A_/_/g' | sed 's/_B_/_/g' | sed 's/_C_/_/g' >> all_samples_completo.txt; done' >> _03_post_processing.sh

echo 'cat ../samples_id.txt | while read in; do cat ${in}/*.fasta | sed "s/^>/\>${in}_/g" | sed 's/_H1//g' | sed 's/_H3//g' | sed 's/_N1//g' | sed 's/_N2//g' >> all_samples_completo.txt; done' >> _03_post_processing.sh
echo 'sed -i "s/__//g" irma_stats.txt' >> _03_post_processing.sh
echo 'sed -i "s/_\t/\t/g" irma_stats.txt' >> _03_post_processing.sh
1 change: 1 addition & 0 deletions bu_isciii/templates/IRMA/RESULTS/irma_results
100644 → 100755
Original file line number Diff line number Diff line change
Expand Up @@ -7,3 +7,4 @@ ln -s ../../ANALYSIS/*_MET/99-stats/multiqc_report.html ./krona_results.html
ln -s ../../ANALYSIS/*FLU_IRMA/04-irma/all_samples_completo.txt .
ln -s ../../ANALYSIS/*FLU_IRMA/04-irma/A_H* .
ln -s ../../ANALYSIS/*FLU_IRMA/04-irma/B .
ln -s ../../ANALYSIS/*FLU_IRMA/04-irma/C .
Loading

0 comments on commit 3accca9

Please sign in to comment.