Skip to content

Commit d2a7029

Browse files
authored
Merge branch 'main' into fixes/liftover-index-creation
2 parents fe3a987 + 30885ec commit d2a7029

17 files changed

Lines changed: 161 additions & 136 deletions

File tree

‎.github/workflows/ci-tests.yml‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -15,16 +15,16 @@ jobs:
1515
steps:
1616
- uses: actions/checkout@v2
1717

18-
- name: Set up JDK 11
18+
- name: Set up JDK 17
1919
uses: actions/setup-java@v2
2020
with:
21-
java-version: '11'
21+
java-version: '17'
2222
distribution: 'adopt'
2323

2424
- name: Setup Nextflow
2525
uses: nf-core/setup-nextflow@v1
2626
with:
27-
version: "latest-edge"
27+
version: "25.10.0"
2828

2929
- name: Install nf-test
3030
run: |

‎.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,3 +7,5 @@ work
77
.nf-test
88
cloudgene.report.json
99
.nf-test.log
10+
imputationserver2.iml
11+
.idea

‎Dockerfile‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ RUN apt-get update && \
77
apt-get -y clean
88

99
# Install miniconda
10-
RUN wget --quiet https://repo.anaconda.com/miniconda/Miniconda3-py39_24.5.0-0-Linux-x86_64.sh -O ~/miniconda.sh && \
10+
RUN wget --quiet https://repo.anaconda.com/miniconda/Miniconda3-py39_24.7.1-0-Linux-x86_64.sh -O ~/miniconda.sh && \
1111
/bin/bash ~/miniconda.sh -b -p /opt/conda
1212
ENV PATH=/opt/conda/bin:${PATH}
1313

@@ -48,7 +48,7 @@ RUN wget https://github.com/lukfor/pgs-calc/releases/download/v${PGS_CALC_VERSIO
4848
ENV PATH="/opt/pgs-calc:${PATH}"
4949

5050
# Install imputationserver-utils
51-
ENV IMPUTATIONSERVER_UTILS_VERSION=v1.5.1
51+
ENV IMPUTATIONSERVER_UTILS_VERSION=v1.5.3
5252
RUN mkdir /opt/imputationserver-utils
5353
WORKDIR "/opt/imputationserver-utils"
5454
#COPY files/imputationserver-utils.tar.gz .

‎README.md‎

Lines changed: 39 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@
77
This repository contains the Imputation Server 2 workflow to facilitate genotype imputation at scale. It serves as the underlying workflow of the [Michigan Imputation Server](https://imputationserver.sph.umich.edu).
88

99
## Citation
10-
> Das S*, Forer L*, Schönherr S*, Sidore C, Locke AE, Kwong A, Vrieze S, Chew EY, Levy S, McGue M, Schlessinger D, Stambolian D, Loh PR, Iacono WG, Swaroop A, Scott LJ, Cucca F, Kronenberg F, Boehnke M, Abecasis GR, Fuchsberger C. Next-generation genotype imputation service and methods. Nature Genetics 48, 1284–1287 (2016).
10+
> Das S*, Forer L*, Schönherr S*, Sidore C, Locke AE, Kwong A, Vrieze S, Chew EY, Levy S, McGue M, Schlessinger D, Stambolian D, Loh PR, Iacono WG, Swaroop A, Scott LJ, Cucca F, Kronenberg F, Boehnke M, Abecasis GR, Fuchsberger C. Next-generation genotype imputation service and methods. Nature Genetics 48, 1284–1287 (2016).
1111
<sub>*Shared first authors</sub>
1212
1313
## License
@@ -23,6 +23,13 @@ If you encounter any problems, feel free to open an issue [here](https://github.
2323

2424
## Version History
2525

26+
[Version 2.0.12](https://github.com/genepi/imputationserver2/releases/tag/v2.0.12) - Improve compression step runtime
27+
28+
[Version 2.0.11](https://github.com/genepi/imputationserver2/releases/tag/v2.0.11) - Add threads parameter to compression
29+
30+
[Version 2.0.7 - 2.0.10](https://github.com/genepi/imputationserver2/releases/tag/v2.0.9) - Fix chrX renaming issue for clouds and enhance error/status messages
31+
32+
2633
[Version 2.0.3 - Version 2.0.6](https://github.com/genepi/imputationserver2/releases/tag/v2.0.6) - Fix QC issues and remove HTSJDK index creation for input validation and QC.
2734

2835
[Version 2.0.2](https://github.com/genepi/imputationserver2/releases/tag/v2.0.2) - Set minimac4 tmp directory (required for larger sample sizes).
@@ -47,7 +54,7 @@ nextflow run main.nf -c conf/test_single_vcf.config
4754
params {
4855
project = "my-test-project"
4956
build = "hg19"
50-
files = "tests/input/three/*.vcf.gz"
57+
files = "tests/data/input/three/*.vcf.gz"
5158
allele_frequency_population = "eur"
5259
mode = "imputation"
5360
refpanel_yaml = "tests/hapmap-2/2.0.0/imputation-hapmap2.yaml"
@@ -70,7 +77,7 @@ nextflow run main.nf -c job.config
7077
| `files` | `null` | List of input files |
7178
| `allele_frequency_population` | `null` | Allele Frequency Population information |
7279
| `refpanel_yaml` | `null` | Reference panel YAML file |
73-
| `mode` | `imputation` | Processing mode (e.g., 'imputation' or `qc-only``) |
80+
| `mode` | `imputation` | Processing mode (e.g., 'imputation' or `qc-only`) |
7481
| `chunksize` | `20000000` | Chunk size for processing |
7582
| `min_samples` | `20` | Minimum number of samples needed |
7683
| `max_samples` | `50000` | Maximum number of samples allowed |
@@ -146,11 +153,11 @@ The `populations` section contains a dictionary mapping population identifiers t
146153

147154
| Identifier | Name |
148155
| ---------- | ---------------------------------------- |
149-
| `id` | The id of the popualtion (e.g. eur) |
156+
| `id` | The id of the population (e.g. eur) |
150157
| `name` | The label of the population. (e.g. EUR) |
151158
| `samples` | Number of samples in the reference panel |
152159

153-
Note: the population id has to be the same as in the legend files.
160+
Note: the population id has to be the same as in the sites files.
154161

155162
#### Quality Filters
156163

@@ -164,38 +171,45 @@ Note: the population id has to be the same as in the legend files.
164171

165172
### Example YAML
166173

167-
Here's an example YAML configuration for a reference panel. This configuration describes a reference panel named "HapMap 2" for an Imputation Server, including details about its version, data sources, and populations represented. The files are stored on AWS S3 and are directly consumed by the pipeline from there.
174+
Here's an example YAML configuration for a reference panel. This configuration describes a reference panel named "HapMap 2" for an Imputation Server, including details about its version, data sources, and represented populations. The files are stored in subdirectories of the application and can be consumed by the pipeline from there.
168175

169176
```yaml
170-
name: HapMap 2
171-
description: HapMap2 Reference Panel for Imputation Server
177+
name: HapMap 2 (GRCh37/hg19)
178+
description: HapMap2 Reference Panel for Michigan Imputation Server
172179
version: 2.0.0
173180
website: http://imputationserver.sph.umich.edu
174181
category: RefPanel
182+
id: hapmap-2
175183

176184
properties:
177-
id: hapmap2
178-
genotypes: s3://cloudgene/refpanels/hapmap/m3vcfs/hapmap_r22.chr$chr.CEU.hg19.recode.m3vcf.gz
179-
legend: s3://cloudgene/refpanels/hapmap/legends/hapmap_r22.chr$chr.CEU.hg19_impute.legend.gz
180-
mapEagle: s3://cloudgene/refpanels/hapmap/map/genetic_map_hg19_withX.txt.gz
181-
refEagle: s3://cloudgene/refpanels/hapmap/bcfs/hapmap_r22.chr$chr.CEU.hg19.recode.bcf
185+
id: hapmap-2
186+
genotypes: ${CLOUDGENE_APP_LOCATION}/msavs/hapmap_r22.chr$chr.CEU.hg19.recode.msav
187+
sites: ${CLOUDGENE_APP_LOCATION}/sites/hapmap_r22.chr$chr.CEU.hg19_impute.sites.gz
188+
mapEagle: ${CLOUDGENE_APP_LOCATION}/map/genetic_map_hg19_withX.txt.gz
189+
refEagle: ${CLOUDGENE_APP_LOCATION}/bcfs/hapmap_r22.chr$chr.CEU.hg19.recode.bcf
182190
build: hg19
191+
qcFilter:
192+
alleleSwitches: 100
183193
populations:
184194
- id: eur
185195
name: EUR
186196
samples: 60
187-
- id: off
197+
- id: "off"
188198
name: Off
189-
samples: -1
199+
samples: -1
190200
```
191201
202+
A full example of a reference panel, including all data and the cloudgene.yaml, can be downloaded [here](https://imputationserver.sph.umich.edu/resources/ref-panels/imputationserver2-hapmap2.zip).
203+
192204
### Note on `$chr` Variable
193205

194206
In the example YAML configuration provided, you may have noticed the presence of the `$chr` variable in some URLs. This variable is a placeholder for the chromosome number and will be replaced by the Nextflow pipeline.
195207

196-
### Legend Files
208+
### Sites Files
209+
210+
A site file is a tab-delimited file consisting of 8 columns: **ID**, **CHROM**, **POS**, **REF**, **ALT**, **AAF_EUR**, **AAF_ALL**, **MAF_EUR**, and **MAF_ALL**. The first five columns (**ID**, **CHROM**, **POS**, **REF**, and **ALT**) are required, while the **Allele Frequency (AAF)** and **Minor Allele Frequency (MAF)** columns are optional.
197211

198-
A legend file is a tab-delimited file consisting of 5 columns (`id`, `position`, `a0`, `a1`, `all.aaf`).
212+
The optional **AAF** and **MAF** columns provide allele frequency information for different populations supported by the reference panel. Specifically, **AAF_EUR** and **MAF_EUR** represent allele frequencies for the European population, while **AAF_ALL** and **MAF_ALL** represent allele frequencies for all populations combined.
199213

200214
---
201215

@@ -209,14 +223,13 @@ A legend file is a tab-delimited file consisting of 5 columns (`id`, `position`,
209223

210224
### Installation
211225

212-
- Install cloudgene3: `curl -s install.cloudgene.io`
213-
- Download latest source code zip file from releases
214-
- Install impuationserver2 app: `./cloudgene install imputationserver2@latest`
215-
- Install hapmap2 referenece panel: `./cloudgene install https://genepi.i-med.ac.at/downloads/imputation/imputation-hapmap2.zip`
226+
- Install cloudgene3: `curl -fsSL https://get.cloudgene.io | bash`
227+
- Install imputationserver2 app: `./cloudgene install genepi/imputationserver2@latest`
228+
- Install hapmap2 reference panel: `./cloudgene install https://imputationserver.sph.umich.edu/resources/ref-panels/imputationserver2-hapmap2.zip`
216229
- Start cloudgene server: `./cloudgene server`
217230
- Open [http://localhost:8082](http://localhost:8082)
218231
- Login with default admin account: username `admin` and password `admin1978`
219-
- Imputation can be tested with the following [test file](https://github.com/genepi/imputationserver2/raw/main/tests/input/chr20-phased/chr20.R50.merged.1.330k.recode.small.vcf.gz)
232+
- Imputation can be tested with the following [test file](https://github.com/genepi/imputationserver2/raw/main/tests/data/input/chr20-phased/chr20.R50.merged.1.330k.recode.small.vcf.gz)
220233

221234
### Default Configuration
222235

@@ -263,7 +276,7 @@ process {
263276
}
264277
```
265278

266-
3. Got to Settings -> General and set Workspace to "S3" and enter the location of a subfolder in an S3 bucket. Enter the location of a subfolder in an S3 bucket. Currently, it must be a subfolder; a bucket won't work (Example: `s3://cloudgene/workspace`).
279+
3. Go to Settings -> General and set Workspace to "S3" and enter the location of a subfolder in an S3 bucket. Enter the location of a subfolder in an S3 bucket. Currently, it must be a subfolder; a bucket won't work (Example: `s3://cloudgene/workspace`).
267280

268281
Optional add [Wave](https://www.nextflow.io/docs/latest/wave.html) and [Fusion](https://www.nextflow.io/docs/latest/fusion.html) support to improve performance:
269282

@@ -281,7 +294,7 @@ fusion {
281294
### Activate mail support
282295

283296
- Configure mail server in Settings -> General -> Mail
284-
- Configure Nextflow to use Cloudgenes mail settings by add the following to the global configuration (Settings -> General -> Nextflow) or adapt/create files `config/nextflow.confing` (see [Nextflow Documention](https://www.nextflow.io/docs/latest/config.html#config-mail) for all available mail settings)
297+
- Configure Nextflow to use Cloudgenes mail settings by add the following to the global configuration (Settings -> General -> Nextflow) or adapt/create files `config/nextflow.config` (see [Nextflow Documentation](https://www.nextflow.io/docs/latest/config.html#config-mail) for all available mail settings)
285298

286299
```
287300
mail {
@@ -299,10 +312,10 @@ mail {
299312

300313
### Adapt default parameters
301314

302-
Parameters can be changed in the `nextflow.config`` file of the application. Example:
315+
Parameters can be changed in the `nextflow.config` file of the application. Example:
303316

304317
```
305-
params.chunk_size = 500000
318+
params.chunksize = 500000
306319
params.imputation.window = 100000
307320
```
308321

‎cloudgene.hla.yaml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ name: HLA Imputation
33
description: "<p> Thank you for using our multi-ethnic HLA imputation panel, built from ~20,000 whole genome sequencing samples from five global populations. </p> <p>
44
Please cite this manuscript if you would like to include imputed results from the panel in your work: </p> <p>
55
Luo, Y., Kanai, M., Choi, W., Li, X., Yamamoto, K., Ogawa, K., Gutierrez-Arcelus, M., Gregersen, P. K., Stuart, P. E., Elder, J. T., Fellay, J., Carrington, M., Haas, D. W., Guo, X., Palmer, N. D., Chen, Y.-D. I., Rotter, J. I., Taylor, K. D., Rich, S., … Raychaudhuri, S. (2020). <b> A high-resolution HLA reference panel capturing global population diversity enables multi-ethnic fine-mapping in HIV host response</b>. https://doi.org/10.1101/2020.07.16.20155606 <br><br>If your input data is <b>GRCh37/hg19</b> please ensure chromosomes are encoded without prefix (e.g. <b>20</b>).<br>If your input data is <b>GRCh38hg38</b> please ensure chromosomes are encoded with prefix 'chr' (e.g. <b>chr20</b>). </p>"
6-
version: 2.0.6
6+
version: 2.0.12
77
website: https://imputationserver.readthedocs.io
88
category: Application
99
submitButton: Start Imputation
@@ -16,6 +16,7 @@ workflow:
1616
# params without UI
1717
params:
1818
project: "${CLOUDGENE_JOB_NAME}"
19+
project_id: "${CLOUDGENE_JOB_ID}"
1920
files: "${files}/*.vcf.gz"
2021
chunksize: 1000000000
2122
minSamples: 0

‎cloudgene.pgs.yaml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
id: imputationserver2-pgs
22
name: Polygenic Score Calculation
33
description: "You can upload genotyped data and the application imputes your genotypes, performs ancestry estimation and finally calculates Polygenic Risk Scores.<br><br>No dataset at hand? No problem, download our example dataset: <a href=\"https://imputationserver.sph.umich.edu/resources/50-samples.zip\" class=\"btn btn-sm btn-secondary\" style=\"color:#ffffff !important\"><i class=\"fa fa-file\"></i> 50-samples.zip</a><br><br>"
4-
version: 2.0.6
4+
version: 2.0.12
55
website: https://imputationserver.readthedocs.io
66
category: Application
77
submitButton: Start Calculation
@@ -14,6 +14,7 @@ workflow:
1414
# params without UI
1515
params:
1616
project: "${CLOUDGENE_JOB_NAME}"
17+
project_id: "${CLOUDGENE_JOB_ID}"
1718
files: "${files}/*.vcf.gz"
1819
mode: "imputation"
1920
allele_frequency_population: "off"

‎cloudgene.yaml‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
id: imputationserver2
22
name: Genotype Imputation
33
description: This is the new Michigan Imputation Server Pipeline using <a href="https://github.com/statgen/Minimac4">Minimac4</a>. Documentation can be found <a href="http://imputationserver.readthedocs.io/en/latest/">here</a>.<br><br>If your input data is <b>GRCh37/hg19</b> please ensure chromosomes are encoded without prefix (e.g. <b>20</b>).<br>If your input data is <b>GRCh38hg38</b> please ensure chromosomes are encoded with prefix 'chr' (e.g. <b>chr20</b>).
4-
version: 2.0.6
4+
version: 2.0.12
55
website: https://imputationserver.readthedocs.io
66
category: Application
77
submitButton: Start Imputation
@@ -14,6 +14,7 @@ workflow:
1414
# params without UI
1515
params:
1616
project: "${CLOUDGENE_JOB_NAME}"
17+
project_id: "${CLOUDGENE_JOB_ID}"
1718
allele_frequency_population: "${population}"
1819
files: "${files}/*.vcf.gz"
1920
encryption:
@@ -107,7 +108,7 @@ workflow:
107108
type: list
108109
value: eagle
109110
values:
110-
eagle: Eagle v2.4 (phased output)
111+
eagle: Eagle v2.4
111112
beagle: Beagle 5.4 (HRC and 1KG only)
112113
no_phasing: No phasing
113114
serialize: false

‎docs/index.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@ nextflow run main.nf -c conf/test_single_vcf.config
2727
params {
2828
project = "my-test-project"
2929
build = "hg19"
30-
files = "tests/input/three/*.vcf.gz"
30+
files = "tests/data/input/three/*.vcf.gz"
3131
allele_frequency_population = "eur"
3232
mode = "imputation"
3333
refpanel_yaml = "tests/hapmap-2/2.0.0/imputation-hapmap2.yaml"
@@ -196,7 +196,7 @@ A legend file is a tab-delimited file consisting of 5 columns (`id`, `position`,
196196
- Start cloudgene server: `./cloudgene server`
197197
- Open [http://localhost:8082](http://localhost:8082)
198198
- Login with default admin account: username `admin` and password `admin1978`
199-
- Imputation can be tested with the following [test file](https://github.com/genepi/imputationserver2/raw/main/tests/input/chr20-phased/chr20.R50.merged.1.330k.recode.small.vcf.gz)
199+
- Imputation can be tested with the following [test file](https://github.com/genepi/imputationserver2/raw/main/tests/data/input/chr20-phased/chr20.R50.merged.1.330k.recode.small.vcf.gz)
200200

201201
### Default Configuration
202202

‎lib/ArrayUtil.groovy‎

Lines changed: 0 additions & 9 deletions
This file was deleted.

‎main.nf‎

Lines changed: 17 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -147,14 +147,21 @@ workflow.onComplete {
147147
//TODO: use templates
148148
//TODO: move in EmailHelper class
149149
if (!workflow.success) {
150-
def statusMessage = workflow.exitStatus != null || workflow.errorReport == "QC step failed" ? "failed" : "canceled"
151-
if (params.send_mail && params.user.email != null){
152-
sendMail{
153-
to "${params.user.email}"
154-
subject "[${params.service.name}] Job ${params.project} ${statusMessage}"
155-
body "Dear ${params.user.name}, \n Your job has been ${statusMessage}.\n\n More details can be found at the following link: ${params.service.url}/index.html#!jobs/${params.project}"
150+
def statusMessage = (workflow.exitStatus != null || workflow.errorReport == "QC step failed")
151+
? "failed"
152+
: "canceled"
153+
154+
def statusText = (statusMessage == "failed")
155+
? "Your job failed."
156+
: "Your job has been canceled."
157+
158+
if (params.send_mail && params.user.email != null) {
159+
sendMail{
160+
to "${params.user.email}"
161+
subject "[${params.service.name}] Job ${params.project} ${statusMessage}"
162+
body "Dear ${params.user.name},\n${statusText}\n\nMore details can be found at the following link: ${params.service.url}/index.html#!jobs/${params.project_id}"
163+
}
156164
}
157-
}
158165
println "::error:: Imputation job ${statusMessage}."
159166
return
160167
}
@@ -178,7 +185,7 @@ workflow.onComplete {
178185
sendMail{
179186
to "${params.user.email}"
180187
subject "[${params.service.name}] Job ${params.project} is complete"
181-
body "Dear ${params.user.name}, \n Your imputation job has finished succesfully. The password for the imputation results is: ${params.encryption_password}\n\n You can download the results from the following link: ${params.service.url}/index.html#!jobs/${params.project}"
188+
body "Dear ${params.user.name},\nYour imputation job has finished succesfully. The password for the imputation results is: ${params.encryption_password}\n\nYou can download the results from the following link: ${params.service.url}/index.html#!jobs/${params.project_id}"
182189
}
183190
println "::message:: Data have been exported successfully. We have sent a notification email to <b>${params.user.email}</b>"
184191
} else {
@@ -192,11 +199,11 @@ workflow.onComplete {
192199
sendMail{
193200
to "${params.user.email}"
194201
subject "[${params.service.name}] Job ${params.project} is complete"
195-
body "Dear ${params.user.name}, \n Your PGS job has finished successfully. \n\n You can download the results from the following link: ${params.service.url}/index.html#!jobs/${params.project}"
202+
body "Dear ${params.user.name},\nYour PGS job has finished successfully.\n\nYou can download the results from the following link: ${params.service.url}/index.html#!jobs/${params.project_id}"
196203
}
197204
println "::message:: Data have been exported successfully. We have sent a notification email to <b>${params.user.email}</b>"
198205
} else {
199206
println "::message:: Data have been exported successfully."
200207
}
201208

202-
}
209+
}

0 commit comments

Comments
 (0)