Add XY filtration workflow #191

Faizal-Eeman · 2024-12-04T00:48:42Z

Description

ADD XY filter workflow

Closes #190

Testing Results

N-T paired WGS (sample_sex = XY)
- samples: NA24149, NA24143
- input YAML: /hot/software/pipeline/pipeline-call-gSNP/Nextflow/development/pre-10.1.0-rc.1/mmootor-add-xy-filtration/pipeline_test/NA24143.yaml
- config: /hot/software/pipeline/pipeline-call-gSNP/Nextflow/development/pre-10.1.0-rc.1/mmootor-add-xy-filtration/pipeline_test/filter-xy-gSNP.config
- output: /hot/software/pipeline/pipeline-call-gSNP/Nextflow/development/pre-10.1.0-rc.1/mmootor-add-xy-filtration/pipeline_test/call-gSNP-10.1.0-rc.1/NA24143/GATK-4.5.0.0/output/
N-T paired WGS (sample_sex = XX)
- samples: NA24149, NA24143
- input YAML: /hot/software/pipeline/pipeline-call-gSNP/Nextflow/development/pre-10.1.0-rc.1/mmootor-add-xy-filtration/pipeline_test/NA24143.yaml
- config: /hot/software/pipeline/pipeline-call-gSNP/Nextflow/development/pre-10.1.0-rc.1/mmootor-add-xy-filtration/pipeline_test/test-XX/filter-xx-gSNP.config
- output: /hot/software/pipeline/pipeline-call-gSNP/Nextflow/development/pre-10.1.0-rc.1/mmootor-add-xy-filtration/pipeline_test/test-XX/call-gSNP-10.1.0-rc.1/NA24143/GATK-4.5.0.0/output/

Checklist

I have read the code review guidelines and the code review best practice on GitHub check-list.
I have reviewed the Nextflow pipeline standards.
The name of the branch is meaningful and well formatted following the standards, using [AD_username (or 5 letters of AD if AD is too long)]-[brief_description_of_branch].
I have set up or verified the branch protection rule following the github standards before opening this pull request.
I have added my name to the contributors listings in the manifest block in the nextflow.config as part of this pull request, am listed
already, or do not wish to be listed. (This acknowledgement is optional.)
I have added the changes included in this pull request to the CHANGELOG.md under the next release version or unreleased, and updated the date.
I have updated the version number in the metadata.yaml and manifest block of the nextflow.config file following semver, or the version number has already been updated. (Leave it unchecked if you are unsure about new version number and discuss it with the infrastructure team in this PR.)
I have tested the pipeline on at least one A-mini sample.

…122-GIABGermlineVariant

alkaZeltser · 2025-01-08T19:17:11Z

@Faizal-Eeman Would it be possible to add a line to the outputed VCF header documenting the XY filtration, similar to how bcftools appends every operation to the header? It would be a good way to maintain a record of what has been done to the file.

alkaZeltser · 2025-01-08T19:37:55Z

Test run complete.

XY filtered call-gSNP output - /hot/software/pipeline/pipeline-call-gSNP/Nextflow/development/pre-10.1.0-rc.1/mmootor-add-xy-filtration/pipeline_test/call-gSNP-10.1.0-rc.1/NA24143/GATK-4.5.0.0/output/Hail-branch-mmootor-fix-spark-permission_GATK-4.5.0.0_TEST_NA24143_Hail-branch-mmootor-fix-spark-permission-GATK-4.5.0.0-TEST-XY-filtered.vcf.bgz

pipeline log - /hot/software/pipeline/pipeline-call-gSNP/Nextflow/development/pre-10.1.0-rc.1/mmootor-add-xy-filtration/pipeline_test/xy-filter.log

I skimmed through the output file and everything looks as expected: PARs remain diploid in both X and Y, non-PARs are haploid in X and Y. Missing genotypes are always in diploid notation: ./. but I don't think that's an issue. @Faizal-Eeman for completeness' sake, would be good to test a female sample? Unless one of the two is already female?

Faizal-Eeman · 2025-01-09T21:17:01Z

@Faizal-Eeman Would it be possible to add a line to the outputed VCF header documenting the XY filtration, similar to how bcftools appends every operation to the header? It would be a good way to maintain a record of what has been done to the file.

@alkaZeltser for now I'm appending the script command like GATK does to VCF header

##source=ApplyVQSR
##source=CombineGVCFs
##XYFiltration=<CommandLine=script/filter_xy_call.py --sample_name Hail-0.2.133_GATK-4.5.0.0_TEST_NA24143 --input_vcf GATK-4.5.0.0_TEST_NA24143_VQSR-SNP-AND-INDEL.vcf.gz --vcf_source_file ./vcf_source.txt --sample_sex XY --par_bed pseudoautosomal_regions_hg38.bed --genome_build GRCh38 --output_dir .>

As to the steps in the workflow, I've added a document to ./docs in the repo and referenced it in call-gSNP README.

Faizal-Eeman · 2025-01-10T01:24:53Z

for completeness' sake, would be good to test a female sample? Unless one of the two is already female?

The same sample was treated as an XX case,

output VCF - /hot/software/pipeline/pipeline-call-gSNP/Nextflow/development/pre-10.1.0-rc.1/mmootor-add-xy-filtration/pipeline_test/test-XX/call-gSNP-10.1.0-rc.1/NA24143/GATK-4.5.0.0/output/Hail-0.2.133_GATK-4.5.0.0_TEST_NA24143_XY_filtered.vcf.bgz

By the time this test finished, I updated the python script to name the output file based on the sample_sex. Latest python script test output is below,
/hot/software/pipeline/pipeline-call-gSNP/Nextflow/development/pre-10.1.0-rc.1/mmootor-add-xy-filtration/script_test/multisample_XX_filtered.vcf.bgz

yashpatel6

Looks good! Anything else to add from @alkaZeltser ?

tyamaguchi-ucla · 2025-01-15T23:42:10Z

README.md

@@ -115,6 +118,8 @@ For normal-only or tumor-only samples, exclude the fields for the other state.
 |:----------------|:---------|:-----|:------------|
 | `dataset_id` | Yes | string | Dataset ID |
 | `blcds_registered_dataset` | Yes | boolean | Set to true when using BLCDS folder structure; use false for now |
+| `genome_build` | Yes | string | Genome build, GRCh37 or GRCh38 |
+| `sample_sex` | Yes | string | Sample Sex, XY or XX |


question (non-blocking):

@Faizal-Eeman @yashpatel6 We might've touched on this before but have we tried to adjust ploidy for male X Y chromosomes when running HC even before this filtering?

pipeline-call-gSNP/module/haplotypecaller.nf

Line 75 in 2381f97

--sample-ploidy 2 \

tyamaguchi-ucla · 2025-01-16T01:19:54Z

README.md

+| `<Hail>_<GATK>_<dataset_id>_<patient_id>_<sample_sex>_filtered.vcf.bgz` | chrX/Y filtered SNP and INDEL recalibrated variants |
+| `<Hail>_<GATK>_<dataset_id>_<patient_id>_<sample_sex>_filtered.vcf.bgz.sha512` | chrX/Y filtered SNP and INDEL recalibrated variants checksum |
+| `<Hail>_<GATK>_<dataset_id>_<patient_id>_<sample_sex>_filtered.vcf.bgz.tbi` | chrX/Y filtered SNP and INDEL recalibrated variants index |
+| `<Hail>_<GATK>_<dataset_id>_<patient_id>_<sample_sex>_filtered.vcf.bgz.tbi.sha512` | chrX/Y filtered SNP and INDEL recalibrated variants index checksum |


suggestion:

I recommend removing _filtered from the name. Many of our VCF outputs undergo some form of filtering, so the term might not add significant value here. Have we decided whether these outputs will be the final outputs or supplementary final outputs?

tyamaguchi-ucla · 2025-01-16T01:21:38Z

README.md

+### 9. Adjust chrX and chrY genotypes based on sample sex from recalibrated VCF
+Apply XY filtration workflow to recalibrated VCF as discribed [here](docs/xy_filtration_workflow.md).


question:

How does this process work with mouse samples (or other species)? Is this process optional?

alkaZeltser · 2025-01-17T02:33:00Z

The same sample was treated as an XX case,
output VCF - /hot/software/pipeline/pipeline-call-gSNP/Nextflow/development/pre-10.1.0-rc.1/mmootor-add-xy-filtration/pipeline_test/test-XX/call-gSNP-10.1.0-rc.1/NA24143/GATK-4.5.0.0/output/Hail-0.2.133_GATK-4.5.0.0_TEST_NA24143_XY_filtered.vcf.bgz
By the time this test finished, I updated the python script to name the output file based on the sample_sex. Latest python script test output is below,
/hot/software/pipeline/pipeline-call-gSNP/Nextflow/development/pre-10.1.0-rc.1/mmootor-add-xy-filtration/script_test/multisample_XX_filtered.vcf.bgz

@Faizal-Eeman hmm I am still confused. I looked at bcftools view -r chrY --no-header multisample_XX_filtered.vcf.bgz and saw many chrY calls, including heterozygous genotypes. If the sample is treated as XX, there should be no chrY calls at all right?

Faizal-Eeman · 2025-01-17T20:04:09Z

The same sample was treated as an XX case,
output VCF - /hot/software/pipeline/pipeline-call-gSNP/Nextflow/development/pre-10.1.0-rc.1/mmootor-add-xy-filtration/pipeline_test/test-XX/call-gSNP-10.1.0-rc.1/NA24143/GATK-4.5.0.0/output/Hail-0.2.133_GATK-4.5.0.0_TEST_NA24143_XY_filtered.vcf.bgz
By the time this test finished, I updated the python script to name the output file based on the sample_sex. Latest python script test output is below,
/hot/software/pipeline/pipeline-call-gSNP/Nextflow/development/pre-10.1.0-rc.1/mmootor-add-xy-filtration/script_test/multisample_XX_filtered.vcf.bgz

@Faizal-Eeman hmm I am still confused. I looked at bcftools view -r chrY --no-header multisample_XX_filtered.vcf.bgz and saw many chrY calls, including heterozygous genotypes. If the sample is treated as XX, there should be no chrY calls at all right?

@alkaZeltser Just compared the chrY calls in the XX output and they are all PAR variants. Based on the XY filtration workflow, we only remove non-PAR chrY calls.

alkaZeltser · 2025-01-17T20:12:15Z

The same sample was treated as an XX case,
output VCF - /hot/software/pipeline/pipeline-call-gSNP/Nextflow/development/pre-10.1.0-rc.1/mmootor-add-xy-filtration/pipeline_test/test-XX/call-gSNP-10.1.0-rc.1/NA24143/GATK-4.5.0.0/output/Hail-0.2.133_GATK-4.5.0.0_TEST_NA24143_XY_filtered.vcf.bgz
By the time this test finished, I updated the python script to name the output file based on the sample_sex. Latest python script test output is below,
/hot/software/pipeline/pipeline-call-gSNP/Nextflow/development/pre-10.1.0-rc.1/mmootor-add-xy-filtration/script_test/multisample_XX_filtered.vcf.bgz

@Faizal-Eeman hmm I am still confused. I looked at bcftools view -r chrY --no-header multisample_XX_filtered.vcf.bgz and saw many chrY calls, including heterozygous genotypes. If the sample is treated as XX, there should be no chrY calls at all right?

@alkaZeltser Just compared the chrY calls in the XX output and they are all PAR variants. Based on the XY filtration workflow, we only remove non-PAR chrY calls.

@Faizal-Eeman Ok but if the individual is truly XX, that means any reads that map to chrY regions, even PAR, are incorrectly mapped right? They're being drawn away from chrX? And how do you interpret a PAR genotype in an XX individual that has chrY calls? Doesn't that technically result in a triploid XXY dosage?

Faizal-Eeman added 30 commits November 25, 2024 16:27

add patient_sex to template

73e1e03

add XY filter script from project-method-AlgorithmEvaluation-BNCH-000…

7cd06a6

…122-GIABGermlineVariant

add par_bed parameter to template.config

cc5ba70

add genome build to template

13fb844

add user input sample id

43fd101

add sample sex; remove redundant code; exremove het calls for XY

cfb4198

fix variables

6d0ed9e

extract autosomes

7abfcf7

merge autosomes and XY filtered calls

9417465

change vcf header extraction location

f8417d5

Add workflow steps to script note

055ab3e

clean up script

6c31337

add skeleton code for vcf header temp file

08213c9

write VCF source to temp file and parameterize it

4be2064

add arg for variant caller

ac2d36c

set variant caller source

4ff9575

improve documentation

7bdd8cc

add hail v0.2.133 and docker image to default config

1eadfbb

add filter-xy NF script

24cb06d

add NF skeleton

27b39da

add script dir var in main

73c7f1e

add xy filtration command

570677b

rename XY script

d247577

change arg variant_caller to vcf_source_file

71c294b

add VCF source extraction code to script section in nextflow module

6cb1d9c

revert script output to bgz

8b8948d

set publishDir

6e22838

add channel for xy filter and call process

f41aab6

include filter XY module in main

28698a2

simplify xy filter channel

e572c84

Faizal-Eeman added 2 commits January 8, 2025 11:29

add default and choices for genome_build

96768ff

add choices for sample_sex in schema

b2cf9e8

Faizal-Eeman added 10 commits January 8, 2025 11:47

add resource allocation to filter_XY_Hail

1e51960

emit xy filtered output

c51d3c3

generate checksum for xy filtered vqsr VCF

a69f46c

add system command to VCF header

bd9e781

Add XY filtration step to README

002ffc3

fix process name

f783530

add XY filteration params to README

f96c632

Update script description

9b6e036

add xy_filtration_workflow.md

6e571bc

add GRCh38 PAR to README

d9a1eae

Faizal-Eeman added 4 commits January 9, 2025 14:08

fix pylint

bd568b6

fix publishDir rules

8f8326d

Update outputs in README

9b06490

Upddate CHANGELOG

ca0e4f4

update output filename

e8d1d5d

Faizal-Eeman requested review from yashpatel6 and alkaZeltser January 10, 2025 01:26

yashpatel6 approved these changes Jan 15, 2025

View reviewed changes

tyamaguchi-ucla reviewed Jan 15, 2025

View reviewed changes

tyamaguchi-ucla self-assigned this Jan 16, 2025

tyamaguchi-ucla reviewed Jan 16, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add XY filtration workflow #191

Add XY filtration workflow #191

Faizal-Eeman commented Dec 4, 2024 •

edited

Loading

alkaZeltser commented Jan 8, 2025

alkaZeltser commented Jan 8, 2025 •

edited

Loading

Faizal-Eeman commented Jan 9, 2025

Faizal-Eeman commented Jan 10, 2025

yashpatel6 left a comment

tyamaguchi-ucla Jan 15, 2025

tyamaguchi-ucla Jan 16, 2025

tyamaguchi-ucla Jan 16, 2025 •

edited

Loading

alkaZeltser commented Jan 17, 2025

Faizal-Eeman commented Jan 17, 2025

alkaZeltser commented Jan 17, 2025

		### 9. Adjust chrX and chrY genotypes based on sample sex from recalibrated VCF
		Apply XY filtration workflow to recalibrated VCF as discribed [here](docs/xy_filtration_workflow.md).

Add XY filtration workflow #191

Are you sure you want to change the base?

Add XY filtration workflow #191

Conversation

Faizal-Eeman commented Dec 4, 2024 • edited Loading

Description

Closes #190

Testing Results

Checklist

alkaZeltser commented Jan 8, 2025

alkaZeltser commented Jan 8, 2025 • edited Loading

Faizal-Eeman commented Jan 9, 2025

Faizal-Eeman commented Jan 10, 2025

yashpatel6 left a comment

Choose a reason for hiding this comment

tyamaguchi-ucla Jan 15, 2025

Choose a reason for hiding this comment

tyamaguchi-ucla Jan 16, 2025

Choose a reason for hiding this comment

tyamaguchi-ucla Jan 16, 2025 • edited Loading

Choose a reason for hiding this comment

alkaZeltser commented Jan 17, 2025

Faizal-Eeman commented Jan 17, 2025

alkaZeltser commented Jan 17, 2025

Faizal-Eeman commented Dec 4, 2024 •

edited

Loading

alkaZeltser commented Jan 8, 2025 •

edited

Loading

tyamaguchi-ucla Jan 16, 2025 •

edited

Loading