Add Group Assignment for Co-Assembly workflow

[Minamehr]

FOR CONTRIBUTOR:

• I have read the Adding workflows guidelines
• License permits unrestricted use (educational + commercial)
• Please also take note of the reviewer guidelines below to facilitate a smooth review process.

FOR REVIEWERS:

• .dockstore.yml: file is present and aligned with creator metadata in workflow. ORCID identifiers are strongly encouraged in creator metadata. The .dockstore.yml file is required to run tests
• Workflow is sufficiently generic to be used with lab data and does not hardcode sample names, reference data and can be run without reading an accompanying tutorial.
• [x ] In workflow: annotation field contains short description of what the workflow does. Should start with This workflow does/runs/performs … xyz … to generate/analyze/etc …
• In workflow: workflow inputs and outputs have human readable names (spaces are fine, no underscore, dash only where spelling dictates it), no abbreviation unless it is generally understood. Altering input or output labels requires adjusting these labels in the the workflow-tests.yml file as well
• In workflow: name field should be human readable (spaces are fine, no underscore, dash only where spelling dictates it), no abbreviation unless generally understood
• Workflow folder: prefer dash (-) over underscore (_), prefer all lowercase. Folder becomes repository in iwc-workflows organization and is included in TRS id
• Readme explains what workflow does, what are valid inputs and what outputs users can expect. If a tutorial or other resources exist they can be linked. If a similar workflow exists in IWC readme should explain differences with existing workflow and when one might prefer one workflow over another
• Changelog contains appropriate entries
• Large files (> 100 KB) are uploaded to zenodo and location urls are used in test file

[github-actions]

Test Results (powered by Planemo)

Test Summary

Test State	Count
Total	1
Passed	1
Error	0
Failure	0
Skipped	0

Passed Tests

• ✅ Group-Assignment-for-Co-Assembly.ga_0

  Workflow invocation details

  • Invocation Messages

  • Steps

    • Step 1: metadata:

      • step_state: scheduled

    • Step 2: Trimmed paired reads:

      • step_state: scheduled

    • Step 3: Unzip:

      • step_state: scheduled

      • Jobs

        • Job 1:

          • Job state is ok

          Traceback:

          Job Parameters:

          • Job parameter	Parameter value
            __workflow_invocation_uuid__	"b3da568c93a411f0ab4e7c1e52def123"
            input	{"values": [{"id": 1, "src": "dce"}]}

        • Job 2:

          • Job state is ok

          Traceback:

          Job Parameters:

          • Job parameter	Parameter value
            __workflow_invocation_uuid__	"b3da568c93a411f0ab4e7c1e52def123"
            input	{"values": [{"id": 4, "src": "dce"}]}

        • Job 3:

          • Job state is ok

          Traceback:

          Job Parameters:

          • Job parameter	Parameter value
            __workflow_invocation_uuid__	"b3da568c93a411f0ab4e7c1e52def123"
            input	{"values": [{"id": 7, "src": "dce"}]}

        • Job 4:

          • Job state is ok

          Traceback:

          Job Parameters:

          • Job parameter	Parameter value
            __workflow_invocation_uuid__	"b3da568c93a411f0ab4e7c1e52def123"
            input	{"values": [{"id": 10, "src": "dce"}]}

    • Step 4: Tag reverse:

      • step_state: scheduled

      • Jobs

        • Job 1:

          • Job state is ok

          Traceback:

          Job Parameters:

          • Job parameter	Parameter value
            __workflow_invocation_uuid__	"b3da568c93a411f0ab4e7c1e52def123"
            how	"add"
            input	{"values": [{"id": 3, "src": "hdca"}]}

    • Step 5: Tag forward:

      • step_state: scheduled

      • Jobs

        • Job 1:

          • Job state is ok

          Traceback:

          Job Parameters:

          • Job parameter	Parameter value
            __workflow_invocation_uuid__	"b3da568c93a411f0ab4e7c1e52def123"
            how	"add"
            input	{"values": [{"id": 2, "src": "hdca"}]}

    • Step 6: Zip collection:

      • step_state: scheduled

      • Jobs

        • Job 1:

          • Job state is ok

          Traceback:

          Job Parameters:

          • Job parameter	Parameter value
            __workflow_invocation_uuid__	"b3da568c93a411f0ab4e7c1e52def123"

        • Job 2:

          • Job state is ok

          Traceback:

          Job Parameters:

          • Job parameter	Parameter value
            __workflow_invocation_uuid__	"b3da568c93a411f0ab4e7c1e52def123"

        • Job 3:

          • Job state is ok

          Traceback:

          Job Parameters:

          • Job parameter	Parameter value
            __workflow_invocation_uuid__	"b3da568c93a411f0ab4e7c1e52def123"

        • Job 4:

          • Job state is ok

          Traceback:

          Job Parameters:

          • Job parameter	Parameter value
            __workflow_invocation_uuid__	"b3da568c93a411f0ab4e7c1e52def123"

    • Step 7: Apply rules:

      • step_state: scheduled

      • Jobs

        • Job 1:

          • Job state is ok

          Traceback:

          Job Parameters:

          • Job parameter	Parameter value
            __workflow_invocation_uuid__	"b3da568c93a411f0ab4e7c1e52def123"
            input	{"values": [{"id": 6, "src": "hdca"}]}
            rules	{"mapping": [{"collapsible_value": {"__class__": "RuntimeValue"}, "columns": [2, 0], "connectable": true, "editing": false, "is_workflow": false, "type": "list_identifiers"}, {"collapsible_value": {"__class__": "RuntimeValue"}, "columns": [1], "connectable": true, "is_workflow": false, "type": "paired_identifier"}], "rules": [{"collapsible_value": {"__class__": "RuntimeValue"}, "connectable": true, "error": null, "is_workflow": false, "type": "add_column_metadata", "value": "identifier0", "warn": null}, {"collapsible_value": {"__class__": "RuntimeValue"}, "connectable": true, "error": null, "is_workflow": false, "type": "add_column_metadata", "value": "identifier1", "warn": null}, {"collapsible_value": {"__class__": "RuntimeValue"}, "connectable": true, "error": null, "is_workflow": false, "type": "add_column_metadata", "value": "tags", "warn": null}]}

    • Step 8: Concatenate:

      • step_state: scheduled

      • Jobs

        • Job 1:

          • Job state is ok

          Command Line:

          • printf "Concatenating files:\n" && mkdir concatenated && printf "IBSD09_test- forward and reverse..." && cat /tmp/tmpy9eojsq8/files/b/2/3/dataset_b230bd99-9374-4268-b50a-7ee654e7bdcd.dat >> concatenated/forward.listed.fastqsanger.listed && cat /tmp/tmpy9eojsq8/files/2/2/b/dataset_22b41fbe-e8ae-48c3-81f8-3b541fae5b7d.dat >> concatenated/reverse.listed.fastqsanger.listed && printf "Done\n" && printf "IBSD08_test- forward and reverse..." && cat /tmp/tmpy9eojsq8/files/a/c/8/dataset_ac89daab-5f6a-4972-a38c-5a2473480dcd.dat >> concatenated/forward.listed.fastqsanger.listed && cat /tmp/tmpy9eojsq8/files/6/8/f/dataset_68f13dea-5582-423a-8a4b-4cda8e36da08.dat >> concatenated/reverse.listed.fastqsanger.listed && printf "Done\n" && sleep 1

          Exit Code:

          • 0

          Standard Output:

          • Concatenating files:
            IBSD09_test- forward and reverse...Done
            IBSD08_test- forward and reverse...Done
            

          Traceback:

          Job Parameters:

          • Job parameter	Parameter value
            __input_ext	"fastqsanger"
            __workflow_invocation_uuid__	"b3da568c93a411f0ab4e7c1e52def123"
            chromInfo	"/tmp/tmpy9eojsq8/galaxy-dev/tool-data/shared/ucsc/chrom/?.len"
            dataset_names	false
            dbkey	"?"
            global_condition	{"__current_case__": 1, "input_type": "paired_collection", "inputs": {"values": [{"id": 41, "src": "dce"}]}, "paired_cat_type": "by_strand"}
            headers	"0"

        • Job 2:

          • Job state is ok

          Command Line:

          • printf "Concatenating files:\n" && mkdir concatenated && printf "IBSD02_test- forward and reverse..." && cat /tmp/tmpy9eojsq8/files/d/1/3/dataset_d13147e5-07aa-43f9-a77c-f189ae1a32cc.dat >> concatenated/forward.listed.fastqsanger.listed && cat /tmp/tmpy9eojsq8/files/4/d/5/dataset_4d500d32-a869-4170-9e0a-725b0223cbcf.dat >> concatenated/reverse.listed.fastqsanger.listed && printf "Done\n" && printf "IBSD01_test- forward and reverse..." && cat /tmp/tmpy9eojsq8/files/5/2/8/dataset_52862f04-2968-468c-8117-5493c5fb34ad.dat >> concatenated/forward.listed.fastqsanger.listed && cat /tmp/tmpy9eojsq8/files/6/c/9/dataset_6c9bec4b-6641-4c5c-95cf-ecce386d7512.dat >> concatenated/reverse.listed.fastqsanger.listed && printf "Done\n" && sleep 1

          Exit Code:

          • 0

          Standard Output:

          • Concatenating files:
            IBSD02_test- forward and reverse...Done
            IBSD01_test- forward and reverse...Done
            

          Traceback:

          Job Parameters:

          • Job parameter	Parameter value
            __input_ext	"fastqsanger"
            __workflow_invocation_uuid__	"b3da568c93a411f0ab4e7c1e52def123"
            chromInfo	"/tmp/tmpy9eojsq8/galaxy-dev/tool-data/shared/ucsc/chrom/?.len"
            dataset_names	false
            dbkey	"?"
            global_condition	{"__current_case__": 1, "input_type": "paired_collection", "inputs": {"values": [{"id": 48, "src": "dce"}]}, "paired_cat_type": "by_strand"}
            headers	"0"

  • Other invocation details

    • history_id

      • caa9b8eb39fab594

    • history_state

      • ok

    • invocation_id

      • caa9b8eb39fab594

    • invocation_state

      • scheduled

    • workflow_id

      • caa9b8eb39fab594

[wm75]

I know that this WF is useful for you to group samples for co-assembly as you're explaining in galaxyproject/galaxy-hub#3342.
However, its functionality is way more generic than what you are using it for and very similar to https://github.com/galaxyproject/iwc/tree/main/workflows/data-fetching/sra-manifest-to-concatenated-fastqs so it should probably live next to it though strictly yours is not fetching any data.
For that same reason I suggest that you also generalize the README and use co-assembly just as an example. (Another usecase is the merging of sequencing runs from the same sample just like for @lldelisle's WF.)

In other words, I see this WF more as a flavor of the sra-manifest-to-concatenated-fastqs one, which is useful when you already have the data in your history.

[wm75]

@Minamehr we discovered a few issues with this WF during the fairymags hackathon this week. Specifically,

1. there are issues with toolshed.g2.bx.psu.edu/repos/artbio/concatenate_multiple_datasets/cat_multi_datasets/1.4.3, which you are using at the final step to merge reads from samples belonging to the same group
   This tool is rather outdated, hard to maintain and has better more flexible alternatives.
2. In the current version, if the metadata file has issues (e.g. typos in sample names) things can take hard to predict paths and grouping can go wrong without failing the WF run.

https://usegalaxy.eu/published/workflow?id=763ca07df904b16f swaps out toolshed.g2.bx.psu.edu/repos/artbio/concatenate_multiple_datasets/cat_multi_datasets/1.4.3 against toolshed.g2.bx.psu.edu/repos/nml/collapse_collections/collapse_dataset/5.1.0 (which required some changes to the Apply rules step before and an extra step after) and also adds sample validation. Please play with it and incorporate things into the PR here.