Combining two channels and emitting tuples with repeated elements

[tamasgal]

Hey, I have the following situation: I need to preprocess a few files before the actual analysis can happen and I have two sets of them:

• calibration files
• actual data files

There are just a few calibration files and each data files requires the right one. To pick the correct calibration, one needs to inspect the file.

This is how my workflow looks like for now, to demonstrate the situation, given the following files:

• calibrations: A.detx, B.detx, C.detx
• data files: 1.evt.gz, 2.evt.gz, 3.evt.gz, 4.evt.gz, 5.evt.gz, 6.evt.gz, 7.evt.gz

workflow {
  data = channel.fromPath('data/*.evt.gz')
  calibrations = channel.fromPath('detectors/*.detector')

  preprocess_data(data)
  preprocess_calibrations(calibrations)

  combined = ???  // create channel with combined data/calibration

  // actual_analysis(combined)
}

I need some mechanism/process (combined in the above workflow) which takes the preprocessed data and detectors and then creates a list of tuples out of them, like: [(B.detx, 1.evt.gz), (A.detx, 2.evt.gz), (A.detx, 3.evt.gz), (A.detx, 4.evt.gz), (A.detx, 5.evt.gz), (A.detx, 6.evt.gz), (C.detx, 7.evt.gz)]. In this combined process, I need access to all the data and calibration files to be able to emit a corresponding pair. As written above, to pick the correct calibration, I need to do some quick checks on the data files...

I checked the operators but could not figure out which one could fit. I also checked the patterns in https://github.com/nextflow-io/patterns/tree/master/docs but they seem to be a bit outdated and I anyways need DSL2.

[abhi18av]

Hi @tamasgal ,

From what I can gather with this snippet, I think you should look into the combine operator which would help you create tuples. You could further use collect to encase all of these tuples into a single value.

Probably, the actual files might follow a different naming convention or you might decide to change the input/output signature of a process later. In general, the Transformation or Combination section would be useful for you from the docs here.

[tamasgal]

Thanks Abhinav, I am already playing around with those operators, but I think it's a bit unclear to me if I should do it using channel operations or in a separate process, since I need a bit more complicated logic to find out which calibration is the correct one.

I guess my problem is that I am thinking of a process which first gets the full list of files from a channel (e.g. via concat and then consuming all of them at once) and after that do the calibration/data matching and emit those pairs into a new channel.

I'll report back once I got a more precise question ;)