Pull out the value from a value channel

[multimeric]

DSL2 Workflows can only accept parameters that are channels. However I would like to convert one of these channels into a single Groovy value, since it will be much easier to work with this way. For example, say I have a channel of numbers, and I want to filter that channel down to only values greater than another number. Using only channels this looks like:

nextflow.enable.dsl=2
workflow filter_channel {
    take:
        numbers
        threshold
    main:
        numbers | 
            combine(threshold) |
            filter {
                def (value, cutoff) = it
                value >= cutoff
            } | map {
                it[0]
            } | view
}

workflow {
    filter_channel(
        Channel.from([1, 2, 3, 4]),
        Channel.value(2)
    )
}

If we could pull values out of channels, I could simply write:

workflow filter_channel {
    take:
        numbers
        threshold
    main:
        threshold_val = threshold | to_val()
        numbers | filter { it > threshold_val} | view
}

[bentsherman]

Hi @multimeric , sorry that your question was never answered. I'd like to answer even if you already figured it out since then, because it's an important question that we get from a lot of people. And it's top-rated on the discussions right now 😄

The short answer is that it is not possible to pull a value out of a channel. Once you wrap something in a channel, you can't take it back out. Your "channel only" solution is actually the only solution. The only way to use the values in a channel is with an operator (or a process / subworkflow, which is a "kind" of operator).

Now for a longer answer, and maybe some speculation.

Some folks may remember that in DSL1 you could call .val or .getVal() on a channel, but no longer in DSL2. Even then, I suspect it was supported only by accident and was never really "supposed" to be used. But there is a reason why it was supported in DSL1 but not in DSL2.

In DSL1, pipelines were "eagerly" executed -- Nextflow would parse your script, build the workflow DAG, and run it all in the same step of "running your script". In DSL2, pipelines are "lazily" executed, meaning that "running your script" only builds the workflow DAG, and only after that does Nextflow execute the workflow itself. Lazy execution is the thing that enables modularity -- importing modules from separate files, composing workflows of workflows of workflows, etc. It also makes something like channel.val impossible, because Nextflow doesn't know what the value is when it executes this code. It hasn't even started the pipeline yet!

Having said all that, I agree that your second example is much more convenient. Maybe Nextflow should be able to just figure out the dataflow relationship here. The fact that it can't is the reason why channel logic is so wacky -- we have to describe the explicit dataflow that we want. If Nextflow could understand the implicit dataflow, maybe then we could write workflows with regular programming constructs -- assignments, expressions, if statements, so on. That's an interesting idea for a DSL3 😄

[bentsherman]

I see, the file is also coming from a process. In that case, you basically need a plain splitFastq() function that does not depend on channels, so that you could do something like this:

MAKE_FASTQ
  | flatMap { it ->
    n = countFastq(it)
    return splitFastq(it, n / 50)
  }

You can probably write these functions yourself by taking inspiration from the splitFastq operator, see FastqSplitter

[lishengting]

@bentsherman That makes sense. I'll try it. Thanks a lot!

[bentsherman]

@lishengting I was thinking about this today and remembered that countFastq and splitFastq are already available as regular functions: https://nextflow.io/docs/latest/script.html#splitting-and-counting-records

So the code example I gave you should just work out of the box. Sorry I didn't realize it earlier

[blex-max]

@bentsherman After some extended discussions about very much wanting regular programming constructs in workflows, I was pointed to this thread - may I ask did anything come of this as an "idea for dsl3"?

[bentsherman]

I think we spoke on Slack, but for anyone else who is pointed to this thread -- you can follow this PR for my latest proposal for how to implement this idea