[FEATURE] Have non blocking channel out of bwamem

[maxulysse]

Copied the discussion from #358 for context.

Basically what would be good is to have non-blocking channel our of the mapping step, so that following steps can happen while mapping is still ongoing.

from @rogerzhanglijie

how to resolve the groupTuple block problems in line 78, because I can not comment on that line, so I comment here,
if the mapping step has multiple runs, groupTuple will block all bam files, until all samples have the aligned results, however, we can not set size parameter in groupTuple, because we don't know how many runs in one sample, there may be 2,3,4,.... and so on.

I don't see a problem here actually.
the reads_input.groupTuple(by: [0,1]) ensure that we have only one fastq pairs in the channel emission.
We group the different bams from the same samples with this:

MERGE_BAM(bam_bwa_multiple) 
 bam_mapped       = bam_bwa_single.mix(MERGE_BAM.out.bam) 

But maybe I misunderstood your comment.
Can you elaborate more if it was the case?

What I mean is in this part of groupTuple , this part is grouping the align result bams according to bam files size, not the input fastq files. because you need to group the output bam according to bam files size, bam_bwa_to_sort channel will not emit element until all aligned bam has finished. what's more, there is no set 'size' parameter in groupTuple operator, so the groupTuple will block until all samples aligned bam finished in bwa-mem, because groupTuple is a block operator if you don't set 'size' parameter.

ok, I'll do some more tests for that.
I copied part of this code from dsl1 and tried to update it, it's definitively possible that I created issues.
Thanks for noticing that

Looking forward to your reply as soon as possible, I have come across the same problem in my code, but I don't know how to solve this problem, because the size numbers are Dynamically changing values for different samples which multiple run data.

Ok, re-read and tested things, and I understand what you want.
Sorry, I was still in a no coffee mode this morning.

[rogerzhanglijie]

can we split the input reads into single and multiple run before align? because if group the bam files after align, the group tuple will block and wait all samples had finished align step (containing single and multiple run)

[maxulysse]

I'm planning of rewriting the part that parse the tsv file(s) from the input data and write the meta map.
I just though this morning that it could be an idea to write in the meta map the number of lane (ie fastq pairs) for the sample.
That way, we'll know already without having to do any channel manipulation.
Still thinking of others way.
I'll consider your ideas too.
Thanks a lot for spotting this issue, and explaining it to me

[FriederikeHanssen]

Addressed in PR #388 by using groupTuple with previous known size. Workflow no advances whenever all parts for one patient are mapped