Update subworkflow for deepvariant to version 1.8.0

[fa2k]

Update the deepvariant variant calling subworkflow (nf-core/deepvariant) to version 1.8.0.

Other module subcommands were updated in #7301.

This work is based on @fellen31's original pull request to update the subworkflow, and adds some handling of the "small model". Summary of changes:

• Container version increased
• Enable --regions parameter to DEEPVARIANT_POSTPROCESSVARIANTS process
• Pass the small model outputs from DEEPVARIANT_MAKEEXAMPLES to DEEPVARIANT_POSTPROCESSVARIANTS if small model is used
• Update tests to use small model (which is default), but add a test to disable the small model
• Add a test to ensure that the subworkflow output is equal to the integrated DEEPVARIANT_RUNDEEPVARIANT process output

The future of the subworkflow in nf-core is a bit uncertain. Recent updates of DeepVariant have required an excessive amount of changes to be made in the subworkflow. Due to the argument handling with ext.args, they would also require many changes by the pipelines using the subworkflow, to keep up with the default arguments used by the run_deepvariant script.

The purpose of this subworkflow is to optimize compute resource usage, allocating GPUs only when needed. With version 1.8.0, Google released a new "fast pipeline" (https://github.com/google/deepvariant/blob/r1.8/docs/deepvariant-fast-pipeline-case-study.md) that could solve the same problem in a different way. At the moment, the fast pipeline is about as complex as the subworkflow approach, so I'm suggesting we keep the subworkflow for now.

PR checklist

Closes #XXX

• This comment contains a description of changes (with reason).
• If you've fixed a bug or added code that should be tested, add tests!
• If you've added a new tool - have you followed the module conventions in the contribution docs
• If necessary, include test data in your PR.
• Remove all TODO statements.
• Emit the versions.yml file.
• Follow the naming conventions.
• Follow the parameters requirements.
• Follow the input/output options guidelines.
• Add a resource label
• Use BioConda and BioContainers if possible to fulfil software requirements.
• Ensure that the test works with either Docker / Singularity. Conda CI tests can be quite flaky:
  • For modules:
    • nf-core modules test <MODULE> --profile docker
    • nf-core modules test <MODULE> --profile singularity
    • nf-core modules test <MODULE> --profile conda
  • For subworkflows:
    • nf-core subworkflows test <SUBWORKFLOW> --profile docker
    • nf-core subworkflows test <SUBWORKFLOW> --profile singularity
    • nf-core subworkflows test <SUBWORKFLOW> --profile conda

[fellen31]

Looks like some tests are failing?

[fa2k]

Oh sorry I missed that. I will check it and let you know when it's fixed. It may be related to the nextflow version.

[fa2k]

@fellen31 I've fixed the test for POSTPROCESSVARIANTS, adding the correct input tuple. Now all the tests pass.

[fellen31]

Nice, thanks for continuing this.

[fellen31]

Not sure why you would need this?

[fa2k]

I was working around an error message I got when using the input variable directly. I have to do some more testing, because it seems to work now.

[fa2k]

The issue seems to be that if I remove def small_model_calls_copy = small_model_calls, I get this error:

│   Nextflow stdout:                                                                                                                              │
│                                                                                                                                                 │
│   ERROR ~ Module compilation error                                                                                                              │
│   - file : /data/projects/modules/subworkflows/nf-core/deepvariant/tests/../../../../modules/nf-core/deepvariant/postprocessvariants/main.nf    │
│   - cause: Variable `small_model_calls` already defined in the process scope @ line 49, column 36.                                              │
│              def small_model_matcher = (small_model_calls[0].baseName =~ /^(.+)-\d{5}-of-(\d{5})$/)                                             │
│                                         ^                                                                                                       │
│                                                                                                                                                 │
│   1 error                                                                                                                                       │
│                                                                                                                                                 │
│                                                                                                                                                 │
│    -- Check script '/data/projects/modules/subworkflows/nf-core/deepvariant/tests/../main.nf' at line: 3 or see                                 │
│ '/data/projects/modules/.nf-test/tests/108496a30634aa9373981b51feca7830/meta/nextflow.log' file for more details                                │
│   Nextflow stderr:                                                                                                                              │

[fellen31]

Hm, and if you remove the def before small_model_matcher?

[fa2k]

Seems to work if I remove the def's on everything except for small_model_arg. I'll create a commit for this.

[fa2k]

Thanks for the tip :)

[fellen31]

Is this run in the CI?

[fa2k]

I haven't done anything explicitly to make it run in CI. I will try to confirm whether it runs. Do you think it should be enabled in CI? We may have a situation (like just happened) where we want to update rundeepvariant without the subworkflow, and can ignore this test result.

[fellen31]

I think you are right. Nice to be able to test that the subworkflow produces the same output as the rundeepvariant, but probably shouldn't be in the CI.

[fa2k]

It's enabled in CI now, that happened automatically. I have to look into how it can be disabled.

[fa2k]

Remove the tags?

[fa2k]

I've added them to the skip file, but the format is a bit different than usual, because I only want to skip one of the test files. I have to check that it still runs main.nf and that it skips equality.nf

[fa2k]

Looks like it's successfully disabled now

[fellen31]

Nice!

Have you tested the differences in time and resources for the subworkflow vs rundeepvariant for 1.8? Would be interesting to see how much difference it makes.

[fa2k]

Have you tested the differences in time and resources for the subworkflow vs rundeepvariant for 1.8? Would be interesting to see how much difference it makes.

No, but it would be interesting to know. I'll try to get this information, but also merge when ready.