better error messages

[brentp]

Nextflow is great!, But, it is sometimes quite difficult to debug nextflow errors due to uninformative error messages. Here is one example:

nextflow.enable.dsl=2                                                                                                                                       
process wtf {                                                                                                                                               
    input:                                                                          
    val(a)                                                                      
    val(b)                                                                      
    val(c)                                                                      
                                                                                
    output:
    path("t.txt")                                                           
                                                                                
    script:                                                                         
    """                                                                           
    echo hi > t.txt                                                               
    """                                                                                                                                                       
}                                                                                                                                                           
workflow {                                                                      
    wtf(1, 2 3, 4)                                                              
}   

This gives:

N E X T F L O W  ~  version 20.10.0
Launching `t.nf` [stoic_wozniak] - revision: a7ed7ff7fb
Script compilation error
- file : /home/brentp/t.nf
- cause: Unexpected input: '{' @ line 19, column 10.
   workflow {
            ^

1 error

I will add others in comments below as I encounter them.

Solution is to add a comma between args in the wtf call:

workflow {                                                                        
 // add   ,  between 2 and 3 
 wtf(1, 2 3, 4)                                                              
}   

[bobamess]

I've seen this error several times and now take it to mean that there is a syntax error, somewhere, as in your example i.e. wtf(1, 2 3, 4) should be wtf(1, 2, 3, 4). But finding out where can be difficult and time consuming. I had one case where there was an extra space at the end of the line after a continuation backslash. It would be useful if the error message could direct you to the actual line where the syntax error is and not just to the line at the beginning of a containing block such as 'workflow {'. But maybe that's just difficult to determine?

[pditommaso]

Yes, we are aware of this. Unfortunately, solving this problem it's not an easy task, since Nextflow does not have its own syntax parser but instead uses AST manipulation techniques to extend the Groovy syntax into Nextflow DSL. I fear that something related to the new syntax parser introduced by Groovy 3.

[brentp]

OK. I figured it was a hard problem. For me, it's one of the main things I encounter -- the debugging is difficult.
Of course feel free to close if this is tracked elsewhere or not resolvable.

[pditommaso]

Better to keep it open as a reminder of a problem we need to solve. If you could provide a hint of the solution, it could help us in the troubleshooting.

[rowanworth]

This one is a real pain when the cause of the syntax error is within a lengthy script embedded in the process. In my case it was an awk regex causing the issue -- the escaping needed different treatment within nextflow. Specifically my regex included a match on / ie. this works on the command-line:

$ echo "2022/01/31T04:05:06" |awk '{nf=split($0, f, /[T:\/]/); print f[4]}' 
04

But within a triple-single-quoted nextflow shell: directive, it does not:

$ cat >regex.nf
date='2022/01/31T04:05:06'

process extractHour {
input:
    stdin date

output:
    stdout into finalResult

shell:
    '''
    awk '{nf=split($0, f, /[T:\/]/); print f[4]}' 
    '''
}

$ nextflow regex.nf
N E X T F L O W  ~  version 21.04.1
Launching `regex.nf` [desperate_ekeblad] - revision: 013076b62b
Script compilation error
- file : /tmp/nf/regex.nf
- cause: Unexpected input: '{' @ line 3, column 21.
   process extractHour {
                       ^

1 error

The fix is to use a double-backslash within the awk script:

shell:
    '''
    awk '{nf=split($0, f, /[T:\\/]/); print f[4]}' 
    '''

(technically this regex also works without the backslash, but that's beside the point :))

[d3v-null]

for such and advanced and powerful dsl, the error messages are very unhelpful for new users. I found that using script templates was the best way to get around this issue.

[jambler24]

I'm having the same problem with poor error reporting. The error is:

- cause: Unexpected input: '{' @ line 106, column 25.
   process Run_trimmomatic {

Which is seen when running a process with this line:

publishDir = "${output_dir}/trimmomatic", pattern: '*.log'

Removing the pattern: '*.log' part fixes it, for not apparent reason.

publishDir = "${output_dir}/trimmomatic"

[bobamess]

I think you will find that all of the following alternatives will work (DSL2)

publishDir "${output_dir}/trimmomatic", pattern: '*.log'

publishDir "${output_dir}/trimmomatic", pattern: "*.log"

publishDir path: "${output_dir}/trimmomatic", pattern: '*.log'

publishDir path: "${output_dir}/trimmomatic", pattern: "*.log"

I think the reason why below does work has something to do with it being interpreted as a name-value pair because of the '='

publishDir = "${output_dir}/trimmomatic"

but for

publishDir = "${output_dir}/trimmomatic", pattern: '*.log'

the value part is taken to be "${output_dir}/trimmomatic", pattern: '*.log' and is not interpreted by Nextflow.

Personally, I prefer to put publishDir in nextflow.config, as example below, and not in the process definition in main.nf, or wherever it is

process {
	withName: my_process {
		publishDir {
			path	= "${output_dir}/trimmomatic"
			pattern = "*.log"
			mode 	= "link" // or "copy" if you prefer
		}
	}
}

[jambler24]

Spot on with the =, that was the issue, thanks!

[cahofer]

I also wanted to point out that the debugging with nextflow is really very non-informative, time consuming and frustrating!

[pditommaso]

thank you!

[pauline-ng]

I spent 1 hour debugging this, because my error was after the line error

line8      workflow pipeline {
line9           take:
line10               var1
line11               var2,
line12               var3

                         ....etc

Gave the error

• cause: Unexpected input: '{' @ line 8.

My mistake was having a comma at line 11. I wasted a lot of time looking for the error before line 8, but the error was later in the code (so don't just look upstream for the error, look downstream as well.)

I agree with others; please improve debugging in nextflow.

[kemin711]

Usually the error is after this point. So the message is useful.