reporting.py error:ValueError: cannot reindex from a duplicate axis

[WeberKang]

Check Documentation

I have checked the following places for your error:

• nf-core website: troubleshooting
• nf-core/cutandrun pipeline documentation

Description of the bug

I'm a beginner, and since I don't have root access to the server and no network, I chose to test the pipeline locally with a conda environment, and since there is no environment.yml like atac-seq. I had to install the appropriate packages myself according to the channel. Everything went smoothly until Process NFCORE_CUTANDRUN:CUTANDRUN:GENERATE_REPORTS

Steps to reproduce

Steps to reproduce the behaviour:

1. Command line:

      --meta meta_table.csv \
      --meta_ctrl meta_table_ctrl.csv \
      --raw_frag "*.frag_len.txt" \
      --bin_frag "*bin500.awk.bed" \
      --seacr_bed "*bed.*.bed" \
      --output . \
      --log log.txt
  
  if [ -f "03_03_frag_len_mqc.txt" ]; then
      cat frag_len_header.txt 03_03_frag_len_mqc.txt > frag_len_mqc.yaml
  fi
  
  python --version | grep -E -o "([0-9]{1,}\.)+[0-9]{1,}" > python.version.txt

2. See error:

  2022-01-04 20:16:02,898:gen_img:INFO - Generating plots to output folder
  Traceback (most recent call last):
    File "/gpfs/home/zengchunhua/.nextflow/assets/nf-core/cutandrun/bin/reporting.py", line 72, in <module>
      parsed_args.func(parsed_args)
    File "/gpfs/home/zengchunhua/.nextflow/assets/nf-core/cutandrun/bin/reporting.py", line 42, in gen_png
      report_gen.generate_cutandrun_reports(output_path)
    File "/gpfs/home/zengchunhua/.nextflow/assets/nf-core/cutandrun/bin/lib/reports.py", line 63, in generate_cutandrun_reports
      plots, data, mqc_frag_hist = self.generate_reports()
    File "/gpfs/home/zengchunhua/.nextflow/assets/nf-core/cutandrun/bin/lib/reports.py", line 271, in generate_reports
      plot4, data4 = self.fraglen_summary_histogram()
    File "/gpfs/home/zengchunhua/.nextflow/assets/nf-core/cutandrun/bin/lib/reports.py", line 454, in fraglen_summary_histogram
      ax = sns.lineplot(data=self.frag_hist, x="Size", y="Occurrences", hue="group", style="replicate", palette = "magma")
    File "/gpfs/home/zengchunhua/bin/Miniconda3/envs/nf-cut/lib/python3.8/site-packages/seaborn/_decorators.py", line 46, in inner_f
      return f(**kwargs)
    File "/gpfs/home/zengchunhua/bin/Miniconda3/envs/nf-cut/lib/python3.8/site-packages/seaborn/relational.py", line 710, in lineplot
      p.plot(ax, kwargs)
    File "/gpfs/home/zengchunhua/bin/Miniconda3/envs/nf-cut/lib/python3.8/site-packages/seaborn/relational.py", line 471, in plot
      for sub_vars, sub_data in self.iter_data(grouping_vars, from_comp_data=True):
    File "/gpfs/home/zengchunhua/bin/Miniconda3/envs/nf-cut/lib/python3.8/site-packages/seaborn/_core.py", line 983, in iter_data
      data = self.comp_data
    File "/gpfs/home/zengchunhua/bin/Miniconda3/envs/nf-cut/lib/python3.8/site-packages/seaborn/_core.py", line 1057, in comp_data
      comp_col.loc[orig.index] = pd.to_numeric(axis.convert_units(orig))
    File "/gpfs/home/zengchunhua/bin/Miniconda3/envs/nf-cut/lib/python3.8/site-packages/pandas/core/indexing.py", line 692, in __setitem__
      iloc._setitem_with_indexer(indexer, value, self.name)
    File "/gpfs/home/zengchunhua/bin/Miniconda3/envs/nf-cut/lib/python3.8/site-packages/pandas/core/indexing.py", line 1637, in _setitem_with_indexer
      self._setitem_single_block(indexer, value, name)
    File "/gpfs/home/zengchunhua/bin/Miniconda3/envs/nf-cut/lib/python3.8/site-packages/pandas/core/indexing.py", line 1851, in _setitem_single_block
      value = self._align_series(indexer, Series(value))
    File "/gpfs/home/zengchunhua/bin/Miniconda3/envs/nf-cut/lib/python3.8/site-packages/pandas/core/indexing.py", line 1987, in _align_series
      ser = ser.reindex(obj.axes[0][indexer[0]], copy=True)._values
    File "/gpfs/home/zengchunhua/bin/Miniconda3/envs/nf-cut/lib/python3.8/site-packages/pandas/core/series.py", line 4345, in reindex
      return super().reindex(index=index, **kwargs)
    File "/gpfs/home/zengchunhua/bin/Miniconda3/envs/nf-cut/lib/python3.8/site-packages/pandas/core/generic.py", line 4811, in reindex
      return self._reindex_axes(
    File "/gpfs/home/zengchunhua/bin/Miniconda3/envs/nf-cut/lib/python3.8/site-packages/pandas/core/generic.py", line 4832, in _reindex_axes
      obj = obj._reindex_with_indexers(
    File "/gpfs/home/zengchunhua/bin/Miniconda3/envs/nf-cut/lib/python3.8/site-packages/pandas/core/generic.py", line 4877, in _reindex_with_indexers
      new_data = new_data.reindex_indexer(
    File "/gpfs/home/zengchunhua/bin/Miniconda3/envs/nf-cut/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 1301, in reindex_indexer
      self.axes[axis]._can_reindex(indexer)
    File "/gpfs/home/zengchunhua/bin/Miniconda3/envs/nf-cut/lib/python3.8/site-packages/pandas/core/indexes/base.py", line 3477, in _can_reindex
      raise ValueError("cannot reindex from a duplicate axis")
  ValueError: cannot reindex from a duplicate axis

Expected behaviour

Log files

Have you provided the following extra information/files:

• The command used to run the pipeline
  nextflow run nf-core/cutandrun -r 1.0.0 -name cut_test -params-file nf-params.json
• The .nextflow.log file

N E X T F L O W  ~  version 21.10.0
Launching `nf-core/cutandrun` [desperate_ekeblad] - revision: 5b9f4fad41 [1.0.0]


------------------------------------------------------
                                        ,--./,-.
        ___     __   __   __   ___     /,-._.--~'
  |\ | |__  __ /  ` /  \ |__) |__         }  {
  | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                        `._,._,'
  nf-core/cutandrun v1.0.0
------------------------------------------------------
Core Nextflow options
  revision     : 1.0.0
  runName      : desperate_ekeblad
  launchDir    : /gpfs/home/zengchunhua/cutandrun
  workDir      : /gpfs/home/zengchunhua/cutandrun/work
  projectDir   : /gpfs/home/zengchunhua/.nextflow/assets/nf-core/cutandrun
  userName     : zengchunhua
  profile      : standard
  configFiles  : /gpfs/home/zengchunhua/.nextflow/assets/nf-core/cutandrun/nextflow.config

Input/output options
  input        : GSE145187.csv

Reference data options
  gtf          : hg38.ncbiRefSeq.gtf
  spikein_fasta: E.coli_K12_MG1655.fa
  fasta        : hg38.fa
  igenomes_base: s3://ngi-igenomes/igenomes/

!! Only displaying parameters that differ from the pipeline defaults !!
------------------------------------------------------
If you use nf-core/cutandrun for your analysis please cite:

* The nf-core framework
  https://doi.org/10.1038/s41587-020-0439-x

* Software dependencies
  https://github.com/nf-core/cutandrun/blob/master/CITATIONS.md
------------------------------------------------------
WARN: =============================================================================
  No genome blacklist file specified, switching to dummy empty file...
===================================================================================


executor >  local (36)
[50/21a1d3] process > NFCORE_CUTANDRUN:CUTANDRUN:... [100%] 1 of 1, cached: 1 ✔
[07/64bd79] process > NFCORE_CUTANDRUN:CUTANDRUN:... [100%] 1 of 1, cached: 1 ✔
[7e/ad17f8] process > NFCORE_CUTANDRUN:CUTANDRUN:... [100%] 1 of 1, cached: 1 ✔
[43/b8ecee] process > NFCORE_CUTANDRUN:CUTANDRUN:... [100%] 1 of 1, cached: 1 ✔
[e9/0fb1e5] process > NFCORE_CUTANDRUN:CUTANDRUN:... [100%] 1 of 1, cached: 1 ✔
[-        ] process > NFCORE_CUTANDRUN:CUTANDRUN:... -
[10/5305ee] process > NFCORE_CUTANDRUN:CUTANDRUN:... [100%] 6 of 6, cached: 6 ✔
[1a/f7dc1c] process > NFCORE_CUTANDRUN:CUTANDRUN:... [100%] 6 of 6, cached: 6 ✔
[dc/b7abdb] process > NFCORE_CUTANDRUN:CUTANDRUN:... [100%] 6 of 6, cached: 6 ✔
[d3/04f80f] process > NFCORE_CUTANDRUN:CUTANDRUN:... [100%] 6 of 6, cached: 6 ✔
[29/ec95bb] process > NFCORE_CUTANDRUN:CUTANDRUN:... [100%] 6 of 6, cached: 6 ✔
[17/7501b9] process > NFCORE_CUTANDRUN:CUTANDRUN:... [100%] 6 of 6, cached: 6 ✔
[dc/ac8320] process > NFCORE_CUTANDRUN:CUTANDRUN:... [100%] 6 of 6, cached: 6 ✔
[88/c28cb7] process > NFCORE_CUTANDRUN:CUTANDRUN:... [100%] 6 of 6, cached: 6 ✔
[96/2c1b12] process > NFCORE_CUTANDRUN:CUTANDRUN:... [100%] 6 of 6, cached: 6 ✔
[5f/228d15] process > NFCORE_CUTANDRUN:CUTANDRUN:... [100%] 6 of 6, cached: 6 ✔
[d8/03e2a9] process > NFCORE_CUTANDRUN:CUTANDRUN:... [100%] 6 of 6, cached: 6 ✔
[49/6b3651] process > NFCORE_CUTANDRUN:CUTANDRUN:... [100%] 6 of 6, cached: 6 ✔
[0e/c34dda] process > NFCORE_CUTANDRUN:CUTANDRUN:... [100%] 6 of 6, cached: 6 ✔
[24/3cfa9c] process > NFCORE_CUTANDRUN:CUTANDRUN:... [100%] 6 of 6, cached: 6 ✔
[89/23b003] process > NFCORE_CUTANDRUN:CUTANDRUN:... [100%] 6 of 6, cached: 6 ✔
[05/4301a9] process > NFCORE_CUTANDRUN:CUTANDRUN:... [100%] 6 of 6, cached: 6 ✔
[a8/a2e1b4] process > NFCORE_CUTANDRUN:CUTANDRUN:... [100%] 6 of 6, cached: 2 ✔
[58/ca0934] process > NFCORE_CUTANDRUN:CUTANDRUN:... [100%] 6 of 6, cached: 2 ✔
[07/ad9048] process > NFCORE_CUTANDRUN:CUTANDRUN:... [100%] 6 of 6, cached: 2 ✔
[e3/c18948] process > NFCORE_CUTANDRUN:CUTANDRUN:... [100%] 2 of 2, cached: 2 ✔
[df/8def4b] process > NFCORE_CUTANDRUN:CUTANDRUN:... [100%] 6 of 6, cached: 4 ✔
[50/58ba92] process > NFCORE_CUTANDRUN:CUTANDRUN:... [100%] 6 of 6, cached: 1 ✔
[30/de8e72] process > NFCORE_CUTANDRUN:CUTANDRUN:... [100%] 6 of 6, cached: 1 ✔
[52/ac8bfb] process > NFCORE_CUTANDRUN:CUTANDRUN:... [100%] 6 of 6, cached: 1 ✔
[ca/4ee9aa] process > NFCORE_CUTANDRUN:CUTANDRUN:... [100%] 6 of 6, cached: 6 ✔
[27/3ba016] process > NFCORE_CUTANDRUN:CUTANDRUN:... [100%] 6 of 6, cached: 6 ✔
[da/0f4063] process > NFCORE_CUTANDRUN:CUTANDRUN:... [100%] 6 of 6, cached: 6 ✔
[fa/9f3c9f] process > NFCORE_CUTANDRUN:CUTANDRUN:... [100%] 6 of 6, cached: 6 ✔
[cd/653326] process > NFCORE_CUTANDRUN:CUTANDRUN:... [100%] 6 of 6, cached: 6 ✔
[a3/0e6d16] process > NFCORE_CUTANDRUN:CUTANDRUN:... [100%] 6 of 6, cached: 6 ✔
[97/3edf1c] process > NFCORE_CUTANDRUN:CUTANDRUN:... [100%] 6 of 6, cached: 6 ✔
[29/36dfd8] process > NFCORE_CUTANDRUN:CUTANDRUN:... [100%] 4 of 4, cached: 4 ✔
[64/69ee9c] process > NFCORE_CUTANDRUN:CUTANDRUN:... [100%] 4 of 4, cached: 4 ✔
[65/9bcd5f] process > NFCORE_CUTANDRUN:CUTANDRUN:... [100%] 1 of 1, cached: 1 ✔
[fb/848416] process > NFCORE_CUTANDRUN:CUTANDRUN:... [100%] 1 of 1 ✔
[eb/fd0e62] process > NFCORE_CUTANDRUN:CUTANDRUN:... [100%] 1 of 1 ✔
[91/96ee6a] process > NFCORE_CUTANDRUN:CUTANDRUN:... [100%] 1 of 1 ✔
[6c/3d0295] process > NFCORE_CUTANDRUN:CUTANDRUN:... [100%] 2 of 2, cached: 2 ✔
[00/a28cdb] process > NFCORE_CUTANDRUN:CUTANDRUN:... [100%] 2 of 2, cached: 2 ✔
[28/814671] process > NFCORE_CUTANDRUN:CUTANDRUN:... [100%] 2 of 2, cached: 2 ✔
[e9/41cd09] process > NFCORE_CUTANDRUN:CUTANDRUN:... [100%] 1 of 1, cached: 1 ✔
[04/3333e8] process > NFCORE_CUTANDRUN:CUTANDRUN:... [100%] 6 of 6, cached: 6 ✔
[84/fd8554] process > NFCORE_CUTANDRUN:CUTANDRUN:... [100%] 6 of 6, cached: 6 ✔
[80/435de6] process > NFCORE_CUTANDRUN:CUTANDRUN:... [100%] 6 of 6, cached: 6 ✔
[75/16ff30] process > NFCORE_CUTANDRUN:CUTANDRUN:... [100%] 6 of 6, cached: 6 ✔
[57/99451a] process > NFCORE_CUTANDRUN:CUTANDRUN:... [100%] 6 of 6, cached: 6 ✔
[09/5d3b6b] process > NFCORE_CUTANDRUN:CUTANDRUN:... [100%] 6 of 6, cached: 6 ✔
[fa/b08584] process > NFCORE_CUTANDRUN:CUTANDRUN:... [100%] 6 of 6, cached: 4 ✔
[ac/78c2e9] process > NFCORE_CUTANDRUN:CUTANDRUN:... [100%] 1 of 1, cached: 1 ✔
[68/890f9a] process > NFCORE_CUTANDRUN:CUTANDRUN:... [100%] 4 of 4, cached: 3 ✔
[fb/49eacb] process > NFCORE_CUTANDRUN:CUTANDRUN:... [100%] 4 of 4, cached: 4 ✔
[70/39494e] process > NFCORE_CUTANDRUN:CUTANDRUN:... [100%] 4 of 4, cached: 4 ✔
[1c/55709e] process > NFCORE_CUTANDRUN:CUTANDRUN:... [100%] 1 of 1, cached: 1 ✔
[ab/c4c3c3] process > NFCORE_CUTANDRUN:CUTANDRUN:... [100%] 1 of 1, cached: 1 ✔
[b5/c44669] process > NFCORE_CUTANDRUN:CUTANDRUN:... [  0%] 0 of 1
[-        ] process > NFCORE_CUTANDRUN:CUTANDRUN:... -
[-        ] process > NFCORE_CUTANDRUN:CUTANDRUN:... -
Error executing process > 'NFCORE_CUTANDRUN:CUTANDRUN:GENERATE_REPORTS'

Caused by:
  Process `NFCORE_CUTANDRUN:CUTANDRUN:GENERATE_REPORTS` terminated with an error exit status (1)

System

• Hardware: HPC
• Executor: PBS local
• OS: Red Hat
• Version 7.5

Nextflow Installation

• Version: 21.10.0.5640

Container engine

• Engine: Conda
• version: 4.11.0

Additional context

The PBS task node does not have a network, I wrote environment.yml based on the pipeline requirements

channels:
  - conda-forge
  - bioconda
  - defaults
dependencies:
  - python=3.8.3
  - numpy
  - pandas=1.2.5
  - seaborn
  - pyranges
  - pysam
  - dask
  - deeptools
  - fastqc
  - trim-galore
  - bowtie2
  - samtools
  - picard
  - bamtools
  - bedtools
  - ucsc-bedgraphtobigwig
  - deeptools
  - multiqc
  - igv
  - pip
  - seacr
  - ucsc-bedclip
  - upsetplot

[chris-cheshire]

Hi @WeberKang thanks for your issue! If you want to run the pipeline locally. I would advise that you install docker desktop and run the with the -profile docker switch. It is not intended functionality for you to install the packages yourself as this is a DSL2 pipeline (ATAC is DSL1 which is why it has a single package list). Failing this, you should also try running with the -profile conda switch. This pipeline really is designed to be run using containers though I must stress.