Configuration of CWL segmentation and analysis workflows

README.md

@@ -0,0 +1,103 @@
+# Common Workflow Language (CWL) Workflows
+
+CWL feature extraction workflow for imaging dataset
+
+##  Workflow Steps:
+
+create a [Conda](https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#activating-an-environment) environment using python = ">=3.9,<3.12"
+
+#### 1. Install polus-plugins.
+
+- clone a image-tools repository
+`git clone https://github.com/camilovelezr/image-tools.git ../`
+- cd `image-tools`
+- create a new branch
+`git checkout -b hd2  remotes/origin/hd2`
+- `pip install .`
+
+#### 2. Install workflow-inference-compiler.
+- clone a workflow-inference-compiler repository
+`git clone https://github.com/camilovelezr/workflow-inference-compiler.git ../`
+- cd `workflow-inference-compiler`
+- create a new branch
+`git checkout -b hd2  remotes/origin/hd2`
+- `pip install -e ".[all]"`
+
+#### 3. Install image-workflow.
+- cd `image-workflows`
+- poetry install
+
+#### Note:
+Ensure that the [docker-desktop](https://www.docker.com/products/docker-desktop/) is running in the background. To verify that it's operational, you can use the following command:
+`docker run -d -p 80:80 docker/getting-started` 
+This command will launch the `docker/getting-started container` in detached mode (-d flag), exposing port 80 on your local machine (-p 80:80). It's a simple way to test if Docker Desktop is functioning correctly.
+
+## Details 
+This workflow integrates eight distinct plugins, starting from data retrieval from [Broad Bioimage Benchmark Collection](https://bbbc.broadinstitute.org/), renaming files, correcting uneven illumination, segmenting nuclear objects, and culminating in the extraction of features from identified objects
+
+Below are the specifics of the plugins employed in the workflow
+1. [bbbc-download-plugin](https://github.com/saketprem/polus-plugins/tree/bbbc_download/utils/bbbc-download-plugin)
+2. [file-renaming-tool](https://github.com/PolusAI/image-tools/tree/master/formats/file-renaming-tool)
+3. [ome-converter-tool](https://github.com/PolusAI/image-tools/tree/master/formats/ome-converter-tool)
+4. [basic-flatfield-estimation-tool](https://github.com/PolusAI/image-tools/tree/master/regression/basic-flatfield-estimation-tool)
+5. [apply-flatfield-tool](https://github.com/PolusAI/image-tools/tree/master/transforms/images/apply-flatfield-tool)
+6. [kaggle-nuclei-segmentation](https://github.com/hamshkhawar/image-tools/tree/kaggle-nuclei_seg/segmentation/kaggle-nuclei-segmentation)
+7. [polus-ftl-label-plugin](https://github.com/hamshkhawar/image-tools/tree/kaggle-nuclei_seg/transforms/images/polus-ftl-label-plugin)
+8. [nyxus-plugin](https://github.com/PolusAI/image-tools/tree/kaggle-nuclei_seg/features/nyxus-plugin)
+
+## Execute CWL workflows
+Three different CWL workflows can be executed for specific datasets
+1. segmentation
+2. analysis
+
+During the execution of the segmentation workflow, `1 to 7` plugins will be utilized. However, for executing the analysis workflow, `1 to 8` plugins will be employed.
+If a user wishes to execute a workflow for a new dataset, they can utilize a sample YAML file to input parameter values. This YAML file can be saved in the desired subdirectory of the `configuration` folder with the name `dataset.yml`
+
+`python -m polus.image.workflows  --name="BBBC001" --workflow=analysis`
+
+A directory named `outputs` is generated, encompassing CLTs for each plugin, YAML files, and all outputs are stored within the `outdir` directory.
+```
+outputs
+├── experiment
+│   └── cwl_adapters
+|   experiment.cwl
+|   experiment.yml
+|
+└── outdir
+    └── experiment
+        ├── step 1 BbbcDownload
+        │   └── outDir
+        │       └── bbbc.outDir
+        │           └── BBBC
+        │               └── BBBC039
+        │                   └── raw
+        │                       ├── Ground_Truth
+        │                       │   ├── masks
+        │                       │   └── metadata
+        │                       └── Images
+        │                           └── images
+        ├── step 2 FileRenaming
+        │   └── outDir
+        │       └── rename.outDir
+        ├── step 3 OmeConverter
+        │   └── outDir
+        │       └── ome_converter.outDir
+        ├── step 4 BasicFlatfieldEstimation
+        │   └── outDir
+        │       └── estimate_flatfield.outDir
+        ├── step 5 ApplyFlatfield
+        │   └── outDir
+        │       └── apply_flatfield.outDir
+        ├── step 6 KaggleNucleiSegmentation
+        │   └── outDir
+        │       └── kaggle_nuclei_segmentation.outDir
+        ├── step 7 FtlLabel
+        │   └── outDir
+        │       └── ftl_plugin.outDir
+        └── step 8 NyxusPlugin
+            └── outDir
+                └── nyxus_plugin.outDir
+
+```
+#### Note:
+Step 7 and step 8 are executed only in the case of the `analysis` workflow.

configuration/analysis/BBBC001.yml

@@ -0,0 +1,13 @@
+---
+name : BBBC001
+file_pattern : /.*/.*/.*/Images/.*/.*_{row:c}{col:dd}f{f:dd}d{channel:d}.tif
+out_file_pattern : x{row:dd}_y{col:dd}_p{f:dd}_c{channel:d}.tif
+image_pattern: x{x:dd}_y{y:dd}_p{p:dd}_c{c:d}.ome.tif
+seg_pattern: x{x:dd}_y{y:dd}_p{p:dd}_c0.ome.tif
+ff_pattern: "x00_y03_p0\\(0-5\\)_c{c:d}_flatfield.ome.tif"
+df_pattern: "x00_y03_p0\\(0-5\\)_c{c:d}_darkfield.ome.tif"
+group_by: c
+map_directory: false
+features: ALL
+file_extension: pandas
+background_correction: false

configuration/analysis/BBBC039.yml

@@ -0,0 +1,13 @@
+---
+name : BBBC039
+file_pattern : /.*/.*/.*/Images/.*/.*_{row:c}{col:dd}_s{s:d}_w{channel:d}.*.tif
+out_file_pattern : x{row:dd}_y{col:dd}_p{s:dd}_c{channel:d}.tif
+image_pattern: x{x:dd}_y{y:dd}_p{p:dd}_c{c:d}.ome.tif
+seg_pattern: x{x:dd}_y{y:dd}_p{p:dd}_c1.ome.tif
+ff_pattern: "x\\(00-15\\)_y\\(01-24\\)_p0\\(1-9\\)_c{c:d}_flatfield.ome.tif"
+df_pattern: "x\\(00-15\\)_y\\(01-24\\)_p0\\(1-9\\)_c{c:d}_darkfield.ome.tif"
+group_by: c
+map_directory: false
+features: "ALL_INTENSITY"
+file_extension: pandas
+background_correction: false

configuration/analysis/sample.yml

@@ -0,0 +1,12 @@
+---
+name : 
+file_pattern :
+out_file_pattern : 
+image_pattern: 
+seg_pattern: 
+ff_pattern:
+df_pattern: 
+group_by: 
+map_directory: 
+features: 
+file_extension: 

configuration/segmentation/BBBC001.yml

@@ -0,0 +1,10 @@
+---
+name : BBBC001
+file_pattern : /.*/.*/.*/Images/.*/.*_{row:c}{col:dd}f{f:dd}d{channel:d}.tif
+out_file_pattern : x{row:dd}_y{col:dd}_p{f:dd}_c{channel:d}.tif
+image_pattern: x{x:dd}_y{y:dd}_p{p:dd}_c{c:d}.ome.tif
+seg_pattern: x{x:dd}_y{y:dd}_p{p:dd}_c0.ome.tif
+ff_pattern: "x00_y03_p0\\(0-5\\)_c{c:d}_flatfield.ome.tif"
+df_pattern: "x00_y03_p0\\(0-5\\)_c{c:d}_darkfield.ome.tif"
+group_by: c
+map_directory: false

configuration/segmentation/BBBC039.yml

@@ -0,0 +1,10 @@
+---
+name : BBBC039
+file_pattern : /.*/.*/.*/Images/.*/.*_{row:c}{col:dd}_s{s:d}_w{channel:d}.*.tif
+out_file_pattern : x{row:dd}_y{col:dd}_p{s:dd}_c{channel:d}.tif
+image_pattern: x{x:dd}_y{y:dd}_p{p:dd}_c{c:d}.ome.tif
+seg_pattern: x{x:dd}_y{y:dd}_p{p:dd}_c1.ome.tif
+ff_pattern: "x\\(00-15\\)_y\\(01-24\\)_p0\\(1-9\\)_c{c:d}_flatfield.ome.tif"
+df_pattern: "x\\(00-15\\)_y\\(01-24\\)_p0\\(1-9\\)_c{c:d}_darkfield.ome.tif"
+group_by: c
+map_directory: false

configuration/segmentation/sample.yml

@@ -0,0 +1,12 @@
+---
+name : 
+file_pattern : 
+out_file_pattern : 
+image_pattern: 
+seg_pattern: 
+ff_pattern:
+df_pattern: 
+group_by: 
+map_directory: 
+features: 
+file_extension: 

cwl_adapters/basic-flatfield-estimation.cwl → cwl-adapters/basic-flatfield-estimation.cwl

@@ -18,10 +18,6 @@ requirements:
       writable: true  # Output directories must be writable
   InlineJavascriptRequirement: {}
 
-# See https://github.com/PolusAI/workflow-inference-compiler/blob/master/docker_remove_entrypoints.py
-baseCommand: python3
-arguments: ["-m", "polus.plugins.regression.basic_flatfield_estimation"]
-
 # "jax._src.xla_bridge - WARNING  - An NVIDIA GPU may be present on this machine, but a CUDA-enabled jaxlib is not installed. Falling back to cpu."
 hints:
   cwltool:CUDARequirement:

cwl_adapters/bbbcdownload.cwl → cwl-adapters/bbbcdownload.cwl

@@ -7,10 +7,6 @@ doc: |-
   Downloads the datasets on the Broad Bioimage Benchmark Collection website
   https://github.com/saketprem/polus-plugins/tree/bbbc_download/utils/bbbc-download-plugin
 
-# See https://github.com/PolusAI/workflow-inference-compiler/blob/master/docker_remove_entrypoints.py
-baseCommand: python3
-arguments: ["-m", "polus.plugins.utils.bbbc_download"]
-
 requirements:
   DockerRequirement:
     dockerPull: polusai/bbbc-download-plugin:0.1.0-dev1

cwl_adapters/file-renaming.cwl → cwl-adapters/file-renaming.cwl

@@ -7,10 +7,6 @@ doc: |-
   Rename and store image collection files in a new image collection
   https://github.com/PolusAI/polus-plugins/tree/master/formats/file-renaming-plugin
 
-# See https://github.com/PolusAI/workflow-inference-compiler/blob/master/docker_remove_entrypoints.py
-baseCommand: python3
-arguments: ["-m", "polus.plugins.formats.file_renaming"]
-
 requirements:
   DockerRequirement:
     dockerPull: polusai/file-renaming-plugin:0.2.1-dev0  # NOTE: 0.2.3 not pushed yet

cwl_adapters/image_assembler.cwl → cwl-adapters/image_assembler.cwl

@@ -8,10 +8,6 @@ doc: |-
   This plugin assembles images into a stitched image using an image stitching vector.
   https://github.com/PolusAI/polus-plugins/tree/master/transforms/images/image-assembler-plugin
 
-# See https://github.com/PolusAI/workflow-inference-compiler/blob/master/docker_remove_entrypoints.py
-baseCommand: python3
-arguments: ["-m", "polus.plugins.transforms.images.image_assembler"]
-
 requirements:
   DockerRequirement:
     dockerPull: polusai/image-assembler-plugin:1.4.0-dev0

cwl_adapters/montage.cwl → cwl-adapters/montage.cwl

@@ -8,10 +8,6 @@ doc: |-
   This plugin generates a stitching vector that will montage images together.
   https://github.com/PolusAI/polus-plugins/tree/master/transforms/images/montage-plugin
 
-# See https://github.com/PolusAI/workflow-inference-compiler/blob/master/docker_remove_entrypoints.py
-baseCommand: python3
-arguments: ["-m", "polus.plugins.transforms.images.montage"]
-
 requirements:
   DockerRequirement:
     dockerPull: polusai/montage-plugin:0.5.0

cwl_adapters/ome-converter.cwl → cwl-adapters/ome-converter.cwl

@@ -8,13 +8,9 @@ doc: |-
   This WIPP plugin converts BioFormats supported data types to the OME Zarr file format.
   https://github.com/PolusAI/polus-plugins/tree/master/formats/ome-converter-plugin
 
-# See https://github.com/PolusAI/workflow-inference-compiler/blob/master/docker_remove_entrypoints.py
-baseCommand: python3
-arguments: ["-m", "polus.plugins.formats.ome_converter"]
-
 requirements:
   DockerRequirement:
-    dockerPull: jakefennick/ome-converter-plugin:0.3.2
+    dockerPull: polusai/ome-converter-plugin:0.3.2-dev2
   # See https://www.commonwl.org/v1.0/CommandLineTool.html#InitialWorkDirRequirement
   InitialWorkDirRequirement:
     listing:

cwl_adapters/precompute_slide.cwl → cwl-adapters/precompute_slide.cwl

@@ -8,10 +8,6 @@ doc: |-
   This plugin generates image pyramids in multiple viewing formats.
   https://github.com/PolusAI/polus-plugins/tree/master/visualization/polus-precompute-slide-plugin
 
-# See https://github.com/PolusAI/workflow-inference-compiler/blob/master/docker_remove_entrypoints.py
-baseCommand: python3
-arguments: ["-m", "polus.plugins.visualization.precompute_slide"]
-
 requirements:
   DockerRequirement:
     dockerPull: polusai/precompute-slide-plugin:1.7.0-dev0

cwl_adapters/BbbcDownload.cwl

@@ -0,0 +1,29 @@
+class: CommandLineTool
+cwlVersion: v1.2
+inputs:
+  name:
+    inputBinding:
+      prefix: --name
+    type: string
+  outDir:
+    inputBinding:
+      prefix: --outDir
+    type: Directory
+outputs:
+  outDir:
+    outputBinding:
+      glob: $(inputs.outDir.basename)
+    type: Directory
+requirements:
+  DockerRequirement:
+    dockerPull: polusai/bbbc-download-plugin:0.1.0-dev1
+  EnvVarRequirement:
+    envDef:
+      HOME: /home/polusai
+  InitialWorkDirRequirement:
+    listing:
+    - entry: $(inputs.outDir)
+      writable: true
+  InlineJavascriptRequirement: {}
+  NetworkAccess:
+    networkAccess: true

cwl_adapters/FileRenaming.cwl

@@ -0,0 +1,41 @@
+class: CommandLineTool
+cwlVersion: v1.2
+inputs:
+  filePattern:
+    inputBinding:
+      prefix: --filePattern
+    type: string
+  inpDir:
+    inputBinding:
+      prefix: --inpDir
+    type: Directory
+  mapDirectory:
+    inputBinding:
+      prefix: --mapDirectory
+    type: boolean?
+  outDir:
+    inputBinding:
+      prefix: --outDir
+    type: Directory
+  outFilePattern:
+    inputBinding:
+      prefix: --outFilePattern
+    type: string
+outputs:
+  outDir:
+    outputBinding:
+      glob: $(inputs.outDir.basename)
+    type: Directory
+requirements:
+  DockerRequirement:
+    dockerPull: polusai/file-renaming-tool:0.2.4-dev1
+  EnvVarRequirement:
+    envDef:
+      HOME: /home/polusai
+  InitialWorkDirRequirement:
+    listing:
+    - entry: $(inputs.outDir)
+      writable: true
+  InlineJavascriptRequirement: {}
+  NetworkAccess:
+    networkAccess: true

pyproject.toml

@@ -0,0 +1,36 @@
+[tool.poetry]
+name = "polus-image-workflows"
+version = "0.1.1-dev1"
+description = "Build and execute pipelines of polus plugins on Compute."
+authors = ["Hamdah Shafqat Abbasi <[email protected]>"]
+readme = "README.md"
+packages = [{include = "polus", from = "src"}]
+
+[tool.poetry.dependencies]
+python = ">=3.9,<3.12"
+typer = "^0.9.0"
+pyyaml = "^6.0.1"
+pydantic = "^2.6.1"
+#polus-plugins = {path = "../image-tools", develop = true}
+workflow-inference-compiler = {path = "../workflow-inference-compiler", develop = true}
+
+[tool.poetry.group.dev.dependencies]
+jupyter = "^1.0.0"
+nbconvert = "^7.11.0"
+pytest = "^7.4.4"
+bump2version = "^1.0.1"
+pre-commit = "^3.3.3"
+black = "^23.3.0"
+ruff = "^0.0.274"
+mypy = "^1.4.0"
+pytest-xdist = "^3.3.1"
+pytest-sugar = "^0.9.7"
+
+[build-system]
+requires = ["poetry-core>=1.0.0"]
+build-backend = "poetry.core.masonry.api"
+
+[tool.pytest.ini_options]
+addopts = [
+    "--import-mode=importlib",
+]