Argo Workflows (#1252)

* initial attempt to add argo-workflows * add enable/disable argo functionality, comment out juptyerflow resources, add custom options for argo helm chart * add value.yaml, add ingressroute to expose argo dashboard * rename helm deployment * add argo check and only check endpoint that are enabled * reorder outputs * add lots of comments for automatic pickup by prometheus and toggling rbac for individual argo workflows components * cleanup comments, add emissary containerRuntime by default * first pass at docs * remove jupyterflow commented out resources * Add auth to argo (#1249) * Mdformat tables (#1186) * Add mdformat-tables * Run mdformat on all files * mdformat only docs folder (restore .github md files) * Fix some vale, restore README/RELEASE * [ImgBot] Optimize images (#1187) /docs/source/images/dev_postman_for_keycloak.png -- 298.79kb -> 270.60kb (9.44%) Signed-off-by: ImgBotApp <[email protected]> Co-authored-by: ImgBotApp <[email protected]> * Bump conda-store version to 0.3.14 (#1192) * Allow terraform init to upgrade providers within version specification (#1194) * Allow terraform init to upgrade providers within version specification Closes #1193 * Black formatting * Adding missing __init__ files (#1196) * Adding missing __init__ files Closes #1195 * Explicitely using qhub for package * Give hint on what to include * Release 0.3.15 for Conda-Store (#1205) * Profilegroups (#1203) * Fix in case groups is None * access all/keycloak/yaml * jupyterlabproflies mapper * Keycloak profiles working * ignore changes to keycloak group attributes * qhub upgrade for jupyterlab profiles * docs for jupyterlabprofiles * Renamed to jupyterlab_profiles * Render `.gitignore`, black py files (#1206) * Render .gitignore * Render clean .gitignore * Add unit test * Upgrade black * Upgrade black * black format * exclude qhub/_version.py from black * Black what needs blackening * Fix * Fix * Update qhub-dask version (#1224) * Fix env doc links and add corresponding tests (#1216) * Fix env doc links and add corresponding tests * fix broken image link * fix black formatting * map(any) -> any (#1213) * Update release notes - justification for changes in `v0.4.0` (#1178) * Update release notes * Remove ref to cookiecutter * Update link-checker version * Fix * Use lycee link-checker instead * Remove lycee, update md config.json * Revert version * Release notes cleanup * Rewording * Add to vocab * Fix table * Add explicit warning about release * Update README.md Fixed some syntax/grammar issues. * Minor updates Co-authored-by: Christopher Ostrouchov <[email protected]> Co-authored-by: Shannon <[email protected]> * Merge spawner and profile env vars * Support for pinning the IP address of the load balancer via terraform overrides (#1235) * Suport adding load balancer annotations and ip via terraform overrides * add documentation for terraform overrides * make terraform overrides being able to override any variable * Bump moment from 2.29.1 to 2.29.2 in /tests_e2e (#1241) Bumps [moment](https://github.com/moment/moment) from 2.29.1 to 2.29.2. - [Release notes](https://github.com/moment/moment/releases) - [Changelog](https://github.com/moment/moment/blob/develop/CHANGELOG.md) - [Commits](moment/moment@2.29.1...2.29.2) --- updated-dependencies: - dependency-name: moment dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update cdsdashboards to 0.6.1, Voila to 0.3.5 (#1240) * Update cdsdashboards to 0.6.1 * voila v0.3.5 * Bump minimist from 1.2.5 to 1.2.6 in /tests_e2e (#1208) Bumps [minimist](https://github.com/substack/minimist) from 1.2.5 to 1.2.6. - [Release notes](https://github.com/substack/minimist/releases) - [Commits](https://github.com/substack/minimist/compare/1.2.5...1.2.6) --- updated-dependencies: - dependency-name: minimist dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * output check fix (#1244) Co-authored-by: Adam-D-Lewis <> * Add auth to argo * add argo_workflows value to qhub init command * update black version * update black in setup.cfg Co-authored-by: Dan Lester <[email protected]> Co-authored-by: imgbot[bot] <31301654+imgbot[bot]@users.noreply.github.com> Co-authored-by: ImgBotApp <[email protected]> Co-authored-by: Christopher Ostrouchov <[email protected]> Co-authored-by: Amit Kumar <[email protected]> Co-authored-by: Shannon <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Adam Lewis <[email protected]> Co-authored-by: Adam-D-Lewis <> * remove comments * Clean up argo auth * Fix typo, add node-selector * Fix, add argo to vocab * fix bug and change argo health endpoint * add overrides to argo workflows * add some argo roles stuff * fix cluster role names * add overrides * clean up * Update readme Co-authored-by: Adam-D-Lewis <> Co-authored-by: eskild <[email protected]> Co-authored-by: Dan Lester <[email protected]> Co-authored-by: imgbot[bot] <31301654+imgbot[bot]@users.noreply.github.com> Co-authored-by: ImgBotApp <[email protected]> Co-authored-by: Christopher Ostrouchov <[email protected]> Co-authored-by: Amit Kumar <[email protected]> Co-authored-by: Shannon <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: iameskild <[email protected]>
nebari-dev · May 26, 2022 · 2687f13 · 2687f13
1 parent d0cc266
commit 2687f13
Show file tree

Hide file tree

Showing 14 changed files with 543 additions and 2 deletions.
diff --git a/README.md b/README.md
@@ -7,7 +7,7 @@
 | :---------- | :-----|
 |   Project   | [![License](https://img.shields.io/badge/License-BSD%203--Clause-gray.svg?colorA=2D2A56&colorB=5936D9&style=flat.svg)](https://opensource.org/licenses/BSD-3-Clause) [![QHUb documentation](https://img.shields.io/badge/%F0%9F%93%96%20Read-the%20docs-gray.svg?colorA=2D2A56&colorB=5936D9&style=flat.svg)](https://docs.qhub.dev/en/stable/) [![PyPI version](https://badge.fury.io/py/qhub.svg)](https://badge.fury.io/py/qhub) |
 |  Community  | [![GH discussions](https://img.shields.io/badge/%F0%9F%92%AC%20-Participate%20in%20discussions-gray.svg?colorA=2D2A56&colorB=5936D9&style=flat.svg)](https://github.com/Quansight/qhub/discussions) [![Open an issue](https://img.shields.io/badge/%F0%9F%93%9D%20Open-an%20issue-gray.svg?colorA=2D2A56&colorB=5936D9&style=flat.svg)](https://github.com/Quansight/qhub/issues/new/choose) |
-|     CI      | [![Build Docker Images](https://github.com/Quansight/qhub/actions/workflows/image.yaml/badge.svg)](https://github.com/Quansight/qhub/actions/workflows/image.yaml) [![Kubernetes Tests](https://github.com/Quansight/qhub/actions/workflows/kubernetes_test.yaml/badge.svg)](https://github.com/Quansight/qhub/actions/workflows/kubernetes_test.yaml) [![Tests](https://github.com/Quansight/qhub/actions/workflows/test.yaml/badge.svg)](https://github.com/Quansight/qhub/actions/workflows/test.yaml) [![Documentation linter](https://github.com/Quansight/qhub/actions/workflows/docs.yml/badge.svg)](https://github.com/Quansight/qhub/actions/workflows/docs.yml) |
+|     CI      | [![Build Docker Images](https://github.com/Quansight/qhub/actions/workflows/image.yaml/badge.svg)](https://github.com/Quansight/qhub/actions/workflows/image.yaml) [![Kubernetes Tests](https://github.com/Quansight/qhub/actions/workflows/kubernetes_test.yaml/badge.svg)](https://github.com/Quansight/qhub/actions/workflows/kubernetes_test.yaml) [![Tests](https://github.com/Quansight/qhub/actions/workflows/test.yaml/badge.svg)](https://github.com/Quansight/qhub/actions/workflows/test.yaml) |
 
 ## Table of contents
 

diff --git a/docs/source/admin_guide/argo-workflows.md b/docs/source/admin_guide/argo-workflows.md
@@ -0,0 +1,40 @@
+# Argo Workflows
+
+Argo Workflows is an open source container-native workflow engine for orchestrating parallel jobs on Kubernetes. Argo workflows comes enabled by default with Qhub deployments.
+
+## Accessing Argo Server
+
+If Argo Workflows is enabled, users can access argo workflows server at: `your-qhub-domain.com/argo`. Log in via Keycloak with your usual credentials.
+
+Refer to the [Argo documentation](https://argoproj.github.io/argo-workflows/) for further details on Argo Workflows.
+
+## Submitting a workflow via Argo Server
+
+You can submit a workflow by clicking "SUBMIT NEW WORKFLOW" on the landing page.
+
+![See Argo Server Landing Page](../images/argo-server-landing-page.png)
+
+## Overrides of Argo Workflows Helm Chart values
+
+Argo Workflows is deployed using Argo Workflows Helm Chart version 0.13.1. The values.yaml for the helm chart can be overridden as needed via the overrides flag. The default values
+file can be found [here](https://github.com/argoproj/argo-helm/blob/argo-workflows-0.13.1/charts/argo-workflows/values.yaml). For example, the following could be done to add
+additional environment variables to the controller container.
+
+```yaml
+argo_workflows:
+  enabled: true
+  overrides:
+    controller:
+      extraEnv:
+        - name: foo
+          value: bar
+```
+
+## Disabling Argo Workflows
+
+To turn off the cluster monitoring on QHub deployments, simply turn off the feature flag within your `qhub-config.yaml` file. For example:
+
+```yaml
+argo_workflows:
+  enabled: false
+```
diff --git a/docs/source/images/argo-server-landing-page.png b/docs/source/images/argo-server-landing-page.png
diff --git a/qhub/initialize.py b/qhub/initialize.py
@@ -59,6 +59,9 @@
     "monitoring": {
         "enabled": True,
     },
+    "argo_workflows": {
+        "enabled": True,
+    },
     "kbatch": {
         "enabled": True,
     },

diff --git a/qhub/schema.py b/qhub/schema.py
@@ -76,6 +76,14 @@ class HelmExtension(Base):
     overrides: typing.Optional[typing.Dict]
 
 
+# ============== Argo-Workflows =========
+
+
+class ArgoWorkflows(Base):
+    enabled: bool
+    overrides: typing.Optional[typing.Dict]
+
+
 # ============== kbatch =============
 
 
@@ -487,6 +495,7 @@ class Main(Base):
     theme: Theme
     profiles: Profiles
     environments: typing.Dict[str, CondaEnvironment]
+    argo_workflows: typing.Optional[ArgoWorkflows]
     kbatch: typing.Optional[KBatch]
     monitoring: typing.Optional[Monitoring]
     clearml: typing.Optional[ClearML]

diff --git a/qhub/stages/checks.py b/qhub/stages/checks.py
@@ -297,7 +297,7 @@ def _attempt_connect_url(
         url, verify=False, num_attempts=NUM_ATTEMPTS, timeout=TIMEOUT
     ):
         for i in range(num_attempts):
-            response = requests.get(service_url, verify=verify, timeout=timeout)
+            response = requests.get(url, verify=verify, timeout=timeout)
             if response.status_code < 400:
                 print(f"Attempt {i+1} health check succeeded for url={url}")
                 return True

diff --git a/qhub/stages/input_vars.py b/qhub/stages/input_vars.py
@@ -284,6 +284,12 @@ def stage_07_kubernetes_services(stage_outputs, config):
         "dask-gateway-profiles": config["profiles"]["dask_worker"],
         # monitoring
         "monitoring-enabled": config["monitoring"]["enabled"],
+        # argo-worfklows
+        "argo-workflows-enabled": config["argo_workflows"]["enabled"],
+        "argo-workflows-overrides": [
+            json.dumps(config.get("argo_workflows", {}).get("overrides", {}))
+        ],
+        # kbatch
         "kbatch-enabled": config["kbatch"]["enabled"],
         # prefect
         "prefect-enabled": config.get("prefect", {}).get("enabled", False),

diff --git a/qhub/template/stages/07-kubernetes-services/argo-workflows.tf b/qhub/template/stages/07-kubernetes-services/argo-workflows.tf
@@ -0,0 +1,26 @@
+# ======================= VARIABLES ======================
+variable "argo-workflows-enabled" {
+  description = "Argo Workflows enabled"
+  type        = bool
+  default     = true
+}
+
+variable "argo-workflows-overrides" {
+  description = "Argo Workflows helm chart overrides"
+  type        = list(string)
+  default     = []
+}
+
+
+# ====================== RESOURCES =======================
+module "argo-workflows" {
+  count = var.argo-workflows-enabled ? 1 : 0
+
+  source       = "./modules/kubernetes/services/argo-workflows"
+  namespace    = var.environment
+  external-url = var.endpoint
+  realm_id     = var.realm_id
+
+  node-group = var.node_groups.general
+  overrides  = var.argo-workflows-overrides
+}
diff --git a/...template/stages/07-kubernetes-services/modules/kubernetes/services/argo-workflows/main.tf b/...template/stages/07-kubernetes-services/modules/kubernetes/services/argo-workflows/main.tf
@@ -0,0 +1,236 @@
+locals {
+  name                  = "argo-workflows"
+  argo-workflows-prefix = "argo"
+}
+
+resource "helm_release" "argo-workflows" {
+  name       = local.name
+  namespace  = var.namespace
+  repository = "https://argoproj.github.io/argo-helm"
+  chart      = "argo-workflows"
+  version    = "0.13.1"
+
+  values = concat([
+    file("${path.module}/values.yaml"),
+    # https://github.com/argoproj/argo-helm/blob/argo-workflows-0.13.1/charts/argo-workflows/values.yaml
+
+
+    jsonencode({
+      singleNamespace = true # Restrict Argo to operate only in a single namespace (the namespace of the Helm release)
+
+      controller = {
+        metricsConfig = {
+          enabled = true # enable prometheus
+        }
+        workflowNamespaces = [
+          "${var.namespace}"
+        ]
+        nodeSelector = {
+          "${var.node-group.key}" = var.node-group.value
+        }
+      }
+
+      server = {
+        # `sso` for OIDC/OAuth
+        extraArgs = ["--auth-mode=sso", "--insecure-skip-verify"]
+        # to enable TLS, `secure = true`
+        secure   = false
+        baseHref = "/${local.argo-workflows-prefix}/"
+
+        sso = {
+          insecureSkipVerify = true
+          issuer             = "https://${var.external-url}/auth/realms/${var.realm_id}"
+          clientId = {
+            name = "argo-server-sso"
+            key  = "argo-oidc-client-id"
+          }
+          clientSecret = {
+            name = "argo-server-sso"
+            key  = "argo-oidc-client-secret"
+          }
+          # The OIDC redirect URL. Should be in the form <argo-root-url>/oauth2/callback.
+          redirectUrl = "https://${var.external-url}/${local.argo-workflows-prefix}/oauth2/callback"
+          rbac = {
+            # https://argoproj.github.io/argo-workflows/argo-server-sso/#sso-rbac
+            enabled         = true
+            secretWhitelist = []
+          }
+          customGroupClaimName = "roles"
+          scopes               = ["roles"]
+        }
+        nodeSelector = {
+          "${var.node-group.key}" = var.node-group.value
+        }
+      }
+
+      containerRuntimeExecutor = "emissary"
+
+    })
+  ], var.overrides)
+}
+
+resource "kubernetes_secret" "argo-oidc-secret" {
+  metadata {
+    name      = "argo-server-sso"
+    namespace = var.namespace
+  }
+  data = {
+    "argo-oidc-client-id"     = module.argo-workflow-openid-client.config.client_id
+    "argo-oidc-client-secret" = module.argo-workflow-openid-client.config.client_secret
+  }
+}
+
+module "argo-workflow-openid-client" {
+  source = "../keycloak-client"
+
+  realm_id     = var.realm_id
+  client_id    = "argo-server-sso"
+  external-url = var.external-url
+  role_mapping = {
+    "admin"     = ["argo_admin"]
+    "developer" = ["argo_developer"]
+    "analyst"   = ["argo_viewer"]
+  }
+
+  callback-url-paths = [
+    "https://${var.external-url}/${local.argo-workflows-prefix}/oauth2/callback"
+  ]
+}
+
+resource "kubernetes_manifest" "argo-workflows-middleware-stripprefix" {
+  manifest = {
+    apiVersion = "traefik.containo.us/v1alpha1"
+    kind       = "Middleware"
+    metadata = {
+      name      = "qhub-argo-workflows-stripprefix"
+      namespace = var.namespace
+    }
+    spec = {
+      stripPrefix = {
+        prefixes = [
+          "/${local.argo-workflows-prefix}/"
+        ]
+        forceSlash = false
+      }
+    }
+  }
+}
+
+resource "kubernetes_manifest" "argo-workflows-ingress-route" {
+  manifest = {
+    apiVersion = "traefik.containo.us/v1alpha1"
+    kind       = "IngressRoute"
+    metadata = {
+      name      = "argo-workflows"
+      namespace = var.namespace
+    }
+    spec = {
+      entryPoints = ["websecure"]
+      routes = [
+        {
+          kind  = "Rule"
+          match = "Host(`${var.external-url}`) && PathPrefix(`/${local.argo-workflows-prefix}`)"
+
+          middlewares = concat(
+            [{
+              name      = kubernetes_manifest.argo-workflows-middleware-stripprefix.manifest.metadata.name
+              namespace = var.namespace
+            }]
+          )
+
+          services = [
+            {
+              name      = "${local.name}-server"
+              port      = 2746
+              namespace = var.namespace
+            }
+          ]
+        }
+      ]
+    }
+  }
+}
+
+resource "kubernetes_service_account" "argo-admin-sa" {
+  metadata {
+    name      = "argo-admin"
+    namespace = var.namespace
+    annotations = {
+      "workflows.argoproj.io/rbac-rule" : "'argo_admin' in groups"
+      "workflows.argoproj.io/rbac-rule-precedence" : "11"
+    }
+  }
+}
+
+resource "kubernetes_cluster_role_binding" "argo-admin-rb" {
+  metadata {
+    name = "argo-admin"
+  }
+
+  role_ref {
+    api_group = "rbac.authorization.k8s.io"
+    kind      = "ClusterRole"
+    name      = "argo-workflows-admin" # role deployed as part of helm chart
+  }
+  subject {
+    kind      = "ServiceAccount"
+    name      = kubernetes_service_account.argo-admin-sa.metadata.0.name
+    namespace = var.namespace
+  }
+}
+
+resource "kubernetes_service_account" "argo-edit-sa" {
+  metadata {
+    name      = "argo-edit"
+    namespace = var.namespace
+    annotations = {
+      "workflows.argoproj.io/rbac-rule" : "'argo_developer' in groups"
+      "workflows.argoproj.io/rbac-rule-precedence" : "10"
+    }
+
+  }
+}
+
+resource "kubernetes_cluster_role_binding" "argo-edit-rb" {
+  metadata {
+    name = "argo-edit"
+  }
+
+  role_ref {
+    api_group = "rbac.authorization.k8s.io"
+    kind      = "ClusterRole"
+    name      = "argo-workflows-edit" # role deployed as part of helm chart
+  }
+  subject {
+    kind      = "ServiceAccount"
+    name      = kubernetes_service_account.argo-edit-sa.metadata.0.name
+    namespace = var.namespace
+  }
+}
+resource "kubernetes_service_account" "argo-view-sa" {
+  metadata {
+    name      = "argo-view"
+    namespace = var.namespace
+    annotations = {
+      "workflows.argoproj.io/rbac-rule" : "'argo_viewer' in groups"
+      "workflows.argoproj.io/rbac-rule-precedence" : "9"
+    }
+  }
+}
+
+resource "kubernetes_cluster_role_binding" "argo-view-rb" {
+  metadata {
+    name = "argo-view"
+  }
+
+  role_ref {
+    api_group = "rbac.authorization.k8s.io"
+    kind      = "ClusterRole"
+    name      = "argo-workflows-view" # role deployed as part of helm chart
+  }
+  subject {
+    kind      = "ServiceAccount"
+    name      = kubernetes_service_account.argo-view-sa.metadata.0.name
+    namespace = var.namespace
+  }
+}
diff --git a/...late/stages/07-kubernetes-services/modules/kubernetes/services/argo-workflows/values.yaml b/...late/stages/07-kubernetes-services/modules/kubernetes/services/argo-workflows/values.yaml
@@ -0,0 +1 @@
+# https://github.com/argoproj/argo-helm/blob/argo-workflows-0.13.1/charts/argo-workflows/values.yaml
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1 @@
		# https://github.com/argoproj/argo-helm/blob/argo-workflows-0.13.1/charts/argo-workflows/values.yaml