diff --git a/README.md b/README.md index d683318..a0f9c2e 100644 --- a/README.md +++ b/README.md @@ -52,7 +52,7 @@ your shell. #### Install the Google Cloud SDK -While not used directly by `dsub` for the `google-v2` or `google-cls-v2` providers, you are likely to want to install the command line tools found in the [Google +While not used directly by `dsub` for the `google-batch` or `google-cls-v2` providers, you are likely to want to install the command line tools found in the [Google Cloud SDK](https://cloud.google.com/sdk/). If you will be using the `local` provider for faster job development, @@ -156,13 +156,13 @@ You'll get quicker turnaround times and won't incur cloud charges using it. ### Getting started on Google Cloud -`dsub` supports the use of two different APIs from Google Cloud for running -tasks. Google Cloud is transitioning from `Genomics v2alpha1` -to [Cloud Life Sciences v2beta](https://cloud.google.com/life-sciences/docs/reference/rest). +`dsub` currently supports the [Cloud Life Sciences v2beta](https://cloud.google.com/life-sciences/docs/reference/rest) +API from Google Cloud and is is developing support for the [Batch](https://cloud.google.com/batch/docs/reference/rest) +API from Google Cloud. -`dsub` supports both APIs with the (old) `google-v2` and (new) `google-cls-v2` -providers respectively. `google-v2` is the current default provider. `dsub` -will be transitioning to make `google-cls-v2` the default in coming releases. +`dsub` supports the v2beta API with the `google-cls-v2` provider. +`google-cls-v2` is the current default provider. `dsub` will be transitioning to +make `google-batch` the default in coming releases. The steps for getting started differ slightly as indicated in the steps below: @@ -171,13 +171,14 @@ The steps for getting started differ slightly as indicated in the steps below: 1. Enable the APIs: - - For the `v2alpha1` API (provider: `google-v2`): + - For the `v2beta` API (provider: `google-cls-v2`): - [Enable the Genomics, Storage, and Compute APIs](https://console.cloud.google.com/flows/enableapi?apiid=genomics,storage_component,compute_component&redirect=https://console.cloud.google.com). + [Enable the Cloud Life Sciences, Storage, and Compute APIs](https://console.cloud.google.com/flows/enableapi?apiid=lifesciences.googleapis.com,storage.googleapis.com,compute.googleapis.com&redirect=https://console.cloud.google.com) - - For the `v2beta` API (provider: `google-cls-v2`): + - For the `batch` API (provider: `google-batch`): + + [Enable the Batch, Storage, and Compute APIs](https://console.cloud.google.com/flows/enableapi?apiid=batch.googleapis.com,storage.googleapis.com,compute.googleapis.com&redirect=https://console.cloud.google.com). - [Enable the Cloud Life Sciences, Storage, and Compute APIs](https://console.cloud.google.com/flows/enableapi?apiid=lifesciences.googleapis.com,storage_component,compute_component&redirect=https://console.cloud.google.com) 1. Provide [credentials](https://developers.google.com/identity/protocols/application-default-credentials) so `dsub` can call Google APIs: @@ -202,10 +203,10 @@ The steps for getting started differ slightly as indicated in the steps below: 1. Run a very simple "Hello World" `dsub` job and wait for completion. - - For the `v2alpha1` API (provider: `google-v2`): + - For the `v2beta` API (provider: `google-cls-v2`): dsub \ - --provider google-v2 \ + --provider google-cls-v2 \ --project my-cloud-project \ --regions us-central1 \ --logging gs://my-bucket/logging/ \ @@ -216,10 +217,10 @@ The steps for getting started differ slightly as indicated in the steps below: Change `my-cloud-project` to your Google Cloud project, and `my-bucket` to the bucket you created above. - - For the `v2beta` API (provider: `google-cls-v2`): + - For the `batch` API (provider: `google-batch`): dsub \ - --provider google-cls-v2 \ + --provider google-batch \ --project my-cloud-project \ --regions us-central1 \ --logging gs://my-bucket/logging/ \ @@ -246,14 +247,13 @@ To this end, `dsub` provides multiple "backend providers", each of which implements a consistent runtime environment. The current providers are: - local -- google-v2 (the default) -- google-cls-v2 +- google-cls-v2(the default) - google-batch (*new*) More details on the runtime environment implemented by the backend providers can be found in [dsub backend providers](https://github.com/DataBiosphere/dsub/blob/main/docs/providers/README.md). -### Differences between `google-v2`, `google-cls-v2` and `google-batch` +### Differences between `google-cls-v2` and `google-batch` The `google-cls-v2` provider is built on the Cloud Life Sciences `v2beta` API. This API is very similar to its predecessor, the Genomics `v2alpha1` API. @@ -265,29 +265,15 @@ Details of Cloud Life Sciences versus Batch can be found in this [Migration Guide](https://cloud.google.com/batch/docs/migrate-to-batch-from-cloud-life-sciences). `dsub` largely hides the differences between the APIs, but there are a -few difference to note: - -- `v2beta` and Cloud Batch are regional services, `v2alpha1` is a global service - -What this means is that with `v2alpha1`, the metadata about your tasks -(called "operations"), is stored in a global database, while with `v2beta` and -Cloud Batch, the metadata about your tasks are stored in a regional database. If -your operation/job information needs to stay in a particular region, use the -`v2beta` or Batch API (the `google-cls-v2` or `google-batch` provider), and -specify the `--location` where your operation/job information should be stored. +few differences to note: -- The `--regions` and `--zones` flags can be omitted when using `google-cls-v2` and `google-batch` +- `google-batch` requires jobs to run in one region The `--regions` and `--zones` flags for `dsub` specify where the tasks should -run. More specifically, this specifies what Compute Engine Zones to use for -the VMs that run your tasks. - -With the `google-v2` provider, there is no default region or zone, and thus -one of the `--regions` or `--zones` flags is required. - -With `google-cls-v2` and `google-batch`, the `--location` flag defaults to -`us-central1`, and if the `--regions` and `--zones` flags are omitted, the -`location` will be used as the default `regions` list. +run. The `google-cls-v2` allows you to specify a multi-region like `US`, +multiple regions, or multiple zones across regions. With the `google-batch` +provider, you must specify either one region or multiple zones within a single +region. ## `dsub` features @@ -463,7 +449,7 @@ mounting read-only: [Compute Engine Image](https://cloud.google.com/compute/docs/images) that you pre-create. -The `google-v2` and `google-cls-v2` providers support these methods of +The `google-cls-v2` and `google-batch` provider support these methods of providing access to resource data. The `local` provider supports mounting a @@ -471,7 +457,7 @@ local directory in a similar fashion to support your local development. ##### Mounting a Google Cloud Storage bucket -To have the `google-v2`, `google-cls-v2`, or `google-batch` provider mount a +To have the `google-cls-v2` or `google-batch` provider mount a Cloud Storage bucket using [Cloud Storage FUSE](https://cloud.google.com/storage/docs/gcs-fuse), use the `--mount` command line flag: @@ -488,7 +474,7 @@ before using Cloud Storage FUSE. ##### Mounting an existing peristent disk -To have the `google-v2` or `google-cls-v2` provider mount a persistent disk that +To have the `google-cls-v2` or `google-batch` provider mount a persistent disk that you have pre-created and populated, use the `--mount` command line flag and the url of the source disk: @@ -496,7 +482,7 @@ url of the source disk: ##### Mounting a persistent disk, created from an image -To have the `google-v2` or `google-cls-v2` provider mount a persistent disk created from an image, +To have the `google-cls-v2` or `google-batch` provider mount a persistent disk created from an image, use the `--mount` command line flag and the url of the source image and the size (in GB) of the disk: @@ -527,7 +513,7 @@ path using the environment variable. `dsub` tasks run using the `local` provider will use the resources available on your local machine. -`dsub` tasks run using the `google`, `google-v2`, or `google-cls-v2` providers can take advantage +`dsub` tasks run using the `google-cls-v2` or `google-batch` providers can take advantage of a wide range of CPU, RAM, disk, and hardware accelerator (eg. GPU) options. See the [Compute Resources](https://github.com/DataBiosphere/dsub/blob/main/docs/compute_resources.md) @@ -634,14 +620,14 @@ For more details, see [Checking Status and Troubleshooting Jobs](https://github. The `dstat` command displays the status of jobs: - dstat --provider google-v2 --project my-cloud-project + dstat --provider google-cls-v2 --project my-cloud-project With no additional arguments, dstat will display a list of *running* jobs for the current `USER`. To display the status of a specific job, use the `--jobs` flag: - dstat --provider google-v2 --project my-cloud-project --jobs job-id + dstat --provider google-cls-v2 --project my-cloud-project --jobs job-id For a batch job, the output will list all *running* tasks. @@ -673,7 +659,7 @@ By default, dstat outputs one line per task. If you're using a batch job with many tasks then you may benefit from `--summary`. ``` -$ dstat --provider google-v2 --project my-project --status '*' --summary +$ dstat --provider google-cls-v2 --project my-project --status '*' --summary Job Name Status Task Count ------------- ------------- ------------- @@ -694,25 +680,25 @@ Use the `--users` flag to specify other users, or `'*'` for all users. To delete a running job: - ddel --provider google-v2 --project my-cloud-project --jobs job-id + ddel --provider google-cls-v2 --project my-cloud-project --jobs job-id If the job is a batch job, all running tasks will be deleted. To delete specific tasks: ddel \ - --provider google-v2 \ + --provider google-cls-v2 \ --project my-cloud-project \ --jobs job-id \ --tasks task-id1 task-id2 To delete all running jobs for the current user: - ddel --provider google-v2 --project my-cloud-project --jobs '*' + ddel --provider google-cls-v2 --project my-cloud-project --jobs '*' ## Service Accounts and Scope (Google providers only) -When you run the `dsub` command with the `google-v2` or `google-cls-v2` +When you run the `dsub` command with the `google-cls-v2` or `google-batch` provider, there are two different sets of credentials to consider: - Account submitting the `pipelines.run()` request to run your command/script on a VM diff --git a/docs/job_control.md b/docs/job_control.md index 6391a59..229cfea 100644 --- a/docs/job_control.md +++ b/docs/job_control.md @@ -61,22 +61,22 @@ dsub ... --after "${JOB_A}" "${JOB_B}" Here is the output of a sample run: ``` -$ JOBID_A=$(dsub --provider google-v2 --project "${MYPROJECT}" --regions us-central1 \ +$ JOBID_A=$(dsub --provider google-cls-v2 --project "${MYPROJECT}" --regions us-central1 \ --logging "gs://${MYBUCKET}/logging/" \ --command 'echo "hello from job A"') Job: echo----180924-112256-64 Launched job-id: echo----180924-112256-64 To check the status, run: - dstat --provider google-v2 --project ${MYPROJECT} --jobs 'echo----180924-112256-64' --status '*' + dstat --provider google-cls-v2 --project ${MYPROJECT} --jobs 'echo----180924-112256-64' --status '*' To cancel the job, run: - ddel --provider google-v2 --project ${MYPROJECT} --jobs 'echo----180924-112256-64' + ddel --provider google-cls-v2 --project ${MYPROJECT} --jobs 'echo----180924-112256-64' $ echo "${JOBID_A}" echo----180924-112256-64 $ JOBID_B=... (similar) -$ JOBID_C=$(dsub --provider google-v2 --project "${MYPROJECT}" --regions us-central1 \ +$ JOBID_C=$(dsub --provider google-cls-v2 --project "${MYPROJECT}" --regions us-central1 \ --logging "gs://${MYBUCKET}/logging/" \ --command 'echo "job C"' --after "${JOBID_A}" "${JOBID_B}") Waiting for predecessor jobs to complete... @@ -86,9 +86,9 @@ Waiting for: echo----180924-112259-48. echo----180924-112259-48: SUCCESS Launched job-id: echo----180924-112302-87 To check the status, run: - dstat --provider google-v2 --project ${MYPROJECT} --jobs 'echo----180924-112302-87' --status '*' + dstat --provider google-cls-v2 --project ${MYPROJECT} --jobs 'echo----180924-112302-87' --status '*' To cancel the job, run: - ddel --provider google-v2 --project ${MYPROJECT} --jobs 'echo----180924-112302-87' + ddel --provider google-cls-v2 --project ${MYPROJECT} --jobs 'echo----180924-112302-87' echo----180924-112302-87 ``` diff --git a/docs/providers/README.md b/docs/providers/README.md index 711dd66..0d55b5d 100644 --- a/docs/providers/README.md +++ b/docs/providers/README.md @@ -10,8 +10,8 @@ implements a consistent runtime environment. The current supported providers are: - local -- google-v2 (the default) -- google-cls-v2 (*new*) +- google-cls-v2 (the default) +- google-batch (*new*) ## Runtime environment @@ -194,13 +194,13 @@ During execution, `runner.sh` writes the following files to record task state: The `local` provider does not support resource-related flags such as `--min-cpu`, `--min-ram`, `--boot-disk-size`, or `--disk-size`. -### `google-v2` and `google-cls-v2` providers +### `google-cls-v2` and `google-batch` providers -The `google-v2` and `google-cls-v2` providers share a significant amount of -their implementation. The `google-v2` provider utilizes the Google Genomics -Pipelines API `v2alpha1` -while the `google-cls-v2` provider utilizes the Google Cloud Life Sciences +The `google-cls-v2` and `google-batch` providers share a significant amount of +their implementation. The `google-cls-v2` provider utilizes the Google Cloud Life Sciences Piplines API [v2beta](https://cloud.google.com/life-sciences/docs/apis) +while the `google-batch` provider utilizes the Google Cloud +[Batch API](https://cloud.google.com/batch/docs/reference/rest) to queue a request for the following sequence of events: 1. Create a Google Compute Engine @@ -282,7 +282,7 @@ its status is `RUNNING`. #### Logging -The `google-v2` provider saves 3 log files to Cloud Storage, every 5 minutes +The `google-cls-v2` and `google-batch` provider saves 3 log files to Cloud Storage, every 5 minutes to the `--logging` location specified to `dsub`: - `[prefix].log`: log generated by all containers running on the VM @@ -293,7 +293,7 @@ Logging paths and the `[prefix]` are discussed further in [Logging](../logging.m #### Resource requirements -The `google-v2` and `google-cls-v2` providers support many resource-related +The `google-cls-v2` and `google-batch` providers support many resource-related flags to configure the Compute Engine VMs that tasks run on, such as `--machine-type` or `--min-cores` and `--min-ram`, as well as `--boot-disk-size` and `--disk-size`. Additional provider-specific parameters are available @@ -311,12 +311,12 @@ large Docker images are used, as such images need to be pulled to the boot disk. #### Provider specific parameters -The following `dsub` parameters are specific to the `google-v2` and -`google-cls-v2` providers: +The following `dsub` parameters are specific to the `google-cls-v2` and +`google-batch` providers: * [Location resources](https://cloud.google.com/about/locations) - - `--location` (`google-cls-v2` only): + - `--location`: - Specifies the Google Cloud region to which the pipeline request will be sent and where operation metadata will be stored. The associated dsub task may be executed in another region if the `--regions` or `--zones` diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index cd74974..cf59c20 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -20,7 +20,7 @@ When submitted with no filter arguments, `dstat` shows information for all tasks in the `RUNNING` state belonging to the current user: ``` -$ dstat --provider google-v2 --project my-project +$ dstat --provider google-cls-v2 --project my-project Job Name Task Status Last Update -------------- ------ ---------------- ------------------- my-job-name task-3 localizing-files 2017-04-06 16:03:34 @@ -44,7 +44,7 @@ them separately. To check on a specific job, pass the `--jobs` (or `-j`) flag. For example: ``` -$ dstat --provider google-v2 --project my-project --jobs my-job-id +$ dstat --provider google-cls-v2 --project my-project --jobs my-job-id Job Name Status Last Update -------------- -------- ------------------- my-job-name Pending 2017-04-11 16:05:35 @@ -59,7 +59,7 @@ value `*` to `dstat`: ``` $ dstat \ - --provider google-v2 \ + --provider google-cls-v2 \ --project my-project \ --jobs my-job-id \ --status '*' @@ -75,7 +75,7 @@ Be sure to quote the `*` to prevent shell expansion. To view results for all jobs associated with your user id: ``` -dstat --provider google-v2 --project my-project --status '*' +dstat --provider google-cls-v2 --project my-project --status '*' ``` ### Check jobs with my own labels @@ -176,7 +176,7 @@ use the `--age` flag. For example, the following command will return all jobs started in the last day: ``` -./dstat --provider google-v2 --project my-project --status '*' --age 1d +./dstat --provider google-cls-v2 --project my-project --status '*' --age 1d ``` The `--age` flags supports the following types of values: @@ -218,7 +218,7 @@ The following examples shows minute-by-minute progression of 3 tasks ``` $ dstat \ - --provider google-v2 \ + --provider google-cls-v2 \ --project my-project \ --jobs my-job-id \ --wait --poll-interval 60 @@ -281,7 +281,7 @@ maintain consistency between dsub versions. ``` $ dstat \ - --provider google-v2 \ + --provider google-cls-v2 \ --project my-project \ --jobs my-job-id \ --full @@ -306,7 +306,7 @@ Note the `Internal ID` in this example provides the ``` $ dstat \ - --provider google-v2 \ + --provider google-cls-v2 \ --project my-project \ --jobs my-job-id \ --format text \ diff --git a/dsub/_dsub_version.py b/dsub/_dsub_version.py index 2e35fe2..67f5361 100644 --- a/dsub/_dsub_version.py +++ b/dsub/_dsub_version.py @@ -26,4 +26,4 @@ 0.1.3.dev0 -> 0.1.3 -> 0.1.4.dev0 -> ... """ -DSUB_VERSION = '0.4.13' +DSUB_VERSION = '0.5.0' diff --git a/dsub/commands/ddel.py b/dsub/commands/ddel.py index 581780f..748e91f 100755 --- a/dsub/commands/ddel.py +++ b/dsub/commands/ddel.py @@ -78,29 +78,26 @@ def _parse_arguments(): help='User labels to match. Tasks returned must match all labels.', metavar='KEY=VALUE') - # Shared between the "google-cls-v2" and "google-v2" providers + # Shared between the "google-cls-v2" and "google-batch" providers google_common = parser.add_argument_group( title='google-common', - description="""Options common to the "google", "google-cls-v2", and - "google-v2" providers""") + description="""Options common to the "google-cls-v2" and + "google-batch" providers""", + ) google_common.add_argument( '--project', help='Cloud project ID in which to find and delete the job(s)') - - google_cls_v2 = parser.add_argument_group( - title='"google-cls-v2" provider options', - description='See also the "google-common" options listed') - google_cls_v2.add_argument( + google_common.add_argument( '--location', default=job_model.DEFAULT_LOCATION, help="""Specifies the Google Cloud region to which the dsub job was - submitted. (default: {})""".format(job_model.DEFAULT_LOCATION)) + submitted. (default: {})""".format(job_model.DEFAULT_LOCATION), + ) return provider_base.parse_args( parser, { 'google-batch': ['project'], 'google-cls-v2': ['project'], - 'google-v2': ['project'], 'test-fails': [], 'local': [], }, sys.argv[1:]) diff --git a/dsub/commands/dstat.py b/dsub/commands/dstat.py index 642c0a6..d13f2ed 100755 --- a/dsub/commands/dstat.py +++ b/dsub/commands/dstat.py @@ -136,28 +136,25 @@ def _parse_arguments(): action='store_true', help='Display a summary of the results, grouped by (job, status).') - # Shared between the "google-cls-v2" and "google-v2" providers + # Shared between the "google-cls-v2" and "google-batch" providers google_common = parser.add_argument_group( title='google-common', - description="""Options common to the "google", "google-cls-v2", and - "google-v2" providers""") + description="""Options common to the "google-cls-v2" and + "google-batch" providers""", + ) google_common.add_argument( '--project', help='Cloud project ID in which to find and the job(s)') - - google_cls_v2 = parser.add_argument_group( - title='"google-cls-v2" provider options', - description='See also the "google-common" options listed') - google_cls_v2.add_argument( + google_common.add_argument( '--location', default=job_model.DEFAULT_LOCATION, help="""Specifies the Google Cloud region to which the dsub job was - submitted. (default: {})""".format(job_model.DEFAULT_LOCATION)) + submitted. (default: {})""".format(job_model.DEFAULT_LOCATION), + ) return provider_base.parse_args( parser, { 'google-batch': ['project'], 'google-cls-v2': ['project'], - 'google-v2': ['project'], 'test-fails': [], 'local': [], }, sys.argv[1:]) diff --git a/dsub/commands/dsub.py b/dsub/commands/dsub.py index d9f3c53..73d1973 100644 --- a/dsub/commands/dsub.py +++ b/dsub/commands/dsub.py @@ -220,19 +220,6 @@ def _google_cls_v2_parse_arguments(args): _check_nvidia_driver_version(args) -def _google_v2_parse_arguments(args): - """Validated google-v2 arguments.""" - if (args.zones and args.regions) or (not args.zones and not args.regions): - raise ValueError('Exactly one of --regions and --zones must be specified') - - if args.machine_type and (args.min_cores or args.min_ram): - raise ValueError( - '--machine-type not supported together with --min-cores or --min-ram.') - - _check_private_address(args) - _check_nvidia_driver_version(args) - - def _local_parse_arguments(args): """Validated local arguments.""" if args.user and args.user != dsub_util.get_os_user(): @@ -420,16 +407,18 @@ def _parse_arguments(prog, argv): '--min-cores', type=int, help="""Minimum CPU cores for each job. The default is provider-specific. - The google-v2 provider default is 1 core. + The google-cls-v2 provider default is 1 core. The local provider does not allocate resources, but uses available - resources of your machine.""") + resources of your machine.""", + ) parser.add_argument( '--min-ram', type=float, help="""Minimum RAM per job in GB. The default is provider-specific. - The google-v2 provider default is 3.75 GB. + The google-cls-v2 provider default is 3.75 GB. The local provider does not allocate resources, but uses available - resources of your machine.""") + resources of your machine.""", + ) parser.add_argument( '--disk-size', default=job_model.DEFAULT_DISK_SIZE, @@ -444,11 +433,12 @@ def _parse_arguments(prog, argv): # Add provider-specific arguments - # Shared between the "google-cls-v2" and "google-v2" providers + # Shared between the "google-cls-v2" and "google-batch" providers google_common = parser.add_argument_group( title='google-common', - description="""Options common to the "google-cls-v2", "google-v2" and - "google-batch" providers""") + description="""Options common to the "google-cls-v2" and + "google-batch" providers""", + ) google_common.add_argument( '--project', help='Cloud project ID in which to run the job') google_common.add_argument( @@ -582,32 +572,27 @@ def _parse_arguments(prog, argv): help="""If set to true, prevents the container for the user's script/command from accessing the external network. (default: False)""") - - google_cls_v2 = parser.add_argument_group( - title='"google-cls-v2" provider options', - description='See also the "google-common" options listed above') - google_cls_v2.add_argument( + google_common.add_argument( '--location', default=job_model.DEFAULT_LOCATION, help="""Specifies the Google Cloud region to which the pipeline request will be sent and where operation metadata will be stored. The associated dsub task may be executed in another region if the --regions or --zones arguments are specified. (default: {})""".format( - job_model.DEFAULT_LOCATION)) + job_model.DEFAULT_LOCATION + ), + ) args = provider_base.parse_args( parser, { 'google-batch': ['project', 'logging'], 'google-cls-v2': ['project', 'logging'], - 'google-v2': ['project', 'logging'], 'test-fails': [], 'local': ['logging'], }, argv) if args.provider == 'google-cls-v2': _google_cls_v2_parse_arguments(args) - if args.provider == 'google-v2': - _google_v2_parse_arguments(args) return args diff --git a/dsub/providers/google_utils.py b/dsub/providers/google_utils.py index 75d82c1..5b4f64d 100644 --- a/dsub/providers/google_utils.py +++ b/dsub/providers/google_utils.py @@ -13,7 +13,7 @@ # limitations under the License. """Utility functions to be used by the Google providers. -This module holds constants and methods useful to google-v2, google-cls-v2, +This module holds constants and methods useful to google-cls-v2 and google-batch providers. """ import os diff --git a/dsub/providers/google_v2.py b/dsub/providers/google_v2.py deleted file mode 100644 index 2a20f7e..0000000 --- a/dsub/providers/google_v2.py +++ /dev/null @@ -1,62 +0,0 @@ -# Copyright 2019 Verily Life Sciences Inc. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -"""Provider for running jobs on Google Cloud Platform. - -This module implements job creation, listing, and canceling using the -Google Genomics Pipelines and Operations APIs v2alpha1. -""" - -from . import google_v2_base -from . import google_v2_versions - -_PROVIDER_NAME = 'google-v2' - - -class GoogleV2JobProvider(google_v2_base.GoogleV2JobProviderBase): - """dsub provider implementation managing Jobs on Google Cloud.""" - - def __init__(self, dry_run, project, credentials=None): - super(GoogleV2JobProvider, - self).__init__(_PROVIDER_NAME, google_v2_versions.V2ALPHA1, - credentials, project, dry_run) - - def _get_pipeline_regions(self, regions, zones): - """Returns the list of regions to use for a pipeline request. - - For v2alpha1, just return the regions value that was set, even if it is - empty. - - Args: - regions (str): A space separated list of regions to use for the pipeline. - zones (str): A space separated list of zones to use for the pipeline. - """ - return regions or [] - - def _pipelines_run_api(self, request): - return self._service.pipelines().run(body=request) - - def _operations_list_api(self, ops_filter, page_token, page_size): - name = 'projects/{}/operations'.format(self._project) - return self._service.projects().operations().list( - name=name, filter=ops_filter, pageToken=page_token, pageSize=page_size) - - def _operations_cancel_api_def(self): - return self._service.projects().operations().cancel - - def _batch_handler_def(self): - return google_v2_base.GoogleV2BatchHandler - - -if __name__ == '__main__': - pass diff --git a/dsub/providers/google_v2_base.py b/dsub/providers/google_v2_base.py index 75a32aa..4147653 100644 --- a/dsub/providers/google_v2_base.py +++ b/dsub/providers/google_v2_base.py @@ -13,9 +13,9 @@ # limitations under the License. """Provider for running jobs on Google Cloud Platform. -This module serves as the base class for the google-v2 and google-cls-v2 -providers. The APIs they are based on are very similar and can benefit from -sharing code. +This module serves as the base class for the google-cls-v2 provider. It also +used to be the base class for the now gone google-v2 provider. +The APIs they were based on were very similar and benefited from sharing code. """ import ast import json diff --git a/dsub/providers/google_v2_operations.py b/dsub/providers/google_v2_operations.py index 429125a..66fa2ae 100644 --- a/dsub/providers/google_v2_operations.py +++ b/dsub/providers/google_v2_operations.py @@ -19,7 +19,7 @@ def set_api_version(api_version): - assert api_version in (google_v2_versions.V2ALPHA1, google_v2_versions.V2BETA) + assert api_version in (google_v2_versions.V2BETA) global _API_VERSION _API_VERSION = api_version @@ -103,10 +103,7 @@ def get_action_by_name(op, name): def get_action_name(action): - if _API_VERSION == google_v2_versions.V2ALPHA1: - return action.get('name') - - elif _API_VERSION == google_v2_versions.V2BETA: + if _API_VERSION == google_v2_versions.V2BETA: return action.get('containerName') else: @@ -144,11 +141,7 @@ def external_network_blocked(op): """Retun True if the blockExternalNetwork flag is set for the user action.""" user_action = get_action_by_name(op, 'user-command') if user_action: - if _API_VERSION == google_v2_versions.V2ALPHA1: - flags = user_action.get('flags') - if flags: - return 'BLOCK_EXTERNAL_NETWORK' in flags - elif _API_VERSION == google_v2_versions.V2BETA: + if _API_VERSION == google_v2_versions.V2BETA: return user_action.get('blockExternalNetwork') else: assert False, 'Unexpected version: {}'.format(_API_VERSION) @@ -157,13 +150,7 @@ def external_network_blocked(op): def is_unexpected_exit_status_event(e): """Retun True if the event is for an unexpected exit status.""" - if _API_VERSION == google_v2_versions.V2ALPHA1: - - return e.get('details', {}).get( - '@type' - ) == 'type.googleapis.com/google.genomics.v2alpha1.UnexpectedExitStatusEvent' - - elif _API_VERSION == google_v2_versions.V2BETA: + if _API_VERSION == google_v2_versions.V2BETA: return 'unexpectedExitStatus' in e @@ -173,12 +160,7 @@ def is_unexpected_exit_status_event(e): def is_failed_event(e): """Retun True if the event is an operation failed event.""" - if _API_VERSION == google_v2_versions.V2ALPHA1: - - return e.get('details', {}).get( - '@type') == 'type.googleapis.com/google.genomics.v2alpha1.FailedEvent' - - elif _API_VERSION == google_v2_versions.V2BETA: + if _API_VERSION == google_v2_versions.V2BETA: return 'failed' in e @@ -188,13 +170,7 @@ def is_failed_event(e): def is_container_stopped_event(e): """Retun True if the event is a container stopped event.""" - if _API_VERSION == google_v2_versions.V2ALPHA1: - - return e.get('details', {}).get( - '@type' - ) == 'type.googleapis.com/google.genomics.v2alpha1.ContainerStoppedEvent' - - elif _API_VERSION == google_v2_versions.V2BETA: + if _API_VERSION == google_v2_versions.V2BETA: return 'containerStopped' in e @@ -204,12 +180,7 @@ def is_container_stopped_event(e): def is_worker_assigned_event(event): """Return True if the event is a "Worker assigned..." event.""" - if _API_VERSION == google_v2_versions.V2ALPHA1: - return event.get('details', {}).get( - '@type' - ) == 'type.googleapis.com/google.genomics.v2alpha1.WorkerAssignedEvent' - - elif _API_VERSION == google_v2_versions.V2BETA: + if _API_VERSION == google_v2_versions.V2BETA: return 'workerAssigned' in event else: @@ -218,12 +189,7 @@ def is_worker_assigned_event(event): def is_pull_started_event(event): """Return True if the event is a "Started Pulling..." event.""" - if _API_VERSION == google_v2_versions.V2ALPHA1: - return event.get('details', {}).get( - '@type' - ) == 'type.googleapis.com/google.genomics.v2alpha1.PullStartedEvent' - - elif _API_VERSION == google_v2_versions.V2BETA: + if _API_VERSION == google_v2_versions.V2BETA: return 'pullStarted' in event else: @@ -245,14 +211,7 @@ def get_container_stopped_error_events(op): if not events: return None - if _API_VERSION == google_v2_versions.V2ALPHA1: - - return [ - e for e in events if is_container_stopped_event(e) and - e.get('details', {}).get('exitStatus', 0) != 0 - ] - - elif _API_VERSION == google_v2_versions.V2BETA: + if _API_VERSION == google_v2_versions.V2BETA: return [ e for e in events if is_container_stopped_event(e) and @@ -284,11 +243,7 @@ def get_worker_assigned_events(op): def get_event_action_id(e): """Return the actionId associated with the specified event.""" - if _API_VERSION == google_v2_versions.V2ALPHA1: - - return e.get('details', {}).get('actionId') - - elif _API_VERSION == google_v2_versions.V2BETA: + if _API_VERSION == google_v2_versions.V2BETA: for event_type in ['containerStopped', 'unexpectedExitStatus']: if event_type in e: @@ -305,11 +260,7 @@ def get_event_description(e): def get_event_stderr(e): """Return the stderr field (if any) associated with the event.""" - if _API_VERSION == google_v2_versions.V2ALPHA1: - - return e.get('details', {}).get('stderr') - - elif _API_VERSION == google_v2_versions.V2BETA: + if _API_VERSION == google_v2_versions.V2BETA: for event_type in ['containerStopped']: if event_type in e: @@ -326,10 +277,7 @@ def get_worker_assigned_event_details(op): if not events: return None - if _API_VERSION == google_v2_versions.V2ALPHA1: - return events[0].get('details', {}) - - elif _API_VERSION == google_v2_versions.V2BETA: + if _API_VERSION == google_v2_versions.V2BETA: return events[0].get('workerAssigned', {}) else: @@ -359,10 +307,7 @@ def get_resources(op): def get_vm_network_name(vm): """Return the name of the network from the virtualMachine.""" - if _API_VERSION == google_v2_versions.V2ALPHA1: - return vm.get('network', {}).get('name') - - elif _API_VERSION == google_v2_versions.V2BETA: + if _API_VERSION == google_v2_versions.V2BETA: return vm.get('network', {}).get('network') else: @@ -370,10 +315,10 @@ def get_vm_network_name(vm): def is_pipeline(op): - """Check that an operation is a genomics pipeline run. + """Check that an operation is a lifesciences pipeline run. - An operation is a Genomics Pipeline run if the request metadata's @type - is "type.googleapis.com/google.genomics.v2alpha1.Metadata". + An operation is a Lifesciences Pipeline run if the request metadata's @type + is "type.googleapis.com/google.cloud.lifesciences.v2beta.Metadata". Args: op: a pipelines operation. @@ -382,11 +327,7 @@ def is_pipeline(op): Boolean, true if the operation is a RunPipelineRequest. """ - if _API_VERSION == google_v2_versions.V2ALPHA1: - return get_metadata_type( - op) == 'type.googleapis.com/google.genomics.v2alpha1.Metadata' - - elif _API_VERSION == google_v2_versions.V2BETA: + if _API_VERSION == google_v2_versions.V2BETA: return get_metadata_type( op) == 'type.googleapis.com/google.cloud.lifesciences.v2beta.Metadata' @@ -402,7 +343,7 @@ def is_dsub_operation(op): been part of dsub operations. - labels: job-id, job-name, and user-id have always existed. The dsub-version - label has always existed for the google-v2 provider. + label has always existed since the google-v2 provider. Args: op: a pipelines operation. diff --git a/dsub/providers/google_v2_pipelines.py b/dsub/providers/google_v2_pipelines.py index c45c093..6acbee8 100644 --- a/dsub/providers/google_v2_pipelines.py +++ b/dsub/providers/google_v2_pipelines.py @@ -22,16 +22,14 @@ def set_api_version(api_version): - assert api_version in (google_v2_versions.V2ALPHA1, google_v2_versions.V2BETA) + assert api_version in (google_v2_versions.V2BETA) global _API_VERSION _API_VERSION = api_version def build_network(name, subnetwork, use_private_address): - if _API_VERSION == google_v2_versions.V2ALPHA1: - network_key = 'name' - elif _API_VERSION == google_v2_versions.V2BETA: + if _API_VERSION == google_v2_versions.V2BETA: network_key = 'network' else: assert False, 'Unexpected version: {}'.format(_API_VERSION) @@ -142,9 +140,6 @@ def build_resources(project=None, 'virtualMachine': virtual_machine, } - if _API_VERSION == google_v2_versions.V2ALPHA1: - resources['projectId'] = project - return resources @@ -214,24 +209,7 @@ def build_action(name=None, 'labels': labels, } - if _API_VERSION == google_v2_versions.V2ALPHA1: - action['name'] = name - - # In v2alpha1, the flags are passed as a list of strings - flags = [] - if always_run: - flags.append('ALWAYS_RUN') - if enable_fuse: - flags.append('ENABLE_FUSE') - if run_in_background: - flags.append('RUN_IN_BACKGROUND') - if block_external_network: - flags.append('BLOCK_EXTERNAL_NETWORK') - - if flags: - action['flags'] = flags - - elif _API_VERSION == google_v2_versions.V2BETA: + if _API_VERSION == google_v2_versions.V2BETA: action['containerName'] = name # In v2beta, the flags are direct members of the action diff --git a/dsub/providers/google_v2_versions.py b/dsub/providers/google_v2_versions.py index 3f08626..f8d1452 100644 --- a/dsub/providers/google_v2_versions.py +++ b/dsub/providers/google_v2_versions.py @@ -13,13 +13,10 @@ # limitations under the License. """Constants and methods for Google's Pipelines API v2.""" -V2ALPHA1 = 'v2alpha1' V2BETA = 'v2beta' def get_api_name(version): - if version == V2ALPHA1: - return 'genomics' if version == V2BETA: return 'lifesciences' diff --git a/dsub/providers/provider_base.py b/dsub/providers/provider_base.py index ae38334..814b2e4 100644 --- a/dsub/providers/provider_base.py +++ b/dsub/providers/provider_base.py @@ -20,13 +20,11 @@ from . import google_base from . import google_batch from . import google_cls_v2 -from . import google_v2 from . import local from . import test_fails PROVIDER_NAME_MAP = { - google_v2.GoogleV2JobProvider: 'google-v2', google_cls_v2.GoogleCLSV2JobProvider: 'google-cls-v2', google_batch.GoogleBatchJobProvider: 'google-batch', local.LocalJobProvider: 'local', @@ -52,7 +50,7 @@ def get_provider(args, resources, credentials_fn=None): if not credentials_fn: credentials_fn = credentials_from_args - provider = getattr(args, 'provider', 'google-v2') + provider = getattr(args, 'provider', 'google-cls-v2') if provider == 'google-batch': return google_batch.GoogleBatchJobProvider( @@ -63,11 +61,6 @@ def get_provider(args, resources, credentials_fn=None): args.project, args.location, credentials=credentials_fn(args)) - elif provider == 'google-v2': - return google_v2.GoogleV2JobProvider( - getattr(args, 'dry_run', False), - args.project, - credentials=credentials_fn(args)) elif provider == 'local': return local.LocalJobProvider(resources) elif provider == 'test-fails': @@ -88,17 +81,16 @@ def create_parser(prog): parser.add_argument( '--provider', - default='google-v2', - choices=[ - 'local', 'google-v2', 'google-cls-v2', 'google-batch', 'test-fails' - ], - help="""Job service provider. Valid values are "google-v2" (Google's - Pipeline API v2alpha1), "google-cls-v2" (Google's Pipelines API v2beta), + default='google-cls-v2', + choices=['local', 'google-cls-v2', 'google-batch', 'test-fails'], + help="""Job service provider. Valid values are + "google-cls-v2" (Google's Pipelines API v2beta), "google-batch" (Google's Batch API v1alpha1), and "local" (local Docker execution). "test-*" providers are for testing purposes only. - (default: google-v2)""", - metavar='PROVIDER') + (default: google-cls-v2)""", + metavar='PROVIDER', + ) return parser @@ -130,8 +122,6 @@ def get_dstat_provider_args(provider, project, location): args = [] if provider_name == 'google-cls-v2': args.append(f'--project {project} --location {location}') - elif provider_name == 'google-v2': - args.append(f'--project {project}') elif provider_name == 'google-batch': args.append(f'--project {project} --location {location}') elif provider_name == 'local': @@ -160,7 +150,10 @@ def emit_provider_message(provider): def check_for_unsupported_flag(args): """Raise an error if the provider doesn't support a provided flag.""" if args.label and args.provider not in [ - 'test-fails', 'local', 'google-v2', 'google-cls-v2', 'google-batch' + 'test-fails', + 'local', + 'google-cls-v2', + 'google-batch', ]: raise ValueError( '--label is not supported by the "%s" provider.' % args.provider) diff --git a/examples/decompress/README.md b/examples/decompress/README.md index 6f1a3d8..7fa2f81 100644 --- a/examples/decompress/README.md +++ b/examples/decompress/README.md @@ -33,7 +33,7 @@ To run a command to decompress the VCF file, type: ``` dsub \ - --provider google-v2 \ + --provider google-cls-v2 \ --project MY-PROJECT \ --zones "us-central1-*" \ --logging "gs://MY-BUCKET/decompress_one/logging" \ @@ -122,7 +122,7 @@ output file name. ``` dsub \ - --provider google-v2 \ + --provider google-cls-v2 \ --project MY-PROJECT \ --zones "us-central1-*" \ --logging "gs://MY-BUCKET/decompress_list/logging" \ diff --git a/examples/decompress/submit_list.sh b/examples/decompress/submit_list.sh index d05bfa2..334d6d9 100755 --- a/examples/decompress/submit_list.sh +++ b/examples/decompress/submit_list.sh @@ -34,7 +34,7 @@ readonly SCRIPT_DIR="$(dirname "${0}")" # Launch the task dsub \ - --provider google-v2 \ + --provider google-cls-v2 \ --project "${MY_PROJECT}" \ --zones "us-central1-*" \ --logging "${MY_BUCKET}/decompress_list/logging/" \ diff --git a/examples/decompress/submit_one.sh b/examples/decompress/submit_one.sh index 3f8308b..0870e61 100755 --- a/examples/decompress/submit_one.sh +++ b/examples/decompress/submit_one.sh @@ -35,7 +35,7 @@ readonly SCRIPT_DIR="$(dirname "${0}")" # Launch the task dsub \ - --provider google-v2 \ + --provider google-cls-v2 \ --project "${MY_PROJECT}" \ --zones "us-central1-*" \ --logging "${MY_BUCKET_PATH}"/logging/ \ diff --git a/examples/fastqc/README.md b/examples/fastqc/README.md index c0848f1..ef02fb7 100644 --- a/examples/fastqc/README.md +++ b/examples/fastqc/README.md @@ -77,7 +77,7 @@ To run FastQC on the BAM file, type: ``` dsub \ - --provider google-v2 \ + --provider google-cls-v2 \ --project MY-PROJECT \ --zones "us-central1-*" \ --logging "gs://MY-BUCKET/fastqc/submit_one/logging" \ @@ -99,9 +99,9 @@ You should see output like: Job: fastqc----170619-105212-67 Launched job-id: fastqc----170619-105212-67 To check the status, run: - dstat --provider google-v2 --project MY-PROJECT --jobs fastqc----170619-105212-67 --status '*' + dstat --provider google-cls-v2 --project MY-PROJECT --jobs fastqc----170619-105212-67 --status '*' To cancel the job, run: - ddel --provider google-v2 --project MY-PROJECT --jobs fastqc----170619-105212-67 + ddel --provider google-cls-v2 --project MY-PROJECT --jobs fastqc----170619-105212-67 Waiting for job to complete... Waiting for: fastqc----170619-105212-67. ``` @@ -156,7 +156,7 @@ output file name. ``` dsub \ - --provider google-v2 \ + --provider google-cls-v2 \ --project MY-PROJECT \ --zones "us-central1-*" \ --logging "gs://MY-BUCKET/samtools/submit_list/logging" \ @@ -175,9 +175,9 @@ Job: fastqc----170522-154943-70 Launched job-id: fastqc----170522-154943-70 3 task(s) To check the status, run: - dstat --provider google-v2 --project MY-PROJECT --jobs fastqc----170522-154943-70 --status '*' + dstat --provider google-cls-v2 --project MY-PROJECT --jobs fastqc----170522-154943-70 --status '*' To cancel the job, run: - ddel --provider google-v2 --project MY-PROJECT --jobs fastqc----170522-154943-70 + ddel --provider google-cls-v2 --project MY-PROJECT --jobs fastqc----170522-154943-70 Waiting for job to complete... Waiting for: fastqc----170522-154943-70. ``` diff --git a/examples/fastqc/submit_list.sh b/examples/fastqc/submit_list.sh index fa5d856..2421cf3 100755 --- a/examples/fastqc/submit_list.sh +++ b/examples/fastqc/submit_list.sh @@ -40,7 +40,7 @@ gcloud builds submit "${SCRIPT_DIR}" \ # Launch the task dsub \ - --provider google-v2 \ + --provider google-cls-v2 \ --project "${MY_PROJECT}" \ --zones "us-central1-*" \ --logging "${OUTPUT_ROOT}/logging/" \ diff --git a/examples/fastqc/submit_one.sh b/examples/fastqc/submit_one.sh index eb8bae3..bb848fb 100755 --- a/examples/fastqc/submit_one.sh +++ b/examples/fastqc/submit_one.sh @@ -41,7 +41,7 @@ gcloud builds submit "${SCRIPT_DIR}" \ # Launch the task dsub \ - --provider google-v2 \ + --provider google-cls-v2 \ --project "${MY_PROJECT}" \ --zones "us-central1-*" \ --logging "${OUTPUT_ROOT}/logging" \ diff --git a/examples/samtools/README.md b/examples/samtools/README.md index 4080bf5..5f968ad 100644 --- a/examples/samtools/README.md +++ b/examples/samtools/README.md @@ -38,7 +38,7 @@ To run a command to index the BAM file, type: ``` dsub \ - --provider google-v2 \ + --provider google-cls-v2 \ --project MY-PROJECT \ --zones "us-central1-*" \ --logging "gs://MY-BUCKET/samtools/submit_one/logging" \ @@ -63,9 +63,9 @@ You should see output like: Job: samtools-i----170522-153810-14 Launched job-id: samtools-i----170522-153810-14 To check the status, run: - dstat --provider google-v2 --project MY-PROJECT --jobs samtools-i----170522-153810-14 --status '*' + dstat --provider google-cls-v2 --project MY-PROJECT --jobs samtools-i----170522-153810-14 --status '*' To cancel the job, run: - ddel --provider google-v2 --project MY-PROJECT --jobs samtools-i----170522-153810-14 + ddel --provider google-cls-v2 --project MY-PROJECT --jobs samtools-i----170522-153810-14 Waiting for job to complete... Waiting for: samtools-i----170522-153810-14. ``` @@ -119,7 +119,7 @@ output file name. ``` dsub \ - --provider google-v2 \ + --provider google-cls-v2 \ --project MY-PROJECT \ --zones "us-central1-*" \ --logging "gs://MY-BUCKET/samtools/submit_list/logging" \ @@ -141,9 +141,9 @@ Job: samtools-i----170522-154943-70 Launched job-id: samtools-i----170522-154943-70 3 task(s) To check the status, run: - dstat --provider google-v2 --project MY-PROJECT --jobs samtools-i----170522-154943-70 --status '*' + dstat --provider google-cls-v2 --project MY-PROJECT --jobs samtools-i----170522-154943-70 --status '*' To cancel the job, run: - ddel --provider google-v2 --project MY-PROJECT --jobs samtools-i----170522-154943-70 + ddel --provider google-cls-v2 --project MY-PROJECT --jobs samtools-i----170522-154943-70 Waiting for job to complete... Waiting for: samtools-i----170522-154943-70. ``` diff --git a/examples/samtools/submit_list.sh b/examples/samtools/submit_list.sh index e8a9b37..82ca243 100755 --- a/examples/samtools/submit_list.sh +++ b/examples/samtools/submit_list.sh @@ -36,7 +36,7 @@ readonly SCRIPT_DIR="$(dirname "${0}")" # Launch the task dsub \ - --provider google-v2 \ + --provider google-cls-v2 \ --project "${MY_PROJECT}" \ --zones "us-central1-*" \ --logging "${OUTPUT_ROOT}/logging/" \ diff --git a/examples/samtools/submit_one.sh b/examples/samtools/submit_one.sh index a1154f4..c3e46ed 100755 --- a/examples/samtools/submit_one.sh +++ b/examples/samtools/submit_one.sh @@ -37,7 +37,7 @@ readonly SCRIPT_DIR="$(dirname "${0}")" # Launch the task dsub \ - --provider google-v2 \ + --provider google-cls-v2 \ --project "${MY_PROJECT}" \ --zones "us-central1-*" \ --logging "${OUTPUT_ROOT}"/logging \ diff --git a/test/integration/e2e_accelerator.google-v2.sh b/test/integration/e2e_accelerator.google-cls-v2.sh similarity index 100% rename from test/integration/e2e_accelerator.google-v2.sh rename to test/integration/e2e_accelerator.google-cls-v2.sh diff --git a/test/integration/e2e_block_external_network.google-v2.sh b/test/integration/e2e_block_external_network.google-cls-v2.sh similarity index 100% rename from test/integration/e2e_block_external_network.google-v2.sh rename to test/integration/e2e_block_external_network.google-cls-v2.sh diff --git a/test/integration/e2e_ddel.sh b/test/integration/e2e_ddel.sh index 7172e3b..40f1b87 100755 --- a/test/integration/e2e_ddel.sh +++ b/test/integration/e2e_ddel.sh @@ -82,11 +82,11 @@ if util::wait_for_canceled_status "${JOB_ID}"; then exit 1 fi -# For the google v2 providers, wait a sufficiently long time so that all the +# For the google providers, wait a sufficiently long time so that all the # startup events occur prior to canceling so that the output event list is # consistent.. if [[ "${DSUB_PROVIDER}" == "google-cls-v2" ]] || \ - [[ "${DSUB_PROVIDER}" == "google-v2" ]]; then + [[ "${DSUB_PROVIDER}" == "google-batch" ]]; then echo "Sleeping for 60s" sleep 60 fi diff --git a/test/integration/e2e_dstat.sh b/test/integration/e2e_dstat.sh index 9bda8fd..e00344d 100755 --- a/test/integration/e2e_dstat.sh +++ b/test/integration/e2e_dstat.sh @@ -71,9 +71,8 @@ function verify_dstat_output() { done # Check provider-specific fields - if [[ "${DSUB_PROVIDER}" == "google" ]] || \ - [[ "${DSUB_PROVIDER}" == "google-cls-v2" ]] || \ - [[ "${DSUB_PROVIDER}" == "google-v2" ]]; then + if [[ "${DSUB_PROVIDER}" == "google-cls-v2" ]] || \ + [[ "${DSUB_PROVIDER}" == "google-batch" ]]; then echo "Checking dstat ${DSUB_PROVIDER} provider fields" verify_dstat_google_provider_fields "${dstat_out}" "${ensure_complete}" fi diff --git a/test/integration/e2e_io_mount_bucket.google-v2.sh b/test/integration/e2e_io_mount_bucket.google-cls-v2.sh similarity index 100% rename from test/integration/e2e_io_mount_bucket.google-v2.sh rename to test/integration/e2e_io_mount_bucket.google-cls-v2.sh diff --git a/test/integration/e2e_io_mount_bucket_requester_pays.google-v2.sh b/test/integration/e2e_io_mount_bucket_requester_pays.google-cls-v2.sh similarity index 100% rename from test/integration/e2e_io_mount_bucket_requester_pays.google-v2.sh rename to test/integration/e2e_io_mount_bucket_requester_pays.google-cls-v2.sh diff --git a/test/integration/e2e_io_mount_disk.google-v2.sh b/test/integration/e2e_io_mount_disk.google-cls-v2.sh similarity index 100% rename from test/integration/e2e_io_mount_disk.google-v2.sh rename to test/integration/e2e_io_mount_disk.google-cls-v2.sh diff --git a/test/integration/e2e_io_mount_existing_disk.google-v2.sh b/test/integration/e2e_io_mount_existing_disk.google-cls-v2.sh similarity index 100% rename from test/integration/e2e_io_mount_existing_disk.google-v2.sh rename to test/integration/e2e_io_mount_existing_disk.google-cls-v2.sh diff --git a/test/integration/e2e_preemptible_retries_fail.google-v2.sh b/test/integration/e2e_preemptible_retries_fail.google-cls-v2.sh similarity index 100% rename from test/integration/e2e_preemptible_retries_fail.google-v2.sh rename to test/integration/e2e_preemptible_retries_fail.google-cls-v2.sh diff --git a/test/integration/e2e_python_api.py b/test/integration/e2e_python_api.py index 631d51e..63841ea 100644 --- a/test/integration/e2e_python_api.py +++ b/test/integration/e2e_python_api.py @@ -23,8 +23,8 @@ from dsub.lib import job_model from dsub.lib import param_util from dsub.lib import resources +from dsub.providers import google_batch from dsub.providers import google_cls_v2 -from dsub.providers import google_v2 from dsub.providers import local # Because this may be invoked from another directory (treated as a library) or @@ -45,8 +45,10 @@ def get_dsub_provider(): elif test.DSUB_PROVIDER == 'google-cls-v2': return google_cls_v2.GoogleCLSV2JobProvider(False, test.PROJECT_ID, 'us-central1') - elif test.DSUB_PROVIDER == 'google-v2': - return google_v2.GoogleV2JobProvider(False, test.PROJECT_ID) + elif test.DSUB_PROVIDER == 'google-batch': + return google_batch.GoogleBatchJobProvider( + False, test.PROJECT_ID, 'us-central1' + ) else: print('Invalid provider specified.', file=sys.stderr) sys.exit(1) diff --git a/test/integration/e2e_user.google-v2.sh b/test/integration/e2e_user.google-cls-v2.sh similarity index 97% rename from test/integration/e2e_user.google-v2.sh rename to test/integration/e2e_user.google-cls-v2.sh index 72c467c..7b3f146 100755 --- a/test/integration/e2e_user.google-v2.sh +++ b/test/integration/e2e_user.google-cls-v2.sh @@ -14,7 +14,7 @@ # See the License for the specific language governing permissions and # limitations under the License. -# For google-v2, user-ids and job names are used as labels. +# For google-cls-v2, user-ids and job names are used as labels. # Test that --user and --name works properly when it has underscores # and capital letters. diff --git a/test/integration/test_setup.sh b/test/integration/test_setup.sh index b8b1763..0e7fe14 100644 --- a/test/integration/test_setup.sh +++ b/test/integration/test_setup.sh @@ -34,7 +34,7 @@ # If the script name is ..sh, pull out the provider. # If the script name is .sh, use "local". # If the DSUB_PROVIDER is set, make sure it is correct for a provider test. -# Special-case the google-v2 tests to be runnable for google-cls-v2 +# Special-case the google-cls-v2 tests to be runnable for google-batch # and google-batch. readonly SCRIPT_NAME="$(basename "$0")" @@ -46,12 +46,9 @@ readonly SCRIPT_DEFAULT_PROVIDER=$( if [[ -z "${DSUB_PROVIDER:-}" ]]; then readonly DSUB_PROVIDER="${SCRIPT_DEFAULT_PROVIDER:-local}" elif [[ -n "${SCRIPT_DEFAULT_PROVIDER}" ]]; then - if [[ "${DSUB_PROVIDER}" == "google-cls-v2" ]] && \ - [[ "${SCRIPT_DEFAULT_PROVIDER}" == "google-v2" ]]; then - echo "Running google-v2 e2e/unit tests with provider google-cls-v2" - elif [[ "${DSUB_PROVIDER}" == "google-batch" ]] && \ - [[ "${SCRIPT_DEFAULT_PROVIDER}" == "google-v2" ]]; then - echo "Running google-v2 e2e/unit tests with provider google-batch" + if [[ "${DSUB_PROVIDER}" == "google-batch" ]] && \ + [[ "${SCRIPT_DEFAULT_PROVIDER}" == "google-cls-v2" ]]; then + echo "Running google-cls-v2 e2e/unit tests with provider google-batch" elif [[ "${DSUB_PROVIDER}" != "${SCRIPT_DEFAULT_PROVIDER}" ]]; then 1>&2 echo "DSUB_PROVIDER is '${DSUB_PROVIDER:-}' not '${SCRIPT_DEFAULT_PROVIDER}'" exit 1 @@ -115,22 +112,6 @@ function dsub_google-cls-v2() { "${@}" } -function dsub_google-v2() { - local zones="${ZONES:-}" - local regions="${REGIONS:-}" - if [[ -z "${regions}" ]] && [[ -z "${zones}" ]]; then - regions="us-central1" - fi - - dsub \ - --provider google-v2 \ - --project "${PROJECT_ID}" \ - --logging "${LOGGING_OVERRIDE:-${LOGGING}}" \ - ${regions:+--regions "${regions}"} \ - ${zones:+--zones "${zones}"} \ - "${@}" -} - function dsub_local() { dsub \ --provider local \ @@ -182,13 +163,6 @@ function dstat_google-cls-v2() { "${@}" } -function dstat_google-v2() { - dstat \ - --provider google-v2 \ - --project "${PROJECT_ID}" \ - "${@}" -} - function dstat_local() { dstat \ --provider local \ @@ -223,13 +197,6 @@ function ddel_google-cls-v2() { "${@}" } -function ddel_google-v2() { - ddel \ - --provider google-v2 \ - --project "${PROJECT_ID}" \ - "${@}" -} - function ddel_google-batch() { local location="${LOCATION:-}" diff --git a/test/integration/test_setup_e2e.py b/test/integration/test_setup_e2e.py index 6a31b22..d2534a5 100644 --- a/test/integration/test_setup_e2e.py +++ b/test/integration/test_setup_e2e.py @@ -206,31 +206,6 @@ def dsub_google_cls_v2(dsub_args): # pyformat: enable -def dsub_google_v2(dsub_args): - """Call dsub appending google-v2 required arguments.""" - # pyformat: disable - google_v2_opt_args = [ - ("BOOT_DISK_SIZE", "--boot-disk-size"), - ("DISK_SIZE", "--disk-size") - ] - # pyformat: enable - - opt_args = [] - for var in google_v2_opt_args: - val = globals().get(var[0]) - if val: - opt_args.append(var[1], val) - - # pyformat: disable - return dsub_command.call([ - "--provider", "google-v2", - "--project", PROJECT_ID, - "--logging", LOGGING, - "--regions", "us-central1" - ] + opt_args + dsub_args) - # pyformat: enable - - def dsub_local(dsub_args): """Call dsub appending local-provider required arguments.""" diff --git a/test/integration/unit_flags.google-v2.sh b/test/integration/unit_flags.google-cls-v2.sh similarity index 92% rename from test/integration/unit_flags.google-v2.sh rename to test/integration/unit_flags.google-cls-v2.sh index 79610c9..067c576 100755 --- a/test/integration/unit_flags.google-v2.sh +++ b/test/integration/unit_flags.google-cls-v2.sh @@ -17,10 +17,10 @@ set -o errexit set -o nounset -# unit_flags.google-v2.sh +# unit_flags.google-cls-v2.sh # # Collection of unit tests for dsub command-line flags -# specific to the google-v2 provider. +# specific to the google-cls-v2 provider. readonly SCRIPT_DIR="$(dirname "${0}")" @@ -42,13 +42,8 @@ function call_dsub() { } readonly -f call_dsub -if [[ "${DSUB_PROVIDER}" == "google-cls-v2" ]]; then - readonly NETWORK_NAME_KEY="network" - readonly CONTAINER_NAME_KEY="containerName" -elif [[ "${DSUB_PROVIDER}" == "google-v2" ]]; then - readonly NETWORK_NAME_KEY="name" - readonly CONTAINER_NAME_KEY="name" -fi +readonly NETWORK_NAME_KEY="network" +readonly CONTAINER_NAME_KEY="containerName" # Define tests @@ -129,7 +124,7 @@ function test_location() { } readonly -f test_location -function test_neither_region_nor_zone_google-cls-v2() { +function test_neither_region_nor_zone() { local subtest="${FUNCNAME[0]}" if call_dsub \ @@ -148,32 +143,6 @@ function test_neither_region_nor_zone_google-cls-v2() { test_failed "${subtest}" fi } -readonly -f test_neither_region_nor_zone_google-cls-v2 - -function test_neither_region_nor_zone_google-v2() { - local subtest="${FUNCNAME[0]}" - - if call_dsub \ - --command 'echo "${TEST_NAME}"'; then - - 1>&2 echo "Neither regions nor zones specified - not detected" - - test_failed "${subtest}" - else - - assert_output_empty - - assert_err_contains \ - "ValueError: Exactly one of --regions and --zones must be specified" - - test_passed "${subtest}" - fi -} -readonly -f test_neither_region_nor_zone_google-v2 - -function test_neither_region_nor_zone() { - test_neither_region_nor_zone_"${DSUB_PROVIDER}" -} readonly -f test_neither_region_nor_zone function test_region_and_zone() { @@ -191,13 +160,8 @@ function test_region_and_zone() { assert_output_empty - if [[ "${DSUB_PROVIDER}" == "google-cls-v2" ]]; then - assert_err_contains \ - "ValueError: At most one of --regions and --zones may be specified" - elif [[ "${DSUB_PROVIDER}" == "google-v2" ]]; then - assert_err_contains \ - "ValueError: Exactly one of --regions and --zones must be specified" - fi + assert_err_contains \ + "ValueError: At most one of --regions and --zones may be specified" test_passed "${subtest}" fi @@ -327,7 +291,7 @@ function test_machine_type_with_ram_and_cpu() { --min-cores 1 \ --min-ram 1; then - 1>&2 echo "min-ram/min-cores set with machine-type on google-v2 provider - not detected" + 1>&2 echo "min-ram/min-cores set with machine-type on google-cls-v2 provider - not detected" test_failed "${subtest}" else @@ -931,14 +895,8 @@ function test_block_external_network() { --regions us-central1 \ --block-external-network; then - # Check that the output contains expected values - if [[ "${DSUB_PROVIDER}" == "google-cls-v2" ]]; then - assert_err_value_equals \ - "[0].pipeline.actions.[3].blockExternalNetwork" "True" - elif [[ "${DSUB_PROVIDER}" == "google-v2" ]]; then - assert_err_value_equals \ - "[0].pipeline.actions.[3].flags.[0]" "BLOCK_EXTERNAL_NETWORK" - fi + assert_err_value_equals \ + "[0].pipeline.actions.[3].blockExternalNetwork" "True" test_passed "${subtest}" else @@ -955,13 +913,8 @@ function test_no_block_external_network() { --regions us-central1; then # Check that the output does not contain block network flag - if [[ "${DSUB_PROVIDER}" == "google-cls-v2" ]]; then - assert_err_value_equals \ - "[0].pipeline.actions.[3].blockExternalNetwork" "False" - elif [[ "${DSUB_PROVIDER}" == "google-v2" ]]; then - assert_err_not_contains \ - "BLOCK_EXTERNAL_NETWORK" - fi + assert_err_value_equals \ + "[0].pipeline.actions.[3].blockExternalNetwork" "False" test_passed "${subtest}" else @@ -977,9 +930,7 @@ trap "exit_handler" EXIT mkdir -p "${TEST_TMP}" echo -if [[ "${DSUB_PROVIDER}" == "google-cls-v2" ]]; then - test_location -fi +test_location echo test_preemptible_zero diff --git a/test/run_tests.sh b/test/run_tests.sh index c2254bf..bba73a8 100755 --- a/test/run_tests.sh +++ b/test/run_tests.sh @@ -206,12 +206,9 @@ function get_test_providers() { if [[ ${test_file} == *.*.sh ]]; then local providers="$(echo -n "${test_file}" | awk -F . '{ print $(NF-1) }')" - # Special case the google-v2 tests - run them against google-cls-v2 as well - if [[ "${providers}" == "google-v2" ]]; then - echo -n "google-v2 google-cls-v2" # Special case the google-batch tests - don't run them when this flag is set # To be renabled once batch client library is available in G3 - elif [[ "${providers}" == "google-batch" ]] && [[ "${NO_GOOGLE_BATCH_TESTS:-0}" -eq 1 ]]; then + if [[ "${providers}" == "google-batch" ]] && [[ "${NO_GOOGLE_BATCH_TESTS:-0}" -eq 1 ]]; then echo -n "" else echo -n "${providers}" @@ -219,7 +216,7 @@ function get_test_providers() { return fi if [[ "${NO_GOOGLE_BATCH_TESTS:-0}" -eq 1 ]]; then - echo -n "local google-v2 google-cls-v2" + echo -n "local google-cls-v2" return fi case "${test_file}" in @@ -236,8 +233,8 @@ function get_test_providers() { e2e_io.sh | \ e2e_io_auto.sh | \ e2e_io_gcs_tasks.sh | \ - e2e_io_mount_bucket.google-v2.sh | \ - e2e_io_mount_bucket_requester_pays.google-v2.sh | \ + e2e_io_mount_bucket.google-cls-v2.sh | \ + e2e_io_mount_bucket_requester_pays.google-cls-v2.sh | \ e2e_io_recursive.sh | \ e2e_io_tasks.sh | \ e2e_logging_content.sh | \ @@ -249,7 +246,7 @@ function get_test_providers() { e2e_logging_paths_retry_failure_tasks.sh | \ e2e_logging_paths_retry_tasks.sh | \ e2e_non_root.sh | \ - e2e_preemptible_retries_fail.google-v2.sh | \ + e2e_preemptible_retries_fail.google-cls-v2.sh | \ e2e_python.sh | \ e2e_requester_pays_buckets.sh | \ e2e_retries_success.sh | \ @@ -258,10 +255,10 @@ function get_test_providers() { e2e_runtime.sh | \ e2e_skip.sh | \ e2e_skip_tasks.sh) - local all_provider_list="${DSUB_PROVIDER:-local google-v2 google-cls-v2 google-batch}" + local all_provider_list="${DSUB_PROVIDER:-local google-cls-v2 google-batch}" ;; *) - local all_provider_list="${DSUB_PROVIDER:-local google-v2 google-cls-v2}" + local all_provider_list="${DSUB_PROVIDER:-local google-cls-v2}" ;; esac