[RFC] GPU support in CLI

[flx42]

The goal of this RFC is to discuss how we can expose the GPU support that was recently added to containerd by @crosbymichael in this PR: containerd/containerd#2330

Since there are multiple ways of implementing this, I think it makes sense to start with the user experience, i.e. the CLI. If we can agree on the format, the next step will be that I write a PR for API changes in the github.com/docker/docker/api/types package. My rationale being that if we decide to not allow some configuration options in the CLI, it would be useless to have api/types expose it. As a concrete example, if we don't allow passing options on a per-GPU basis, it shouldn't be part of the API.

After discussing with @crosbymichael, he advised to limit the support to GPUs. We are excluding other types of specialized hardware such as FPGAs, InfiniBand, ASICs... from this RFC.

This RFC introduces some vocabulary, then presents the approach I find is the most compelling.

GPU Vendors

• While the containerd PR is only for NVIDIA GPUs, the CLI should be able to support multiple vendors.
• For simplicity, we should enforce the GPU vendor to be a lowercase string with no special characters.
  e.g. nvidia, intel or amd.
• Reverse-DNS notation (e.g. com.nvidia) is too verbose and should probably be avoided for a CLI.
• The list of support GPU vendors could be hardcoded in the CLI and checked, since it's unlikely to change quickly, but it is not a requirement.

GPU identifiers

• GPU support should not be a boolean flag like --with-gpus, users should be able to select a subset of all GPUs, similarly to --cpuset-cpus: nvidia=0,1,2,3
• We should not assume a numerical identifier for GPUs (unlike --cpuset-cpus), GPUs can have a string representation, for instance a UUID: nvidia=GPU-fef8089b-4820-abfc-e83e-94318197576e
• The CLI can only verify the basic format of GPU identifiers, it can't detect if they are ultimately valid (e.g. if the GPU exists or not), this will be done later and will be vendor-specific.

GPU options

• In addition to selecting GPUs, the user should be to add options to modify the runtime behavior. For the containerd PR it would mean having a way to modify this struct from the CLI: https://github.com/containerd/containerd/blob/master/contrib/nvidia/nvidia.go#L88-L96
• The baseline would be to have options that apply to all GPUs/to the system: nvidia.capabilities = compute.
• An extension would be to have options per-GPU, for instance to enforce that GPU 0 can only use 2 GB of memory, whereas GPU 1 can use 6 GB: limit_mem[0] = 2GB limit_mem[1] = 6GB.
• Same as for identifiers, the CLI can't perform actual validation of the options passed by the user, it's vendor-specific.

Suggested approach

Add two CLI options:

• --gpus for specifying a list of GPUs (like --cpuset-cpus)
• --gpus-opt for specifying a list of GPU options (like --storage-opt)

docker run --gpus nvidia=0,1,2,3 --gpus nvidia=GPU-abcd --gpus-opt nvidia.capabilities=compute --gpus-opt nvidia.kmods=true

The important element being the vendor name, which serves as a key for specifying options and identifiers.
I think this is easy to read since you don't have to pack everything in a single argument.

If we want per-GPU options, we can introduce some sort of indexing in --gpus-opt:

--gpus-opt nvidia.0.limit_mem=2GB
--gpus-opt nvidia[0].limit_mem=2GB

Discarded approaches

I think the approach above is the best, but here are some thoughts on other possibilities.

Single CSV argument

Similar to --mount, add a new argument --gpus which expects a CSV value.

docker run --gpus vendor=nvidia,options=capabilities=compute,options=kmods=true,0,1,2,3,GPU-abcd

This is explicit and everything is contained in one option, but it's also bizarre since we have nested equal signs, and we have to expand the list of GPU identifiers as multiple arguments, to avoid needing complicated quoting from the shell.

A follow-up question would then be: can you specify multiple times the same vendor? If yes, what if the listed options don't match?

Similar to --device

Add a new argument --gpus which expects 2 or 3 strings separated by :

docker run --gpus nvidia:0,1,2,3,GPU-abcd:capabilities=compute,kmods=true

Less problematic than approach above, but hard to read.

NVIDIA specific

Same than the suggested approach, but without being generic:

docker run --nvidia-gpus 0,1,2,3 --nvidia-gpus GPU-abcd --nvidia-opt capabilities=compute --nvidia-opt kmods=true

It removes the need to use the vendor name as the key, and is fine for now, but is likely to be removed in the future if we add more vendors. Also, this doesn't match the approach used by the containerd PR.

[flx42]

cc @3XX0 @crosbymichael

[cpuguy83]

First approach SGTM

[cwgem]

• What's the recommended method of centralizing these options for mass deployment? Something like docker compose? Given how lengthy this makes the CLI arguments (even more so if per GPU properties are supported) I feel like someone would rather dump all the config info in a JSON file or so and have the CLI read off that instead.
• What's the support matrix look like? Not sure if "whatever docker runs on" would be specific enough (OSX for example). I'm assuming OS package availability and kernel version would play into this.

[cpuguy83]

What's the recommended method of centralizing these options for mass deployment?

alias nvidia-docker="docker run --gpus=nvidia:0,1,2,3 --gpus-opt="..."

Really, if I just had a standard set of things I'd want to include, I'd put it into an variable or a shell function that I can use to inject it into my docker run with docker run ${my_gpu_opts}

What's the support matrix look like?

The docker client doesn't need to care about this. If the system that the engine is running on doesn't support it (e.g., likely that docker4mac doesn't unless they end up injecting a GPU passthrough or something?) then it would error out from the daemon side, likely at runtime.

[cwgem]

I thought about responding with additional feedback but not even worth it at this point. If your response to someone actually commenting on an RFC (Request For Comments) is some kind of low level trolling plus affirming that's okay via a laughing face from the original RFC author why even bother?

[cpuguy83]

I made a lighthearted joke at no ones expense and gave some real feedback. This isn't my proposal.

[crosbymichael]

@flx42 Ya, I like the suggested approach and namespaced keys for the opts. I'm guessing it would all people to get a quick default, docker run --gpus 0 but then expand on the options if needed.

[crosbymichael]

Maybe a compose section looks something like:

- gpus:
    - type: "nvidia"
    - devices: 
         - 0
         - "guid"
    - opts:
         - "..."

[cpuguy83]

Yeah, but not sure how we are reconciling this with what's supported on Swarm.

[cpuguy83]

The alternative would be to shove this into the "mounts" API, though we can still keep the somewhat specialized CLI for ease of use.

[flx42]

I thought about responding with additional feedback but not even worth it at this point. If your response to someone actually commenting on an RFC (Request For Comments) is some kind of low level trolling plus affirming that's okay via a laughing face from the original RFC author why even bother?

I laughed because it's true, that's something we might need to do, and we will call that backwards-compatibility :). I apologize if it appeared dismissive of your reply.

I think, as mentioned above, we need a sensible default, therefore we need to make sure --gpus nvidia=0,1 is sufficient for most (all?) use-cases. If you are doing advanced use cases such as resource limitation or security tuning, you will indeed need additional CLI arguments, but in this case is it that different from specifying all the cgroups, mounts and security options on the CLI today?

[cpuguy83]

@flx42 Would it make sense to take that a step further and support something like --gpus=nvidia and just include everything?

[flx42]

@cpuguy83 I was also wondering if we should include a special value to select all the GPUs:

--gpus nvidia # your proposal
--gpus nvidia=* # might be problematic with shell globbing, but --device-cgroup-rule is similar
--gpus nvidia=all # ambiguous with GPU ID

I'm fine with your option.

[cpuguy83]

I think --gpus nvidia should be good, we may want to make sure that --gpus nvidia= (no value after the =) is invalid, in terms of flag parsing.

@cwgem does this work in terms of simplifying the experience?

---

Separate topic: Where should this fit into the API?

[flx42]

@cpuguy83 started the API discussion here: moby/moby#37434

[crosbymichael]

Do you think we should make the CLI more abstract where you don't specify GPU device id but number of gpus added? Or is this not a good abstraction for this layer of the stack?

[flx42]

I think this would cause some issues, this doesn't feel to be the right place in the stack for resource management/allocation.

• It would probably require each device to be exclusively assigned to a single container, there is no reason to add this requirement.
• Many scripts or even job scheduling systems do their own resource management, and I believe some of them call the docker CLI directly. In this case they know exactly which GPU UUID they want to use.

[thaJeztah]

Happy to see this moving, and to see that the (technical) design is starting to take form!

Just a quick first look (I'll have to give this a bit more thinking);

The list of support GPU vendors could be hardcoded in the CLI and checked, since it's unlikely to change quickly, but it is not a requirement.

Given that a CLI can be used against different versions of the daemon, we try to avoid validation as much as possible in the CLI, and delegate this to the daemon (i.e.; the API can return an error if a vendor and/or option is not supported); this way any client using the API would also be covered.

This is similar to (e.g.) a specific logging-driver not being available on a daemon

question: is dockerd aware which vendors are supported, or only containerd? Is a daemon-configuration needed (similar to custom runtimes, see moby/moby#22845) to add new vendors?

The CLI can only verify the basic format of GPU identifiers, it can't detect if they are ultimately valid (e.g. if the GPU exists or not), this will be done later and will be vendor-specific.

Same as above; validation of what's valid should be handle by the daemon/runtime.

Possibly (and, if needed), a minimal validation (e.g., length, accepted characters - if we can define) could be possible to fail early

Suggested approach

Add two CLI options:

• --gpus for specifying a list of GPUs (like --cpuset-cpus)
• --gpus-opt for specifying a list of GPU options (like --storage-opt)

Some questions/remarks here;

• would it be possible to have GPUs from different vendors (and / or "types") attached to a container?
• if so, how to correlate options with the right CPU?
  • for example, two nvidia GPUs, same type, but both should have different options set (just making this up; I don't know if this is something that would be done in practice)
  • for example, two nvidia GPUs. but different types; option nvidia.foo only applies to one of them
  • same as above, but one intel GPU, and one nvidia (partially could be covered by a nvidia. / intel. prefix for options)
• TL;DR; two separate options could introduce the same problem we had with --volume-driver and --volume

Single CSV argument

...
This is explicit and everything is contained in one option, but it's also bizarre since we have nested equal signs, and we have to expand the list of GPU identifiers as multiple arguments, to avoid needing complicated quoting from the shell.

I agree the CSV format can become ugly/verbose. It's also the most non-ambiguous, extensible format currently (and would address my questions above for the "suggested approach")

we have to expand the list of GPU identifiers as multiple arguments

If we support passing CSV arguments multiple times, expansion would not be needed;

--gpu vendor=nvidia,identifier=GPU-abcd,options="capabilities=compute,kmods=true",..
--gpu vendor=nvidia,identifier=0,options="capabilities=compute,kmods=true",..
--gpu vendor=intel,identifier=1,options=..

I expect users that require many options to not be typing these by hand, but using (e.g.) a docker-compose file to set these.

The format in a compose file could be something like;

gpus:
  - identifier: GPU-abcd 
    type: nvidia
    options:
      - capabilities=compute
      - kmods=true
  - identifier: 0
    type: nvidia
    options: ["capabilities=compute", "kmods=true"\
  - identifier: 1
    type: intel
    options:
      - "..."

Similar to --device

We should definitely avoid this format; colon-delimited, positional options have proven to be very problematic, and can easily become ambiguous

NVIDIA specific

This would not be extensible, and would require the CLI to be tied to the daemon version (and supported options)

[flx42]

Given that a CLI can be used against different versions of the daemon, we try to avoid validation as much as possible in the CLI, and delegate this to the daemon (i.e.; the API can return an error if a vendor and/or option is not supported); this way any client using the API would also be covered.

That makes sense, OK.

question: is dockerd aware which vendors are supported, or only containerd? Is a daemon-configuration needed (similar to custom runtimes, see moby/moby#22845) to add new vendors?

Today the implementation for NVIDIA relies on the API provided by containerd. Other vendors could integrate either at the dockerd or containerd level, but I don't think we want a plugin system unless we take one step back, implement this RFE and force all vendors to go through a prestart hook (it doesn't have to be a fancy hook like ours, it can be written in bash and do a few mknod/mount calls).

Same as above; validation of what's valid should be handle by the daemon/runtime.

Even if we wanted to, it would not be possible for GPU identifiers since you might be on a MacOS machine connected to a remote docker daemon.

would it be possible to have GPUs from different vendors (and / or "types") attached to a container?

Sure: --gpus nvidia=0,1,2,3 --gpus intel=0

for example, two nvidia GPUs, same type, but both should have different options set (just making this up; I don't know if this is something that would be done in practice)

That makes sense, yes, look at my example above for limiting GPU memory:

--gpus-opt nvidia.0.limit_mem=2GB
--gpus-opt nvidia[0].limit_mem=2GB

But I've excluded this feature from the initial API, I don't think we need it right away.

same as above, but one intel GPU, and one nvidia (partially could be covered by a nvidia. / intel. prefix for options)

The prefix would solve the problem indeed:

--gpus-opt nvidia.myopt=42
--gpus-opt intel.myopt=0

--gpu vendor=nvidia,identifier=GPU-abcd,options="capabilities=compute,kmods=true",..

The CLI will see the following (because bash):

--gpu vendor=nvidia,identifier=GPU-abcd,options=capabilities=compute,kmods=true,..

Which is not what we want, now it's a quoting nightmare.

cc @shin- since compose was mentioned multiple times already :)

[thaJeztah]

That makes sense, yes, look at my example above for limiting GPU memory:

--gpus-opt nvidia.0.limit_mem=2GB
--gpus-opt nvidia[0].limit_mem=2GB

The problem I see with the numeric indexes is that (translating that to the compose file), options also have to be provided in the same order/position;

For example;

--gpus nvidia=0,3,4 --gpus-opt nvidia.2.limit_mem=2GB

Would (if I see it correctly) become something like

gpus:
    types:
      - nvidia
      - nvidia
      - nvidia
    identifiers:
     - 0
     - 3
     - 4
    opts:
      -
      -
      - ["limit_mem=2GB", "other_opt=foo"]

[flx42]

@thaJeztah I'm not sure I understand, why is the CLI format impacting the docker-compose format? Does it matter as long as we can have the same semantic with the API behind?

I think the best compose format would be what you described above:

gpus:
  - identifier: GPU-abcd 
    type: nvidia
    options:
      - capabilities=compute
      - kmods=true
  - identifier: 0
    type: nvidia
    options: ["capabilities=compute", "kmods=true"\
  - identifier: 1
    type: intel
    options:
      - "..."

[thaJeztah]

Perhaps it's not a strict rule, but so far, there's been a mapping between long-form options on the CLI and how they are used in the Docker Compose file

e.g.--mount type=foo,source=bar,target=baz translates to

volumes:
  - type: foo
    source: bar
    target: baz

[RenaudWasTaken]

So to recap a bit the conversation if we want the compose format to look like this

gpus:
  - identifier: GPU-abcd 
    type: nvidia
    options:
      - capabilities=compute
      - kmods=true
  - identifier: 0
    type: nvidia
    options: ["capabilities=compute", "kmods=true"]
  - identifier: 1
    type: intel
    options:
      - "..."

The CLI would look like this:
docker run --gpus type=nvidia,identifier=GPU-abcd,options=[capabilities=compute, kmods=true] --gpus type=nvidia,identifier=0, options=[capabilities=compute, kmods=true]

Declining it into multiple:

• Expose all the GPUs: docker run -it --gpus type=nvidia nvidia/cuda:9.0-base nvidia-smi
• Expose some of the GPUs: docker run -it --gpus type=nvidia,identifier=1,2,3 nvidia/cuda:9.0-base nvidia-smi

Compared to today:

• Expose all the GPUs: docker run -it --runtime=nvidia nvidia/cuda:9.0-base nvidia-smi
• Expose some of the GPUs: docker run -it --runtime=nvidia -e NVIDIA_VISIBLE_DEVICES=1,2,3 nvidia/cuda:9.0-base nvidia-smi

This looks fine for the compose file, but I'm a bit uncomfortable with having to write everything on the CLI.
What do you think @thaJeztah ?

[cpuguy83]

@RenaudWasTaken Why wouldn't that be multiple --gpu flags instead?

Also, can we s/identifier/id/?

[RenaudWasTaken]

@cpuguy83 my thinking was that you would use multiple --gpu flags when you need to have different options or provider.
e.g: docker run --gpus type=nvidia,identifier=GPU-abcd,options=[capabilities=compute, kmods=true] --gpus type=intel,identifier=[0,1,2], options=[capabilities=compute, kmods=true]

Translating to the following compose:

gpus:
  - identifier: GPU-abcd 
    type: nvidia
    options:
      - capabilities=compute
      - kmods=true
  - identifier: 0,1,2
    type: intel
    options:
      - capabilities=compute
      - kmods=true

But if you just want to expose 3 nvidia-gpus it would look like this:
docker run -it --gpus type=nvidia,identifier=[1,2,3] nvidia/cuda:9.0-base nvidia-smi

Translating to the following compose:

gpus:
  - identifier: 1,2,3
    type: nvidia
    options:
      - capabilities=compute
      - kmods=true

Also I'm wondering if gpu might make more sense (rather than id or identifier). e.g:
docker run -it --gpus type=nvidia,gpu=[1,2,3] nvidia/cuda:9.0-base nvidia-smi

Side note I don't think that value can be either a string or an array?

What do you think?

[tiborvass]

This is my proposal based on conversations I had with @RenaudWasTaken

Design taken:

• for various identified usecases, find the most simple UI that is consistent and compatible with the UI for the other usecases
• encourage portability
• future-proof

Proposed syntax:

• --gpu gpuRequest where gpuRequest (defaults to empty) has CSV syntax with the following fields:
  • count (implicit): the number of GPUs, default is 1, accepts "all"
  • type: the GPU device plugin, defaults to "all" # bikeshed: type? vendor? provider?
  • options: CSV (within CSV) syntax of provider-specific options to pass, defaults to empty string
  • id: comma-separated list of provider-scoped IDs, defaults to empty string

In the first iteration, let's assume there be only one GPU provider per engine ("nvidia").
However, the design should also work with multiple providers at once. (See "Future" at the end)

Most common usecases:

• "just gimme all the GPUs"
  • docker run --gpu all # equivalent to count=all, this is special-cased, useful in compute env (cloud, datacenter)
• "just gimme 2 GPUs"
  • docker run --gpu 2 # equivalent to count=2, describes the number of GPUs to provide, not the ID, useful in compute env
• "I know which GPU I want":
  • docker run --gpu id=1 # useful when using a local machine that has a less powerful primary graphics card

In most of the cases the above should be sufficient. It feels simple, as portable as possible (id is not portable) and allows for more complex queries.

Less common usecases:

• "I want to tweak provider-specific settings":
  • Limit GPU capabilities
    • docker run --gpu options="capabilities=compute"
  • Limit GPU resources (as a possible example for the future)
    • docker run --gpu options="memory.usage<=30GB"
• "I want provider-specific additional constraints, I know it can fail if provider decides constraints are not met":
  • Minimal amount of GPU memory (as a possible example for the future)
    • docker run --gpu options="memory.capacity>=16GB"
  • CUDA version
    • docker run --gpu options="cuda>=10"
  • Nvidia driver version
    • docker run --gpu options="driver>=384"
  • Brand
    • docker run --gpu options="brand=tesla"
• All of the above
  • docker run --gpu 'options="capabilities=compute,memory.usage<=30GB,memory.capacity>=16GB,cuda>=10,driver>=384,brand=tesla"'
    • Note that count=1 by default, and only one GPU provider ("nvidia") can be available at once
    • Note also that single quotes are needed for the shell to escape the double quotes
• "I'm testing GPU drivers, I don't need actual GPUs"
  • docker run --gpu count=0
• Multiple IDs:
  • docker run --gpu 'id="0,1,2,3"' # note that single quotes are needed for the shell to escape the double quotes, otherwise it should error out with helpful error message
  • docker run --gpu id=0 --gpu id=1 --gpu id=2 --gpu id=3 # this also works
• UUIDs:
  • docker run --gpu id=GPU-fef8089b…
• Multiple GPUs, different options (very unlikely this is ever going to be a usecase, but at least syntax is ready)
  • "I want a gpu with at least 8GB memory capacity, and another one that has at least 16GB"
    • docker run --gpu options="memory.capacity>=8GB" --gpu options="memory.capacity>=16GB"
  • GPU provider can decide to error out if different options are incompatible:
    • docker run --gpu options="cuda>=10" --gpu options="cuda=9"

Future (just to show that the following is possible)

• service create --gpu / k8s equivalent
  • How would a scheduler decide whether a node supports a GPUrequest ?
    • Ask each node's GPU provider (nope)
    • Decide based on a subset of GPURequest fields
      • "type" and "id" only? In which case store that information in each node description
      • Split "options" into "requirements/constraints" and "limits" and only decide based on "requirements"
• RUN --gpu in Dockerfile, this needs buildkit work
• graduate from provider-specific options to general gpuRequest fields (e.g. memory?)
• (very unlikely usecase, but technically possible) multi-provider on same node: collect as many GPU devices from each provider as the cumulative sum of "count" in GPURequests sent sequentially to each GPU provider
  • one could set GPU provider priority

Things I need to think more about:

• ENV hack vs Image config
  • An image should be able to say "I require a GPU to run"
• Implicit vs Explicit GPU request
  • If an image requires GPU, today we could rely on the application to error out
  • In the future it could be docker itself not starting the container without --gpu
  • But I don't think we should ever implicitly enable GPU just because an image needs it
• workflow of installing GPU provider (apt install vs docker plugin install)
• vocabulary: type vs provider vs vendor ...

[thaJeztah]

Very quick glance at your suggestion, but one thing that's a bit unclear how that'd work;

docker run —gpu options=“brand=brandX”

If options is vendor-specific; wouldn't that mean that (e.g.) brandX would know about a brand= restriction, but brandY wouldn't ? IoW, brandX would detect; "it's not for me", but brandY wouldn't know about it, but would still take a load that's only intended for brandX

I think for "per brand" options to be an option, we need a defined option that specified; only take this into account if the branch matches, and if so, options can be parsed by that driver/brand.

[thaJeztah]

“I want a gpu with at least 8GB memory capacity, and another one that has at least 16GB”

• docker run —gpu options=“memory.capacity>=8GB” —gpu options=“memory.capacity>=16GB”

This one may be tricky as well (as memory.capacity>=16GB also satisfies memory.capacity>=8GB); wondering if (related to my previous comment) we should split "options" from "constraints" as they seem to fall in a different category (options is to configure a driver/GPU, whereas constraints determine if a driver/GPU should be matched).

If we separate them, (possibly) we could define constraints that are driver/vendor-agnostic

[RenaudWasTaken]

@tiborvass thanks for your detailed API post, I think it would work pretty well for GPUs and it is pretty straightforward to use in most cases!

@thaJeztah can you give an exemple of the brand problem you are surfacing? I'm not completely grasping the issue.

[thaJeztah]

Oh! Sorry, I forgot to update; had a quick chat on Slack with @tiborvass, and I was confusing brand=foo for being a replacement for type=foo / vendor=foo / provider=foo;

• type: the GPU device plugin, defaults to "all" # bikeshed: type? vendor? provider?

Because of that confusion I thought that brand=foo was intended as the global "filter" to determine if a GPU should be used or not (which wouldn't work, as a "nvidia" type gpu wouldn't know about (say) options="opencl>=2", and an "amd" type GPU wouldn't know about options="cuda>=10" (and vice-versa))

---

This does bring up one thing (briefly touched on in #1200 (comment) as "I know it can fail if provider decides constraints are not met"); what is the desired outcome of (In a situation where a node can have a mixture of nvidia and/or amd GPUs);

docker run --gpu options="cuda>=10,opencl>=2"

• nvidia gpu is available; the GPU is used, and ignores the opencl>=2 option, or
• nvidia gpu is available; the GPU is used, but fails, because it doesn't know about the opencl>=2 option (and vice-versa for an amd GPU)

If options are vendor-specific, we could;

• only allow options if type= / vendor= / provider= is set
• if (vendor-specific) options are allowed to be mixed, require options to be namespaced/prefixed (options="amd.opencl>=2,nvidia.cuda>=10")
  • GPUs would "pick" all options that have their prefix/namespace, and ignore other options
  • Options without a prefix (no . in the option-name) are considered "generic" options; these options should be applicable to "any" type/vendor/provider
  • Having a prefix prevents conflicting options (optionFoo=bar having a different meaning for "vendor A" than for "vendor B"), and keeps the option open for "generic" options (as mentioned above), without the risk of a vendor "claiming" an option.

Downside of the "namespace" approach, is that, while it makes sense if you want to mix/mash GPUs (didn't set type/vendor/provider, aka "pick any GPU"), it is quite redundant if I did set one (why would I have to prefix each option, if I already specified I only want nvidia GPUs?)

If we really don't want to namespace/prefix options, we could of course leave the meaning of options="foobar=12" to the provider's ("nvidia" in this case) interpretation, but I can see value of having "generic" options for the "just gimme all the GPUs" (type=all (any?) situation), e.g.;

docker run --gpu options="memory.capacity>=16GB"

---

I'm a bit unclear on;

Multiple GPUs, different options
.. GPU provider can decide to error out if different options are incompatible

What is the expected result if multiple --gpu options are set? i.e. (perhaps not the best example);

docker run \
  --gpu type=nvidia,options="cuda=9,memory.capacity>=8GB" \
  --gpu type=nvidia,options="cuda=10,memory.capacity>=16GB"

• I want to run my container, and require 2 GPUs; one must be cuda 9, and requires at least 8GB, and the other one must be cuda 10, and requires at least 16GB, or
• I want to run my container, and require either a cuda 9 GPU with at least 8GB, or a cuda 10 GPU with 16GB or more, or
• variation of the above; "grab all you can", give me as many cuda 9 or cuda 10 GPUs as are available

(Perhaps this can be addressed by setting count=all (use if available), or count=1 (require 1))

[mviereck]

• It seems that the --gpus option is tied to NVIDIA hardware only and fails with other GPU vendors.
• It seems that the --gpus option is tied to NVIDIA docker images only and fails with other images.

Am I right? If yes, this new option is a vendor-lock-in and a shame.
Please add GPU support for all GPU vendors. Seriously.

[RenaudWasTaken]

Hello @mviereck !

The --gpus option is implemented more as a plugin than as a "vendor-lock-in". For exemple it doesn't work until you install the nvidia-container-toolkit (in other words the nvidia plugin).

We have plans to make it more like other plugins (docker plugin install), so that it's easier for everyone to build their own plugins. So far we have lacked time to do so. If you want to help drive this effort we'd be happy to help you (e.g: through review)!
The way the interface for GPUs has been designed should easily map to a REST API.

[mviereck]

@RenaudWasTaken Thank you for your friendly response!

If you want to help drive this effort we'd be happy to help you (e.g: through review)!

I am not experienced in Go, so I cannot contribute code directly, sorry.
I have opened a new ticket on this topic with some proposals, maybe you want to look at it, too: #2063