Proposal: support for using global variables in config file

[daniel-hutao]

Description

We sometimes need to use a configuration item multiple times in the config file, such as github.owner. If it can be configured through a variable, it will simplify the maintenance of the config file.

Describe the Proposal

maybe:

variables:
  GITHUB_ACCOUNT: daniel-hutao
  FOO: bar
tools:
  - name: xxx
    plugin: xxx
    options:
      owner: ${{ GITHUB_ACCOUNT }}

[daniel-hutao]

Wait for more comments about global variables
/cc @devstream-io/devstream

[imxw]

If there are global variables, will there be local variables in each plugin scope to override the global variables？

[daniel-hutao]

If there are global variables, will there be local variables in each plugin scope to override the global variables？

Good thinking. However, there are very few scenarios in which a configuration item needs to be repeatedly used in one plugin at present. It may be considered later when this scenario is more often. I think we can start by supporting global variables first.

[IronCore864]

Proposal

Example:

variables.yaml:

varNameA: A
varNameB: B

config.yaml:

tools:
- name: app
  plugin: some-plug-in
  options:
    variablesReferenceExampleA: [[ .varNameA ]]
    variablesReferenceExampleB: some_prefix/[[ .varNameB ]]/some_suffix
    outputsReferenceExample: ${{ name.plugin.outputs.key }}

1 How to Render Variables

We want to use Go's text/template to render variables.

It's the most simple way. After reading variables.yaml into a struct and config into bytes, before unmarshal config into a struct, we use Go's template to replace those vars.

2 Variables Naming Convention

Use camelCase; follow yaml convention.

3 Delimiters

Because our output's syntax is "${{ name.plugin.outputs.key }}", we can't use "${{" and "}}" as delimiters any more.

We can use "{{" and "}}" (Go template's default delimiters), but they are too similar to our output's syntax.

So, we use "[[" and "]]" as delimiters.

4 Where to Define Variables

We define variables in a separate file, rather than keeping it in the config.

Reasons:

• At the moment, we keep all config in one file, we do not support multiple files yet. In the future, when we have more plugins, we will consider splitting one single config into multiple files, and at that time, we shall use a separate file for variables as well. So, this design has some "future-proof."
• If we separate vars with config, it's easier to read the whole file, render, then unmarshal.

[IronCore864]

@daniel-hutao @imxw please review this simple design.

[daniel-hutao]

Great design! @IronCore864
But I'm afraid the format "[[ .varNameA ]]" has a little bit complex to normal users, especially the "." prefix with the variable.

Can we use the "[[ varNameA ]]" without "." ?
If the go template asks the "." must exist, can we add it by some way like "regexp"？

Anyway, I think many users will be confused with the "."

[IronCore864]

I think we must have the dot:

Execution of the template walks the structure and sets the cursor, represented by a period '.' and called "dot", to the value at the current location in the structure as execution proceeds.

Regex is a good idea.

Let's try using [[ varNameA ]] instead, and use regex to add the missing dot before rendering the template.

[daniel-hutao]

lgtm