IDEA: auto generate pipeline definitions

[dnitsch]

Background:

Whilst it is possible to define complex and nested "pipelines" in taskctl, when running in a specific CI implementation like Gitlab/Github/TFS/BambooCI/CircleCI, etc.

It would be ill-advised to run the entire taskctl pipeline in a single CI tool job/step and thus losing out on the features they providew like specific re-runs of failed jobs, publish test reports natively, etc...

It is often reverted to running the specific tasks directly in some other YAML defined CI pipeline.

The proposal would see an autogeneration of CI YAML definitions for certain tools, enabling the user write and test and maintain a single taskctl.yml and generating a pipeline code for a specific tool.

Possible Solution:

One way to achieve this would be to have a top level property for the target generation, adding a generate-ci-def command.

The generated YAML at first would be kept to a single file, hence perhaps focusing on tools like Gitlab at first that lend themselves nicely to only having single file. Slowly, more complex relations between "CI" pipeline files which would correspond to taskctl pipelines.

taskctl genereate-ci-def <pipeline_name> would need to create a graph starting from the pipeline named and eventually use CI tool specific includes/relationship markers. This could be done multiple times to generate e.g. a pr pipeline, deploy pipeline, release pipeline, etc...

The generated file must have space for user to add additional keywords/jobs/steps that will not be overridden upon regeneration - this could be achieved by tokenizing the generated output with markers that would wrap each job/step in these markers, or perhaps only include a readme section that would explain a user specifiied markers would be ignored by the parser e.g.:

generated:
    yaml: .....
### BEGIN_TASKCTL_SKIP ###
user: 
    defined: stuff_goes_here
### END_TASKCTL_SKIP ###

This would be similar to tools like tfdocs, asyncapi generator

By design, where possible a task running in a container context would specify that container as the context to the CI tool - e.g. in Gitlab/CircleCI/Bamboo/TravisCI/GithubActions natively support running jobs/steps/tasks in containers, the inclination is to only support tools that do.

Constraints:

• Only deal with CI tools that have a well defined modern YAML interface (so things like Jenkins/Octopus Deploy/etc... will forever be out of scope)
• Only support tools that support both host based runner and container, host based for tasks that are using the native mvdn.shell context and container for the container based contexts