If an init container in a TaskRun fails, entire TaskRun should fail

[bobcatfish]

Expected Behavior

If an init container in the Pod created for a TaskRun fails, then the subsequent steps/containers will not be able to run. Therefore in this case, the entire TaskRun should be marked as failed.

Actual Behavior

If an init container fails, the TaskRun status makes it look like it is still executing (i.e. the status of the init containers is not taken into account when determining if the TaskRun has failed), e.g. this is the status of a TaskRun I created where the init container that tried to checkout a GCS storage resource failed:

  status:
    conditions:
    - lastTransitionTime: "2019-03-26T22:49:55Z"
      reason: Building
      status: Unknown
      type: Succeeded
    podName: publish-run-pod-55cc64
    startTime: "2019-03-26T22:49:21Z"
    steps:
    - running:
        startedAt: "2019-03-26T22:49:39Z"
    - running:
        startedAt: "2019-03-26T22:49:40Z"
    - terminated:
        containerID: docker://0b40cffc7651cf781b8e4c0242377e3ca2499f9a0a0c572ced025b3f562ec643
        exitCode: 0
        finishedAt: "2019-03-26T22:49:27Z"
        reason: Completed
        startedAt: "2019-03-26T22:49:27Z"
    - running:
        startedAt: "2019-03-26T22:49:39Z"
    - running:
        startedAt: "2019-03-26T22:49:39Z"
    - terminated:
        containerID: docker://33fb0da5e4dadf73c2f9c8e4d62518d960ac71ce159f26286b1558766191302d
        exitCode: 1
        finishedAt: "2019-03-26T22:49:40Z"
        reason: Error
        startedAt: "2019-03-26T22:49:37Z"
    - running:
        startedAt: "2019-03-26T22:49:39Z"
    - running:
        startedAt: "2019-03-26T22:49:40Z"
    - running:
        startedAt: "2019-03-26T22:49:40Z"
    - running:
        startedAt: "2019-03-26T22:49:40Z"
    - terminated:
        containerID: docker://8deba837738b7303625824c64aaed620025ac58aeba44337a64be535ffe4d74a
        exitCode: 0
        finishedAt: "2019-03-26T22:49:41Z"
        reason: Completed
        startedAt: "2019-03-26T22:49:41Z"

The condition is:

    conditions:
    - lastTransitionTime: "2019-03-26T22:49:55Z"
      reason: Building
      status: Unknown
      type: Succeeded

Yet the init container failed:

    - terminated:
        containerID: docker://33fb0da5e4dadf73c2f9c8e4d62518d960ac71ce159f26286b1558766191302d
        exitCode: 1
        finishedAt: "2019-03-26T22:49:40Z"
        reason: Error
        startedAt: "2019-03-26T22:49:37Z"

(Note I ran into this because I tried to run our release Task and ran into #646)

Steps to Reproduce the Problem

1. Create a Task that takes a PipelineResource, e.g. a GCS storage resource
2. Create a TaskRun that points at a PipelineResource which will have an error, e.g. reproduce Fetching empty GCS storage bucket of type dir fails #646 by providing an empty bucket but using dir type

Additional Info

When we switched away from using init containers for steps, the status was briefly completely broken. In #634 we updated this logic to look at container statuses instead of init containers - but it looks like we need to look at both (and our integration tests need to cover this).

[bobcatfish]

Looking at this a bit more, the problem might be a bit deeper than I thought, I'm seeing the same behaviour even when it's a step that fails, i.e. the status stays at:

    conditions:
    - lastTransitionTime: "2019-03-26T22:49:55Z"
      reason: Building
      status: Unknown
      type: Succeeded

🤔

[vdemeester]

/assign

[vdemeester]

Reproduced by the following TaskRun. I think the entrypoint waits for ever in case of one step failing — looking into fixing that.

apiVersion: tekton.dev/v1alpha1
kind: TaskRun
metadata:
  name: build-simple
spec:
  taskSpec:
    steps:
    - name: echo
      image: docker.io/library/busybox
      command:
      - /bin/sh
      args:
      - -c
      - "echo echo"
    - name: exit
      image: docker.io/library/busybox
      command:
      - /bin/sh
      args:
      - -c
      - "exit 1"
    - name: echo-again
      image: docker.io/library/busybox
      command:
      - /bin/sh
      args:
      - -c
      - "echo again"
  trigger:
    type: manual

[bobcatfish]

Awwwwwesome @vdemeester , can confirm that the problem I was having yesterday isn't happening anymore today!!

    - lastTransitionTime: "2019-03-27T18:22:02Z"
      message: 'build step "build-step-build-push-base-images" exited with code 1
        (image: "docker-pullable://gcr.io/kaniko-project/executor@sha256:d9fe474f80b73808dc12b54f45f5fc90f7856d9fc699d4a5e79d968a1aef1a72");
        for logs run: kubectl -n default logs publish-run-pod-200eff -c build-step-build-push-base-images'
      status: "False"
      type: Succeeded

Thanks for the fix - so fast!!!! 🏎️