Cannot build app with podman and pack v0.32+

[matejvasek]

Summary

Building an app using pack and podman fails with message:

ERROR: failed to build: failed to write image to the following tags: [pack.local/builder/71706f696b6577696675:latest: expected config hash "01940b90b5c5a92125b3eabce41d6f2824bf65bcb8c64c50944e437bd2ede1f8"; got "7c7bff10088f1a8fff50a26d68f1e52eb621296146867c6324e3ddfd50741189"]

The regression seems to be introduced in buildpacks/imgutil@67824e9 before it worked.

Reproduction

Steps

Current behavior

Expected behavior

---

Environment

pack info

docker info

[matejvasek]

/cc @natalieparellano

[natalieparellano]

@matejvasek thanks for this. I think we have a couple options here:

1. Revert to old imgutil; the "fix" I introduced is more of a hack and we hope to solve it more cleanly via other mechanisms
2. Remove this check as it seems too brittle

@jabrown85 do you have any thoughts here?

[jabrown85]

I'd be curious to cut a version without the the validate and see if the image is as expected. Can you share your podman information? Does this happen on fresh builds or re-builds?

[matejvasek]

When I neutralize the validateInspect() then the build proceeds and it looks like the app image is OK (I tested only Go Hello World program).

[matejvasek]

The error happens even on supposedly clean build (new image name).

[matejvasek]

podman system service unix:///tmp/docker.sock --log-level=debug --time 0

export DOCKER_HOST=unix:///tmp/docker.sock
pack build applications/golang2 -Bghcr.io/knative/builder-jammy-tiny:latest --docker-host=inherit --trust-builder=1 --path=/tmp/tmp.UVLH8A9F24

[jjbustamante]

I will suggest option 2 and remove the validateInspect, I think we didn't do before and everything was working with podman

[natalieparellano]

We used to do the check with inspect, _, err := i.docker.ImageInspectWithRaw(context.Background(), id) where id := fmt.Sprintf("%x", sha256.Sum256(configFile)) and configFile is the file we generated. So in theory, the check should be equivalent but somewhere it's gone wrong...

[natalieparellano]

IMO it would be safer to just roll back imgutil until we can understand what changed.

[matejvasek]

What two digests are being compared? One in daemon and another in tar, or what is being compared?

[matejvasek]

What exactly is happening? Are we crafting image tar that is subsequently loaded into daemon?

[natalieparellano]

@matejvasek we used to compare the digest of the config file that we loaded to the image ID returned by inspect. I believe this is to protect against collisions, in case two image builds with the same tag are happening in parallel. When we added the workaround for containerd storage, we stopped using the image ID in the comparison, because the ID when containerd storage is used is the digest of the manifest. Instead, we compute the digest again and check that it matches what we sent in.

[matejvasek]

@natalieparellano I was not watching this for some time, is there any progress on this issue?

[natalieparellano]

@matejvasek I think we just need to remove the digest comparison in imgutil. It should be a quick fix.

[natalieparellano]

This should be fixed in buildpacks/imgutil#239 - I believe all that is left is to update the imgutil version in pack (cc @jjbustamante)

[jjbustamante]

This should be fixed in buildpacks/imgutil#239 - I believe all that is left is to update the imgutil version in pack (cc @jjbustamante)

I will create a PR to bump the imgutil version 😄