For Apple M1, binary substitution breaks codesign

[bobot]

In #4389 and #4135 problem between dune substitutions and codesign has been raised. But unfortunately both times another bug was also the culprit. It is time for dune to call codesign -s - $binary automatically after substitution.

• applying only for architecture = arm64 and system = macosx

However I don't know on which files it should be applied:

• should we do it only for substituted files for which the source is verfied (codesign -v <path>)
• for all executables?

Also the CI doesn't check that? Could we test on M1?

@correnson

[bobot]

esy seems to use this test:

https://github.com/esy/esy/blob/2a863c5529ea20b45bca12881d351fc189d0e3c3/esy-build-package/Build.re#L384-L388

[correnson]

I can confirm the bug:

• the freshly build or promoted binary is always correctly signed;
• if no relocation is performed by dune, the installed binary is OK;
• the installed relocated binary is no more signed for macOS M1, be it promoted or not.

I think the esy methods is OK for determining whether to sign or not: they check the platform by running uname, which returns Darwin for macOS and the architecture by running uname -m, which returns x86_64 on i86 and arm64 on M1.

[bobot]

What is returned by ocamlopt -config | grep -e system -e architecture on both system?

[correnson]

Using ocamlopt -config seems also ok, relevant fields are architecture and system:

architecture: amd64 | arm64
system: macosx

[bobot]

However I don't know on which files it should be applied:

• should we do it only for substituted files for which the source is verfied (codesign -v <path>)
• for all executables?

[correnson]

I think only (and systematically) for executables that are rewritten on-the-fly by dune. Others are correctly signed by the compiler.

[toots]

+1 here, got hit by the issue today.

[Blaisorblade]

I think I'm hit by this issue as well, and it interferers with developing dune itself...

[toots]

Any update on this? This hit us again in dev and will be problem releasing our next major version. Thanks y'all!

[emillon]

I don't have a mac to test this, but here are some notes if someone wants to give it a go (or verify a PR):

• I think that the piece of code to modify is Artifact_substitution.copy_file: after the call to copy_file_non_atomic, insert a call that signs the temporary file
• signing is a noop unless we're on arm and at least big sur (technically this applies to the target toolchain, but we don't have a way codesign when crosscompiling anyway).
• esy's test is good enough to start but I wonder if there's a platform API we can call to determine whether codesigning is mandatory on a given version of mac os
• the codesign commandline used by esy is here, we can use that
• my understanding is that the error message described in that file applies when rewriting is done in-place. in that case the kernel's cache mapping inodes to signatures gets confused. we're not affected since we're already doing the rewriting to a temporary file
• there's the question of which identity to use for signing. esy uses - which corresponds to ad-hoc signing. this works I wonder if we can use the same identity as the one in the original file. What is the identity used in the pristine binaries? does codesign -v (or -vv) display it?

[Et7f3]

Is their a test/repoduction that show the failure ?

[emillon]

I don't think so. We don't have M1 CI so the test would not fail in CI, but it's a good first step if there's no test that attempts to run a substituted binary for now.

[Et7f3]

M1 CI doesn't seem to be a thing yet actions/runner-images#2187 so the first step is to add a substitute binary test ?

[emillon]

Yes I think that this would be important to add.

[Blaisorblade]

but it's a good first step if there's no test that attempts to run a substituted binary for now.

Not to dissuade you from your plans, but I think such tests must already exist — last I tried, quite a few tests segfaulted on generated binaries; I confirmed that manually running codesign* on those binaries let them run correctly.

*It must have been something simple like codesign -s -, as I didn't setup any custom identity.

[correnson]

• Don't need a CI here, using codesign - definitely works (this is what we do manually to fix the issue)
• For identification via commands, uname = Darwin and uname -m = arm64 identifies a mac M1.
• For identification via ocaml config, system: macos and architecture: arm64 identifies a mac M1.

[emillon]

thanks, it's good to know that the current test suite already catches that on M1. can anyone paste the output of codesign -v and codesign -vv on a pristine binary produced by ocamlopt?

[Alizter]

Regarding M1 runners, Github does have support for them you just have to provide them yourself I think. We could ask OCamllabs or whoever appropriate for some. cc @rgrinberg

[toots]

Thanks for looking at this y'all. I'll try to give it a pass when I get some time back on the M1.

I believe that binaries generated during a local opam build are not really intended to be distributed outside of the local machine at this would require much more work with dynamically linked libraries, binary hardening, etc. so self signing sounds like a pretty acceptable solution.

I'd also caution that this might not be only for M1 lacs only in the future but any Mac with a recent enough macos.

[emillon]

Can anyone try #6137 to see if it works? Also still interested in the output of codesign -v / -vv for a normal ocaml binary. Thanks!

[samoht]

Without #6137

❯ dune exec -- cohttp-lwt-unix/bin/cohttp_curl_lwt.exe --help
❯ codesign -v _build/default/cohttp-lwt-unix/bin/cohttp_curl_lwt.exe
❯ codesign -vv _build/default/cohttp-lwt-unix/bin/cohttp_curl_lwt.exe
_build/default/cohttp-lwt-unix/bin/cohttp_curl_lwt.exe: valid on disk
_build/default/cohttp-lwt-unix/bin/cohttp_curl_lwt.exe: satisfies its Designated Requirement

[samoht]

@emillon is this what you are looking after?

❯ ocamlopt foo.ml -o foo
❯ ./foo
❯ codesign -v ./foo
❯ codesign -vv ./foo
./foo: valid on disk
./foo: satisfies its Designated Requirement
❯ cat foo.ml
let () = ()

[emillon]

thanks. can you attach the output of codesign -vvvv as well? I'd like to see if it can print the identity it's been signed with

[samoht]

It was tricky to find but the magic argument is --dispay :-)

❯ codesign --verbose=4 --display foo
Executable=/Users/thomas/git/cohttp/foo
Identifier=foo
Format=Mach-O thin (arm64)
CodeDirectory v=20400 size=3324 flags=0x20002(adhoc,linker-signed) hashes=101+0 location=embedded
VersionPlatform=1
VersionMin=786432
VersionSDK=787200
Hash type=sha256 size=32
CandidateCDHash sha256=9ac8f2b102c6f2b8dafc777574099063f0e209f9
CandidateCDHashFull sha256=9ac8f2b102c6f2b8dafc777574099063f0e209f97f21d7afe4986451a0c30ab3
Hash choices=sha256
CMSDigest=9ac8f2b102c6f2b8dafc777574099063f0e209f97f21d7afe4986451a0c30ab3
CMSDigestType=2
Executable Segment base=0
Executable Segment limit=196608
Executable Segment flags=0x1
Page size=4096
CDHash=9ac8f2b102c6f2b8dafc777574099063f0e209f9
Signature=adhoc
Info.plist=not bound
TeamIdentifier=not set
Sealed Resources=none
Internal requirements=none

[emillon]

Great! so this confirms that this is an adhoc signature as well. so we don't have to be more clever about that part. thanks!

[toots]

Can anyone try #6137 to see if it works?

This seems to work here!

[toots]

Dope, thank you all for working on this!