lacks a signature by a trusted key, when targetUser is other than root

[joshuacox]

colmena works great until one of my hosts has developed a "because it lacks a signature by a trusted key" error. (of note this randomly happens to one or more hosts using the alternate targetUser, but not all of them which is very inconsistent).

Performing a colmena upload-keys completes without error.

How do I force that host to update its keys, or how do I debug as to what is going wrong?

WORKAROUND: I had been using an alternate targetUser, changing this back to root enabled this host to pass the tests. However, all of my other hosts are still using the alternate user, this does seem to be a bug of some sort.

[qenya]

This happens occasionally for me too. I've discovered that if I colmena apply --build-on-remote once for the problematic host, it eliminates the error and I can build locally and apply again without issues. (Obviously this isn't feasible for all hosts - your workaround is probably more reliable.)

My suspicion is that it is caused by a version mismatch of something between the local machine and the remote - next time it happens I will attempt to isolate it.

[joshuacox]

ya I have since moved all back to using root in this particular cluster.

However, I'd be happy to lend a hand and test anything I can, or anything else I can to do to help hunt down the problem.

[bmwagner18]

I'm also having this issue. I've been getting by using --build-on-target, but I would like to figure out the root cause. If there's anything I can do to help diagnose this please let me know, because the errors basically mean nothing to me...

[DerRockWolf]

I'm not (yet) using colmena, but the "because it lacks a signature by a trusted key" error happens when the closures are copied to the target nix store (can also be reproduced with nix copy) and the targetUser not included in the target hosts nix-daemon trusted-users list. root is a trusted user per default.

Note that:

Adding a user to trusted-users is essentially equivalent to giving that user root access to the system. For example, the user can access or replace store path contents that are critical for system security.

Alternatively you can sign the store paths with a private key that the target host trusts. But colmena can't do this automatically.

See nix store sign & nix store verify or take a look at this small doc I've wrote

[plxty]

Generate key with nix-store --generate-binary-cache-key key-name secret-key-file public-key-file , then set services.nix-serve.secretKeyFile in host and nix.settings.trusted-public-keys in remote solves it (by signing the closure).

For deploying the trusted public key, you may need to change the /etc/nix/nix.conf to add the content of public-key-file to trusted-public-keys.

[DerRockWolf]

Generate key with nix-store --generate-binary-cache-key key-name secret-key-file public-key-file , then set services.nix-serve.secretKeyFile in host and nix.settings.trusted-public-keys in remote solves it (by signing the closure).

For deploying the trusted public key, you may need to change the /etc/nix/nix.conf to add the content of public-key-file to trusted-public-keys.

This is more a workaround, or?

I suppose that the nix-serve service also needs to be enabled and put in nix.settings.substituters on the target?
This has the ugly side effect where all substitutions will fail if the host running nix-serve is offline.

[plxty]

This is more a workaround, or?

I think it's the most appropriated way for what the nix is designed, the remote node should take security into consideration.

I suppose that the nix-serve service also needs to be enabled and put in nix.settings.substituters on the target? This has the ugly side effect where all substitutions will fail if the host running nix-serve is offline.

Hmmm I'm actually using another way, is to patch the colmena to have a colmena apply --sign <private_key> option to sign the closure automatically, https://github.com/plxty/n9/blob/805d71a06473d6ca53e7789e90e32453043bba94/patches/colmena-nix-store-sign.patch#L188 instead of spwaning a nix-server, it may not be merged as it only works with nix3 command.