[RFE] Handle configuration updates in /etc

[krnowak]

Currently we modify the ebuilds of the packages that are putting files into /etc by telling them to put them somewhere in /usr and to install tmpfiles.d file that will tell systemd-tmpfiles to copy those files from /usr to /etc or create symlinks. The reason for it is that /etc is not updated when the OS update is performed. Instead, systemd-tmpfiles is invoked on every boot.

The plan is to allow the ebuilds to put files in /etc. The image building process could later move those files into /usr/flatcar-etc. The newly-built image's /etc would be a copy of /usr/flatcar-etc. In case of OS update, we would want to do a three-way merge of old /usr/flatcar-etc, new /usr/flatcar-etc and /etc. The merge would need to happen during boot, likely before switching root from initramfs to new sysroot.

In case when we want to put some extra configuration, we can create our own package that will contain those files. In case where we do not want some config from a package to be installed, we could use INSTALL_MASK.

The /usr/flatcar-etc is obviously not a name set in stone. It could be very well /usr/share/flatcar-etc or else. Still needs to be below /usr.

What needs figuring out is a way to make current systemd-tmpfiles way of things to cooperate with /usr/flatcar-etc way of things, so packages can be converted incrementally, so at some point phasing out the systemd-tmpfiles way of things would be a non-event.

It's a part of #96.

[krnowak]

The initial plan could be:

• Phase 1 (the "simplify installation of configuration to /etc from ebuilds" phase):
  • Copy /etc to /usr/share/flatcar-etc when building the image.
  • Add a unit running at boot (at around the same time as systemd-tmpfiles-setup.service) that would run systemd-tmpfiles on files in /usr/share/flatcar-etc-tmpfiles.
    • In the beginning this would be the no-op, as the directory will be empty.
  • Allow packages to install their config files to /etc.
  • Move the installation of tmpfiles from the package to coreos-base/flatcar-configs package:
    • Change those tmpfiles to copy stuff or symlink from /usr/share/flatcar-etc.
    • Put those tmpfiles in /usr/share/flatcar-etc-tmpfiles.
    • Maybe at this point some packages could be moved back to portage-stable.
  • For configs that we do not want to install, use INSTALL_MASK.
  • For configs that we do want to install or override, put them in coreos-base/flatcar-configs package.
    • The package probably would be installed after other packages were installed to avoid the interference of the INSTALL_MASK setting.
    • Or we could have /etc/share/flatcar-etc-overrides, where this package would install directly (thus sidestepping the INSTALL_MASK on /etc/). The contents of this directory would be copied to /etc/share/flatcar-etcand/etc` and directory would be removed after that.
• Phase 2 (the "handle the /etc configuration updates on OS updates" phase):
  • Implement 3-way merge and drop the unit from phase 1.
    • Both of those things would probably be a part of bootengine.
  • Maybe move, instead of copying, /etc to /usr/share/flatcar-etc when building.
    • /etc could be constructed from scratch from /usr/share/flatcar-etc on first boot?
  • Drop the tmpfiles from flatcar-configs package.

[pothos]

On Phase 1: I'm a bit confused about the packages and their tmpfiles directives. I would say that instead of using systemd-tmpfiles in the new service unit we could look at what is in /usr/share/flatcar-etc and copy these files to /etc if they are not there yet - the other way is to generate a list of these files at image build time and generate a tmpfile config from it as done already for the directories. It should work to keep the existing tmpfile directives in the downstream patches and gradually remove them.
I would also add delete /etc from the rootfs in the image build to make sure we test that we don't rely on files in it.

On Phase 2: We can start with a simplified version that be able to remove files and check if they are changed or not so that they get a fresher copy from the new "/usr". For that I would copy /usr/share/flatcar-etc over to /current-flatcar-etc if differs and move the existing /current-flatcar-etc to /old-flatcar-etc, then perform the simplified merge (keep user's file if changed by user) and remove /old-flatcar-etc. The 3-way merge would be the next iteration on this (because it's a bit tricker to get right I think but we have the possibility to do the diff between old-flatcar-etc and user's etc, and the diff between old-flatcar-etc and current-flatcar-etc).

[krnowak]

On Phase 1: I'm a bit confused about the packages and their tmpfiles directives. I would say that instead of using systemd-tmpfiles in the new service unit we could look at what is in /usr/share/flatcar-etc and copy these files to /etc if they are not there yet - the other way is to generate a list of these files at image build time and generate a tmpfile config from it as done already for the directories. It should work to keep the existing tmpfile directives in the downstream patches and gradually remove them.

I'd say that

we could look at what is in /usr/share/flatcar-etc and copy these files to /etc if they are not there yet

is basically what systemd-tmpfiles is doing with our tmpfiles already, so I'd leverage it as long as we could until we get to the phase 2 where we drop it. And about:

the other way is to generate a list of these files at image build time and generate a tmpfile config from it as done already for the directories

Instead of doing it, I'd reuse what we already have. So at this point we can just slightly modify them instead of doing all the work again.

Basically the point of phase 1 would be not to change the outcome, but prepare for the phase 2 and also start simplifying the ebuilds.

I would also add delete /etc from the rootfs in the image build to make sure we test that we don't rely on files in it.

Yeah, I wanted to do in in second phase to avoid changing the outcome in the phase 1.

On Phase 2: We can start with a simplified version that be able to remove files and check if they are changed or not so that they get a fresher copy from the new "/usr". For that I would copy /usr/share/flatcar-etc over to /current-flatcar-etc if differs and move the existing /current-flatcar-etc to /old-flatcar-etc, then perform the simplified merge (keep user's file if changed by user) and remove /old-flatcar-etc. The 3-way merge would be the next iteration on this (because it's a bit tricker to get right I think but we have the possibility to do the diff between old-flatcar-etc and user's etc, and the diff between old-flatcar-etc and current-flatcar-etc).

We could do it, sure. It depends if I'll be able to find a ready-to-use solution for doing the 3-way merge. If there is one, then we could skip doing the simplified version.

[pothos]

This is what I meant for phase 1 and generating the tmpfile config:
flatcar/scripts#264

This should take care of being able to use upstream ebuilds.

[pothos]

This is done now through an overlayfs mount and should be part of the next major Alpha