Improve package layering UX

[jlebon]

This is a follow-up to #401. Fedora CoreOS now ships with an archive repo which allows reliable package layering operations. However, there is still work remaining to make it easier to use. Right now, users have to create a systemd unit which runs rpm-ostree install --reboot ... which is not great.

Specifically:

1. we need a better interface for package layering via Ignition, whether that be via file drop-ins translated by FCOS glue code, or drop-ins read by rpm-ostree directly (related: rethinking origins rpm-ostree#2326)
2. we want "Ignition-like" semantics for the extensions: this means that the extensions ideally should be present from the get go in the real root on the first boot (or we should fail the boot if something goes wrong)
3. we need an FCC-friendly way to specify these packages, either directly having users use the file drop-in interface or via sugar

The sugar that was proposed in #401 was something like:

extensions:
- usbguard

[jlebon]

we want "Ignition-like" semantics for the extensions: this means that the extensions ideally should be present from the get go in the real root on the first boot (or we should fail the boot if something goes wrong)

So one way to do this is to call out to rpm-ostree before switchroot to do the install and then doing livefs (though I think we can do even better here since the sysroot isn't technically in use yet -- we'd need to enhance rpm-ostree for that).

A first cut would be to just reboot for now, the way we will do for kargs. This won't work for live ISO/PXE though.

[jlebon]

For #767, note that these improvements should keep modular extensions in mind.

[LorbusChris]

Just an idea: Afaik, *COS extensions today aren't something that RPM is aware of.
I wonder whether we could re-use Groups defined in the spec for this (although I believe Groups have been deprecated in RPM in Fedora)

[jlebon]

Another path that came up in today's community meeting related to this is embedding packages at install time so that they can be layered offline on first boot. See #862 (comment).

[alvarlagerlof]

Has there been any progress on this?

[dustymabe]

As an intermediate step to having Ignition support doing package layering in the initramfs we could take what is in the docs and modify it slightly and add it to the host. Then users could define their own RPMs in a dropin. Here's the full example:

variant: fcos
version: 1.3.0
storage:
  files:
    - path: /etc/systemd/system/rpm-ostree-install.service.d/rpms.conf
      mode: 0644
      contents:
        inline: |
          [Service]
          Environment=RPMS="foo bar baz"
systemd:
  units:
    - name: rpm-ostree-install.service
      enabled: true
      contents: |
        [Unit]
        Description=Layer additional rpms
        Wants=network-online.target
        After=network-online.target
        # We run before `zincati.service` to avoid conflicting rpm-ostree transactions.
        Before=zincati.service
        ConditionPathExists=!/var/lib/%N.stamp
        [Service]
        Type=oneshot
        RemainAfterExit=yes
        ExecStart=/usr/bin/rpm-ostree install --apply-live --allow-inactive $RPMS
        ExecStart=/bin/touch /var/lib/%N.stamp
        [Install]
        WantedBy=multi-user.target

The proposal is that rpm-ostree-install.service is shipped in the base layer of FCOS and the user only has to create the dropin with the RPMs they want to layer in their Butane/Ignition config.

Of course we can also add some sort of ConditionPathExists= to make sure it only ran when users really wanted layered RPMs.

[xanderificnl]

Feel free to ignore the message below. I realized this was quite inefficient since I was rebooting anyway so I've modified my approach to mirror the one above.

---

I was having some issues with rpm-ostree seemingly being killed/locked files (/etc/passwd) etc. There is already an issue about that somewhere on this issue tracker.

For now I've settled on using this for deployments:

• prevents double running rpm-ostree-installer@'s
• reboots after every install (considering it's only intended for new deployments, I felt it was acceptable) to prevent the /etc/passwd lockings

variant: fcos
version: 1.5.0
storage:
  directories:
    - path: /var/cache/rpm-ostree-install
systemd:
  units:
   - name: [email protected]
      enabled: true
      contents: |
        [Unit]
        Description=Layer %i with rpm-ostree
        Wants=network-online.target
        After=network-online.target
        Before=zincati.service
        ConditionPathExists=!/var/cache/rpm-ostree-install/%i.stamp

        [Service]
        Type=oneshot
        RemainAfterExit=yes
        ExecStart=/usr/bin/flock /var/cache/rpm-ostree-install/.lock -c "/usr/bin/rpm-ostree install --assumeyes --idempotent --reboot --allow-inactive %i | tee /var/cache/rpm-ostree-install/%i.stamp"

        [Install]
        WantedBy=multi-user.target
    - name: [email protected]
      enabled: true
    - name: [email protected]
      enabled: true
    - name: [email protected]
      enabled: true

[Mossman1215]

👋 hi all,
I've been experimenting with fcos and finding it really cool.
the gap I'm trying to fill is core os install plus 2-3 packages which falls below the level of effort to making a respin distro.
Having a service file works but seems unusual compared to other first boot systems like kickstart/anaconda or cloud-init which can install packages on first boot as part of their initialization routines.

[usp-npe]

Any update on this? It would be really great to have an easier way to auto-install packages on CoreOS without having to create systemd units, and to reduce the size and complexity of the butane YAML files

[lukasbestle]

I've been trying to use the documented setup that uses a systemd unit on first boot, but in ~10 tries it only succeeded 2 times, the rest of the attempts failed with Could not resolve host: mirrors.fedoraproject.org (even though the unit is set up to be run After=network-online.target). It just feels like a hack, so I have given up on it for now.

It would be really awesome to have robust support for layering right in Ignition and thus Butane.

[navaati]

Hi. Yup, can confirm I’ve had that weird networking problem like that too. I don’t fully remember, but I think it was DNS related: you’ve got connectivity at the IP level, but some DNS caching in systemd-resolved is hitting you… or something. Not sure, it was a while ago.

Anyway, here is the line I added to the unit to make it work, just before the ExecStart=/usr/bin/rpm-ostree line:

      ExecStart=sh -c 'while ! ping -c 1 mirrors.fedoraproject.org; do sleep 1; done'

Hacky, sure 🤷… Hope that helps someone !

[lukasbestle]

Thanks, that makes the hack a bit more hacky but might actually work. :)

[lukasbestle]

Another reason that makes these "systemd unit on first boot" hacks to hacks: If a system with layered packages is reprovisioned but the /var filesystem is kept, the stamps will still be there, but the packages won't be. Ignition lacks a way of deleting files, so the stamps need to be deleted manually before the reprovisioning.

[navaati]

Hi CoreOS team. Is the following (from here) still the vision for you ?

we want "Ignition-like" semantics for the extensions: this means that the extensions ideally should be present from the get go in the real root on the first boot (or we should fail the boot if something goes wrong)

So one way to do this is to call out to rpm-ostree before switchroot to do the install and then doing livefs (though I think we can do even better here since the sysroot isn't technically in use yet -- we'd need to enhance rpm-ostree for that).

A first cut would be to just reboot for now, the way we will do for kargs. This won't work for live ISO/PXE though.

It seems ideal to me, and as you can see in this thread the current "boot, rpm-ostree, reboot" does not feel like an actual solution. Package layering, despite not fitting in the vision of "everything container", is important and is needed for some usecases that still really make sense with Fedora CoreOS rather than Fedora Server: for example hypervisors (libvirt) or HPC (drivers).

I’m asking because users are now at a weird crossroads between "Fedora Server + bootc" or "Fedora CoreOS with Ignition", but there is no "Fedora CoreOS + bootc" (AFAIK), and I’d rather use Fedora CoreOS for a variety of reasons.

So what is the vision here: is Ignition still the way and layering from initramfs before switchroot is actually planned, or will bootc get adapted to Fedora CoreOS and will be the recommended solution for customization ?

Thanks for your efforts in any case !

[jlebon]

or will bootc get adapted to Fedora CoreOS and will be the recommended solution for customization ?

Fedora CoreOS currently ships both bootc and dnf. Users today can derive from FCOS images and add their own packages and deploy that. However, nodes deployed this way may encounter some issues as they are not guaranteed to follow the update graph. In practice, as long as you ensure that you build your layered images on every FCOS release, and you keep your nodes up to date, you're unlikely to hit issues. We're working on better tooling to support server-side layering. Stay tuned for more in this area.

That said, there is still a use case for client-side package layering a-la-rpm-ostree. There are ongoing discussions on how to support that in the new bootc+dnf world in https://gitlab.com/fedora/bootc/tracker/-/issues/4.

[jbtrystram]

Note that we are also looking into system extensions to support some pre-defined set of packages, and those can be installed/removed without rebooting.

[jlebon]

This issue was filed before container layering was a thing. Now, a lot of users will be satisfied with the UX provided by that: build a new container image with your packages, and roll it out. In that case, a good FCOS-level UX is about how to write the seed Ignition/Butane to have bootc immediately switch over to the built container image. This is tracked in #1611.

So then, let's keep this ticket scoped to having a good UX for client-side layering. There, we currently are discussing two approaches:

• dnf package layering
• sysexts, ideally managed by bootc

In either case, we should make sure that there is a nice declarative interface to activate it. Then a user would just write e.g. that dropin via Butane, or we could add sugar to reduce friction even more.