selinux cap query - control_t and cache_t

[kevin-littlejohn]

Hiya,

Trying to start some containers during bootstrap time, as a way to ensure they're always running/started before everything else/not in with the normal orchestrated containers. It's failing because bootstrap appears to run as control_t in selinux, and the containers I'm running want to create a file in /run inside the container which is cache_t, and only runtime_s are allowed to write to cache_t.

Any chance the selinux policy could be tweaked to allow control_t to write to cache_t, or does that create other issues?

Fwiw, the container I'm wanting to run is an intrusion detection thing using eBPF, on a non-containerised host we'd just run it at host level, but on here it makes sense (to me, anyway) to run it privileged in the host containerd. If there's a better way to do this, also open to that., but the idea is to make it as simple as possible for people using bottlerocket, we just want to provide a bootstrap container and say "put this in your userdata.toml".

Thanks,

[arnaldo2792]

Hey, @kevin-littlejohn can you please clarify if your intrusion detection container is actually long-lived? Or, does the container load the eBPF program and exits? If the former, a host container would be best since long-lived bootstrap containers will block your boot.

As for the control_t/cache_t access, does your process attempt to create a file in /run in the host, or /run within the filesystem of the container?

[kevin-littlejohn]

The plan was to use the bootstrap container to launch the long-running ids container - so bootstrap effectively calls "nerdctl run" on the appropriate container. That means the ids has to run in the host-containerd not the orchestrated containerd, but I'm ok with that - if anything, it means it's a little less obvious that it's running, which is good.

I believe the container is attempting to create something in the container /run (this is a vendor container and it all happens within a binary, so I'm a little constrained on what I can see happening). But those files are part of an overlay that is marked cache_t I think. I seem to be able to hack around the /run restriction, but then it tries to write to another directory in it's own container and hits similar restrictions - so I'm playing whack-a-mole. Are containers run in host/bootstrap meant to be read-only?

[vyaghras]

Hi @kevin-littlejohn, is it possible for you to share user-data file? Also Can you share journalctl -k logs? Can you see any AVC denials in the logs when you are trying to create the file in /run.

Are containers run in host/bootstrap meant to be read-only?

No.

[larvacea]

This command will execute in a bootstrap container, and it does create the expected file in the host's /run:

touch /.bottlerocket/rootfs/run/my_test_file

Thus touch in the bootstrap container does have all the permissions it needs to create a file there. The problem you are experiencing may be that the container's file systems are not identical to the host's.

Here's what the path /run looks like from inside a host container (the admin container):

[root@admin]# ls /run
agetty.reload  api.sock  mount  sshd.pid  systemd  user

Here's the host's /run, using the path from that touch command:

[root@admin]# ls .bottlerocket/rootfs/run/
api.sock              cache   containerd   dbus    docker.pid   host-containerd  log         mdadm  netdog   tmpfiles.d  user
bootstrap-containers  chrony  credentials  docker  docker.sock  lock             machine-id  mount  systemd  udev        xtables.lock

And from a process running on the host, to demonstrate that the container's .bottlerocket/rootfs/run directory is the host's /run directory:

bash-5.1# ls /run
api.sock              cache   containerd   dbus    docker.pid   host-containerd  log         mdadm  netdog   tmpfiles.d  user
bootstrap-containers  chrony  credentials  docker  docker.sock  lock             machine-id  mount  systemd  udev        xtables.lock

The admin container is a superpowered host container, so one could do the same dance with an arbitrary host container.

[kevin-littlejohn]

userdata file just runs a container I've previously setup as bootstrap. That container tho, has a script that does some dodgy things, but fundamentally:

export CONTAINERD_ADDRESS=unix:///run/host-containerd/containerd.sock

# We have to run nerdctl in a chroot because of the pathing for the overlay files - it reaches for /var/lib/host-containerd
# and there doesn't seem to be a flag to change this.
mkdir -p /.bottlerocket/rootfs/mnt/bin
cp /usr/local/bin/nerdctl /.bottlerocket/rootfs/mnt/bin/nerdctl

# Temp home to store docker auth because /root is not writable
export HOME=$(chroot /.bottlerocket/rootfs mktemp -d)

# Login to ECR and pull the crowdstrike image
aws ecr get-login-password --region ap-southeast-1 | chroot /.bottlerocket/rootfs /mnt/bin/nerdctl login --username AWS --password-stdin $ACCTID.dkr.ecr.ap-southeast-1.amazonaws.com
chroot /.bottlerocket/rootfs /mnt/bin/nerdctl pull $CROWDSTRIKE_IMAGE

# directory and file we need for the crowdstrike container
chroot /.bottlerocket/rootfs touch /var/log/falconstore
chroot /.bottlerocket/rootfs mkdir -p /var/log/falcon-logs

# Start the crowdstrike container
chroot /.bottlerocket/rootfs /mnt/bin/nerdctl run \
    --address $CONTAINERD_ADDRESS \
    --privileged \
    --network=host \
    --cap-add=SYS_ADMIN \
    --cap-add=SYS_PTRACE \
    --cap-add=SYS_CHROOT \
    --cap-add=DAC_READ_SEARCH \
    -v /var/log/falconstore:/opt/CrowdStrike/falconstore \
    -v /var/log/falcon-logs:/var/log \
    --entrypoint /opt/CrowdStrike/falcon-daemonset-init \
    $CROWDSTRIKE_IMAGE -i
chroot /.bottlerocket/rootfs /mnt/bin/nerdctl run \
    --address $CONTAINERD_ADDRESS \
    --privileged \
    --network=host \
    --cap-add=SYS_ADMIN \
    --cap-add=SETGID \
    --cap-add=SETUID \
    --cap-add=SYS_PTRACE \
    --cap-add=SYS_CHROOT \
    --cap-add=DAC_OVERRIDE \
    --cap-add=SETPCAP \
    --cap-add=DAC_READ_SEARCH \
    --cap-add=BPF \
    --cap-add=PERFMON \
    --cap-add=SYS_RESOURCE \
    --cap-add=NET_RAW \
    --cap-add=CHOWN \
    --cap-add=NET_ADMIN \
    -v /var/log/falconstore:/opt/CrowdStrike/falconstore \
    -v /var/log/falcon-logs:/var/log \
    -v /run:/run \
    $CROWDSTRIKE_IMAGE

Given that bootstrap containers aren't meant to be read-only, most of this shouldn't be needed, but I'm getting the following:

Feb 18 01:56:33 ip-10-53-18-21.ap-southeast-1.compute.internal audit[1432]: AVC avc: denied { write } for pid=1432 comm="falcon-sensor" name="CrowdStrike" dev="overlay" ino=12634894 scontext=system_u:system_r:control_t:s0-s0:c0.c1023 tcontext=system_u:object_r:cache_t:s0 tclass=dir permissive=0

This is an attempt by the falcon binary to write to /opt/CrowdStrike within the container. I previously got similar errors for /opt/CrowdStrike/falconstore and /var/log and /run, which is why those are mounted above.

[arnaldo2792]

Hey @kevin-littlejohn, given your use case it seems like host containers are a better fit for what you are trying to accomplish (run a privileged container prior to the orchestrated workloads). May I ask what limitations host containers have that prevented you from using them for your need?

To give you an idea of what's going on, since you are chroot ing to the host namespaces the nerdctl command will end up running as control_t in the host. AFAIK, nerdctl will try to set up part of the container's filesystem (since it uses containerd's smart client). Since nerdctl doesn't support SELinux, it might not be setting up the correct label for the filesystem, so when the nerdctl command runs as control_t it fails to write to the final filesystem because the filesystem wasn't labeled properly by nerdctl.

[kevin-littlejohn]

Is there a way to mount a host directory to a host container directory? The vendor container wants to cache an agent ID (effectively a host ID) in a host directory, needs that dir mounted, and is failing because of that.

[arnaldo2792]

Hey @kevin-littlejohn, I see you are trying to use the falcon-sensor but it is behind a paywall so it is hard for me to debug what directory is it trying to write to. If you want, you can cut us a support ticket and ask the support folks to direct the ticket to the Bottlerocket maintainers to expedite the help. On a different note, have you try to use the falco-helm charts? If you are using the kubernetes variants, that should be the way to go, unless your use case prevents you from using the helm charts.

To clarify a few things, the falcon-sensor as a host container should be able to write to directories in the container's filesystem, so if it is trying to write to /run, /var/run or /tmp it should succeed:

# This is the admin host container
[root@admin]# touch /run/test
[root@admin]# touch /tmp/test
[root@admin]# touch /var/run/test1
[root@admin]#

It is odd that the falcon-sensor explicitly requires to write to a host directory, it really shouldn't unless it has to share that host ID with other parts of the system, in which case it does make sense for it to write to a host directory. superpowered host containers have access to Bottlerocket's filesystem at /.bottlerocket/rootfs/ but I don't know if you can point the falcon-sensor to store artifacts at a location of your choice.