":Z" option for datastorage makes Quay not able to start

[PhilippeRaven]

Sorry for the title, here is a more complete explanation of the issue :

Conditions :
RHEL 8.8. Podman with rootless usage for Quay. SELinux in Permissive mode.
Config file for rootless podman storage (~/.config/containers/storage.conf) :

[storage]
driver = "overlay"
rootless_storage_path = "/srv/quay/.local/share/containers/storage" 

Running mirror-registry install with :
./mirror-registry install --initUser registry_user --pgStorage /srv/quay/pg-storage --quayHostname registry.domain.tld:8443 --quayRoot /srv/quay/ --quayStorage /srv/quay/

Install fails when waiting for Quay to be up and reachable, /var/log/messages shows this error from podman :
podman[12471]: Error: lsetxattr /srv/quay/.local/share/containers/storage/overlay/04a62cb87f5a3730dbce9dfdf02c40121014b062f215eb33a9b4bd6545bce7b6/merged/bin: operation not supported

It works when I set SELinux to disabled.
So I added context rules on some directories, set SELinux to permissive again, still failing.

The only way I found to make it work with Permissive mode is to remove the ":Z" option from the datastorage line of the file ~/.config/systemd/user/quay-app.service :
Before :
-v /srv/quay/:/datastorage:Z \

After :
-v /srv/quay/:/datastorage \

If you have an idea or need more informations, I'll be glad to provide !

[BadgerOps]

Hey @PhilippeRaven out of curiosity, what is the filesystem on /srv/quay ?

[PhilippeRaven]

Hey @PhilippeRaven out of curiosity, what is the filesystem on /srv/quay ?

Thanks for the reply :)
It's EXT4.

[BadgerOps]

@PhilippeRaven is it an NFS export that you're mounting? Also, if not, are you doing any mount options for the EXT4 filesystem?

The error you're seeing is most likely filesystem related, and is when selinux is trying to set attributes.

Most of the time when I've seen the above error, it is on an NFS mount, but could also be due to EXT4 mount options.

Hope that helps with some investigation steps!

[PhilippeRaven]

@PhilippeRaven is it an NFS export that you're mounting? Also, if not, are you doing any mount options for the EXT4 filesystem?

The error you're seeing is most likely filesystem related, and is when selinux is trying to set attributes.

Most of the time when I've seen the above error, it is on an NFS mount, but could also be due to EXT4 mount options.

Hope that helps with some investigation steps!

It's not an NFS export. It's a local disk dedicated to the /srv/quay mountpoint. No mount option apart from "defaults".

Well like you say, it's SELinux related, and it's even more obvious because when I disabled SELinux it works, but when it's in Permissive mode it doesn't work (that's weird).
And that's why I make rules dedicated to storage subdirectories in SElinux conf, to try to handle the problem, but nothing works.

So currently my only workaround is to remove the ":Z" option in the service file.

[BadgerOps]

Ok, this adds even more questions!

1. The error you're getting even in Permissive mode leads me to believe you've got a couple potential issues
   a. Your kernel / OS is not configured to use EXT4 with xattrs - unlikely, but possible. What is the OS, Kernel version(s)?
   b. More likely, but shouldn't cause the errors noted above, your directory permissions are messed up - you can validate by

ls -alZ  /srv/quay/

You're looking for unconfined_u or something along those lines. What you should see is something like:

drwx------. 2 quay quay sysadm_u:object_r:container_file_t:s0  <date> quay-config

or something similar, depending on your system.

You might try running restorecon -R /srv/quay to relabel the contents of that directory.

Also, checking selinux avc logs to see what errors are showing up there would be helpful too! (Sounds like you might have already checked that?)

Good luck!

[PhilippeRaven]

Ok, this adds even more questions!

1. The error you're getting even in Permissive mode leads me to believe you've got a couple potential issues
   a. Your kernel / OS is not configured to use EXT4 with xattrs - unlikely, but possible. What is the OS, Kernel version(s)?
   b. More likely, but shouldn't cause the errors noted above, your directory permissions are messed up - you can validate by

ls -alZ  /srv/quay/

You're looking for unconfined_u or something along those lines. What you should see is something like:

drwx------. 2 quay quay sysadm_u:object_r:container_file_t:s0  <date> quay-config

or something similar, depending on your system.

You might try running restorecon -R /srv/quay to relabel the contents of that directory.

Also, checking selinux avc logs to see what errors are showing up there would be helpful too! (Sounds like you might have already checked that?)

Good luck!

Sorry for the delay, and thanks for the answer !

So to answer your questions :

• It's a RHEL 8.8 with kernel 4.18.0-477.27.1.el8_8.x86_64 to be precise.
• I made lot of researches about the permissions and my persmissions are like you said. I even forced those permissions on all the subfolders of .local through the selinux files_contexts.local file and relabeled using the restorecon command.

About the logs, the only message I get is the one in my ticket :

podman[12471]: Error: lsetxattr /srv/quay/.local/share/containers/storage/overlay/04a62cb87f5a3730dbce9dfdf02c40121014b062f215eb33a9b4bd6545bce7b6/merged/bin: operation not supported

So I don't think you have more informations now :(

[BadgerOps]

Have you reached out via the Red Hat support portal yet? There might be a known issue around this deployment methodology. I noticed https://access.redhat.com/solutions/6985126 has a similar error message.

[PhilippeRaven]

I did not currently. Usually, I am the support :)
Thanks for pointing me this KT solution, I'll try to contact RH support on the next setup. Right now I just made the modification in the systemd service file to workaround.