multipathd fails to start after 3139.2.3 upgrade to 3227.2.0 #809
Labels
kind/bug
Something isn't working
Comments
|
full log |
|
ldconfig cache is out of date, and needs an update after the "/lib"/"/lib64" split. I'll check if this is the only issue and then see how to get this done during/after the update. |
jepio
added a commit
to flatcar/scripts
that referenced
this issue
Jul 25, 2022
This made no difference back when lib was a symlink to lib64, but now that they are separate, libs belongs in /usr/lib64. This mostly doesn't show up because ldconfig configures the ld.so cache to include both locations, but when updating from an older release ld.so.cache is out of date. Unfortunately ld.so.cache does not get updated until after multipathd, which causes multipathd to dump core. This may also affect other packages that need access to libgcc early. See also: flatcar/Flatcar#809
1 task
8 tasks
jepio
added a commit
to flatcar/scripts
that referenced
this issue
Jul 27, 2022
This made no difference back when lib was a symlink to lib64, but now that they are separate, libs belongs in /usr/lib64. This mostly doesn't show up because ldconfig configures the ld.so cache to include both locations, but when updating from an older release ld.so.cache is out of date. Unfortunately ld.so.cache does not get updated until after multipathd, which causes multipathd to dump core. This may also affect other packages that need access to libgcc early. See also: flatcar/Flatcar#809
jepio
added a commit
to flatcar/scripts
that referenced
this issue
Jul 27, 2022
This made no difference back when lib was a symlink to lib64, but now that they are separate, libs belongs in /usr/lib64. This mostly doesn't show up because ldconfig configures the ld.so cache to include both locations, but when updating from an older release ld.so.cache is out of date. Unfortunately ld.so.cache does not get updated until after multipathd, which causes multipathd to dump core. This may also affect other packages that need access to libgcc early. See also: flatcar/Flatcar#809
jepio
added a commit
to flatcar/scripts
that referenced
this issue
Jul 27, 2022
This made no difference back when lib was a symlink to lib64, but now that they are separate, libs belongs in /usr/lib64. This mostly doesn't show up because ldconfig configures the ld.so cache to include both locations, but when updating from an older release ld.so.cache is out of date. Unfortunately ld.so.cache does not get updated until after multipathd, which causes multipathd to dump core. This may also affect other packages that need access to libgcc early. See also: flatcar/Flatcar#809
|
This is a blocker for our on-prem cluster as well. Is there a workaround that would allow us to upgrade to the latest flatcar image, while we wait for fixes to be merged into a new release? |
|
Create this file before rebooting This should work as a workaround, normally ldconfig will run during boot after an update but it runs after multipathd starts, and there would be an ordering cycle if one were to add a dependency between We plan to have a bugfix release out around next week. |
|
tyvm @jepio ! |
|
Fix has been released, I guess we can close this issue. Thanks @defo89 for the initial report! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
When 3139.2.3 is upgraded to 3227.2.0,
multipathdservice fails to start.Flatcar deployed with OVA template to vSphere 7.0.3.
Impact
Server is available only after ~50 minutes after reaching the timeout to start multipathd.
Environment and steps to reproduce
ignition-v2 file to reproduce:
3227.2.0and rebootAfter ~50 minutes server boots up with failed service:
Journal shows attempts to start the service (during those server is not accessible):
Additional information
The issue occurs only in the upgrade case.
Issue does not occur if server is deployed directly with 3227.2.0.
Looking at differences before/after the upgrade, here is what I was able to gather.
3139.2.3:
3227.2.0:
Service will run successfully after manual start (
systemctl start multipathd) or server reboot. Main concern is prolonged server downtime during the upgrade and possibility to avoid such occurrences in the next upgrade (for this or any other service).The text was updated successfully, but these errors were encountered: