Cloudstack Agent Java IO exception

[midhunpjos]

problem

Network Actions,VM Deployments,router health checks are failing.
On cloudstack agent logs below is printed

2025-02-06 09:14:09,343 WARN [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-4:[]) (logid:64df14bf) Exception [Cannot run program "/usr/share/
cloudstack-common/scripts/network/domr/router_proxy.sh": error=0, Failed to exec spawn helper: pid: 605941, exit value: 1] occurred when attempting to run command [/usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh filesystem_writable_check.py 169.254.156.203 ]. java.io.IOException: Cannot run prog
ram "/usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh": error=0, Failed to exec spawn helper: pid: 605941, exit value: 1
at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1143)
at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1073)
at com.cloud.utils.script.Script.execute(Script.java:232)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeInVR(LibvirtComputingResource.java:553)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeInVR(LibvirtComputingResource.java:541)
at com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.checkRouterFileSystem(VirtualRoutingResource.java:419)
at com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.execute(VirtualRoutingResource.java:391)
at com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.executeQueryCommand(VirtualRoutingResource.java:227)
at com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.executeRequest(VirtualRoutingResource.java:129)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtNetworkElementCommandWrapper.execute(LibvirtNetworkElementCommandWrapper.java:35)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtNetworkElementCommandWrapper.execute(LibvirtNetworkElementCommandWrapper.java:29)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1945)
at com.cloud.agent.Agent.processRequest(Agent.java:686)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1109)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.io.IOException: error=0, Failed to exec spawn helper: pid: 605941, exit value: 1
at java.base/java.lang.ProcessImpl.forkAndExec(Native Method)
at java.base/java.lang.ProcessImpl.(ProcessImpl.java:314)
at java.base/java.lang.ProcessImpl.start(ProcessImpl.java:244)
at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1110)
... 13 more

2025-02-06 09:14:46,697 WARN [kvm.storage.MultipathSCSIAdapterBase] (MultipathMapCleanupJob:[]) (logid:) Exception [Cannot run program "/usr/share/cloudstac
k-common/scripts/storage/multipath/cleanStaleMaps.sh": error=0, Failed to exec spawn helper: pid: 605943, exit value: 1] occurred when attempting to run command [/usr/share/cloudstack-common/scripts/storage/multipath/cleanStaleMaps.sh ]. java.io.IOException: Cannot run program "/usr/share/cloudstack-common/script
s/storage/multipath/cleanStaleMaps.sh": error=0, Failed to exec spawn helper: pid: 605943, exit value: 1
at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1143)
at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1073)
at com.cloud.utils.script.Script.execute(Script.java:232)
at com.cloud.hypervisor.kvm.storage.MultipathSCSIAdapterBase.runScript(MultipathSCSIAdapterBase.java:382)
at com.cloud.hypervisor.kvm.storage.MultipathSCSIAdapterBase.cleanupStaleMaps(MultipathSCSIAdapterBase.java:572)
at com.cloud.hypervisor.kvm.storage.MultipathSCSIAdapterBase$1.run(MultipathSCSIAdapterBase.java:122)
at java.base/java.util.TimerThread.mainLoop(Timer.java:566)
at java.base/java.util.TimerThread.run(Timer.java:516)
Caused by: java.io.IOException: error=0, Failed to exec spawn helper: pid: 605943, exit value: 1
at java.base/java.lang.ProcessImpl.forkAndExec(Native Method)
at java.base/java.lang.ProcessImpl.(ProcessImpl.java:314)
at java.base/java.lang.ProcessImpl.start(ProcessImpl.java:244)
at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1110)
... 7 more

versions

Cloudstack 4.20 + KVM
CEPH as Primary Storage

The steps to reproduce the bug

1.Deploy Instance, add firewall rules for guest networks are failing
2.
3.
...

What to do about it?

Restarting Cloudstack agent resolved the issue

[DaanHoogland]

@midhunpjos , it looks like the agent is not allowed to write on disk. If so only restarting the agent would not solve the issue on NFS. I do not know what the behaviour on Ceph is. Can you reliably reproduce this?

[midhunpjos]

@DaanHoogland It happened to all Hosts in a cluster and just restarting agent everything was back to normal in all hosts.

[midhunpjos]

googled the error , it looks like a config is needed

-Djdk.lang.Process.launchMechanism=vfork

maybe add to JAVA_OPTS in /etc/default/cloudstack-agent

Noted @weizhouapache. Will relay on this if the issue is happening again

[PPisz]

We have the same problem regularly, today it occurred for about 1 hour on all hosts and disappeared by itself. The environment is Ubuntu 22, CS 4.19.3 - we will test the solution given in the comment. Despite deep environment checks we were not able to pinpoint what is happening. During this problem you cannot ssh to VR via cloud0 (we have a simple linux bridge + vlan configuration). Not all VRs have the issue at the same time, restarting or migrating the VR to another host does not solve the issue.

[PPisz]

After upgrading to 4.20 (which we rolled back from) we also had the bug described above.

[midhunpjos]

@weizhouapache Seems like unattended-upgrades are causing this and after every upgrade we need the agent restart
ref: https://bugs.launchpad.net/ubuntu/+source/openjdk-17/+bug/2055280
From my server logs I could see the JRE being upgraded frequently causing this issue.
Disabling unattended-upgrades can be a workaround

[weizhouapache]

good finding @midhunpjos