[manila-csi-plugin] csi.sock: connect: no such file or directory

[emreberber]

/kind bug

What happened:

When deploying the openstack-manila-csi Helm package, the controller and node-plugin pods stuck in an Error state.

~ ❯ kubectl logs manila-csi-openstack-manila-csi-controllerplugin-0 -c cephfs-nodeplugin                                                                                   
I1106 17:12:02.818479       1 driver.go:127] Driver: cephfs.manila.csi.openstack.org
I1106 17:12:02.818546       1 driver.go:128] Driver version: [email protected]
I1106 17:12:02.818552       1 driver.go:129] CSI spec version: 1.8.0
I1106 17:12:02.818558       1 driver.go:132] Operating on CEPHFS shares
I1106 17:12:02.818564       1 driver.go:137] Topology awareness disabled
I1106 17:12:02.818571       1 driver.go:159] Providing controller service
I1106 17:12:02.818577       1 driver.go:221] Enabling controller service capability: CREATE_DELETE_VOLUME
I1106 17:12:02.818582       1 driver.go:221] Enabling controller service capability: CREATE_DELETE_SNAPSHOT
I1106 17:12:02.818585       1 driver.go:221] Enabling controller service capability: EXPAND_VOLUME
I1106 17:12:02.818590       1 driver.go:240] Enabling volume access mode: MULTI_NODE_MULTI_WRITER
I1106 17:12:02.818595       1 driver.go:240] Enabling volume access mode: MULTI_NODE_SINGLE_WRITER
I1106 17:12:02.818599       1 driver.go:240] Enabling volume access mode: MULTI_NODE_READER_ONLY
I1106 17:12:02.818603       1 driver.go:240] Enabling volume access mode: SINGLE_NODE_WRITER
I1106 17:12:02.818607       1 driver.go:240] Enabling volume access mode: SINGLE_NODE_READER_ONLY
I1106 17:12:02.818612       1 driver.go:184] Providing node service
I1106 17:12:02.818789       1 common.go:111] Probing CSI driver for readiness
I1106 17:12:02.819558       1 builder.go:59] [ID:1] FWD GRPC error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /var/lib/kubelet/plugins/cephfs.csi.ceph.com/csi.sock: connect: no such file or directory"
F1106 17:12:02.819644       1 main.go:111] Driver node service initialization failed: failed to initialize proxied CSI driver: probe failed: CSI driver probe failed: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /var/lib/kubelet/plugins/cephfs.csi.ceph.com/csi.sock: connect: no such file or directory"

How to reproduce it:

helm upgrade  --install manila-csi cpo/openstack-manila-csi

Anything else we need to know?:

When I try with version 1.30.0, there are no issues. It deploys successfully.

Environment:
Kubernetes ❯ 1.26.6
manila-csi-plugin ❯ v1.31.0

I can't find any Kubernetes compatibility matrix. If there is one available, could you please share it? Has anyone encountered and resolved this issue?

[kayrus]

@emreberber are you going to use manila CSI plugin with the Cephfs driver? Then you need to run the https://github.com/ceph/ceph-csi and point it to the --fwdendpoint=$(FWD_CSI_ENDPOINT).

basically manila CSI driver reads requests from the --endpoint=$(CSI_ENDPOINT), modifies these requests, and forwards them to NFS CSI plugin or Cephfs CSI plugin.

[emreberber]

Thanks for your comment

I found out later. It was necessary to deploy Ceph CSI along with Manila CSI. I could not see if this requirement is written in the document.

[gouthampacha]

@emreberber this is called out here: https://github.com/kubernetes/cloud-provider-openstack/blob/04bd8e1343d630a0db3e9093dd71f59e477ee7ad/docs/manila-csi-plugin/using-manila-csi-plugin.md#deployment

[emreberber]

Maybe we can rewrite this section to make it clearer. To be honest, I didn’t fully understand it when I first read it.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale