Update vector to >=0.3.0 to support remote docker hosts

[jweberde]

Bug report

• I confirm this is a bug with Supabase, not with my own application.
• I confirm I have searched the Docs, GitHub Discussions, and Discord.

Describe the bug

Running the Supabase CLI Version 1.183.5 clean installation using docker with a remote docker endpoint e.g. tcp://192.168.10.10:2375 will lead to startup error of the vector container. While starting the project using npx supabase start the vector container will timeout with health check failure and therefore the whole start is canceled:

supabase_vector_project_id container is not ready: unhealthy

Detailed Docker Logs

docker logs 191a972a7de0
2024-07-17T07:54:53.803819Z  INFO vector::app: Internal log rate limit configured. internal_log_rate_secs=10
2024-07-17T07:54:53.803939Z  INFO vector::app: Log level is enabled. level="vector=info,codec=info,vrl=info,file_source=info,tower_limit=trace,rdkafka=info,buffers=info,lapin=info,kube=info"
2024-07-17T07:54:53.803995Z  INFO vector::app: Loading configs. paths=["/etc/vector/vector.yaml"]
2024-07-17T07:54:53.829575Z  WARN vector::config::loading: Transform "router._unmatched" has no consumers
2024-07-17T07:54:54.075426Z ERROR vector::topology: Configuration error. error=Source "docker_host": Unknown scheme: tcp
2024-07-17T07:54:54.634091Z  INFO vector::app: Internal log rate limit configured. internal_log_rate_secs=10
2024-07-17T07:54:54.634222Z  INFO vector::app: Log level is enabled. level="vector=info,codec=info,vrl=info,file_source=info,tower_limit=trace,rdkafka=info,buffers=info,lapin=info,kube=info"
2024-07-17T07:54:54.634285Z  INFO vector::app: Loading configs. paths=["/etc/vector/vector.yaml"]
2024-07-17T07:54:54.663345Z  WARN vector::config::loading: Transform "router._unmatched" has no consumers
2024-07-17T07:54:54.863284Z ERROR vector::topology: Configuration error. error=Source "docker_host": Unknown scheme: tcp

Seems to be an error with the Remote TCP Docker Host

Configuration error. error=Source "docker_host": Unknown scheme: tcp

To Reproduce

Configure your Mac Local Docker Daemon to be exposed via TCP. For example via socat

1. Setup Docker Daemon via TCP not unix socket.

docker run -d -v /var/run/docker.sock:/var/run/docker.sock -p 127.0.0.1:1234:1234 bobrik/socat TCP-LISTEN:1234,fork UNIX-CONNECT:/var/run/docker.sock

# expose as default docker host for the current running terminal session
export DOCKER_HOST=tcp://localhost:1234

# verify access to docker daemon
docker ps -a

2. Init Vanilla Supabase Project

npx supabase init

3. Start Project

npx supabase start

4. Monitor Terminal for Errors

supabase_vector_project_id container is not ready: unhealthy

Expected behavior

The supabase vector container should support Remote Docker Daemon

Screenshots

If applicable, add screenshots to help explain your problem.

System information

• OS: macOS
• Version of supabase-cli: 1.183.5
• Version of Node.js: 20.14.0
• Docker Server Version: 24.0.7

Workaround which will disable Vector for now

# Stop all existing containers deleting their volumes
npx supabase stop --no-backup

# Start without vector in case you are not using it
npx supabase start -x vector

[encima]

Thanks for reporting this @jweberde (and for the workaround).
We use an unmodified timberio/vector image in Supabase (currently timberio/vector:0.28.1-alpine) so the issue is with the image there.

Would you mind opening the issue with them as a feature request and linking to this issue? I will close this one as it is not something we can support without a custom build.

Thanks!

[jweberde]

@encima I discussed the topic with vector community and they told me that in Vector >= timberio/vector:0.30 (May) the issue has been fixed and tcp scheme is supported.

I was able to find the fix - vectordotdev/vector@e1c0c02
and verify the comment given by the community. https://vector.dev/releases/0.30.0/

So basically the problem is because of an outdated version of vector used by supabase.
Unfortunately I am not able to overwrite the vector image version myself as it is not mapped: https://github.com/supabase/cli/blob/develop/pkg/config/config.go#L361

[encima]

Thanks for checking this @jweberde !

OK, let's rename the issue to request the image to be updated in the latest CLI release.

As a workaround, you can use the Docker self hosting method and specify the image version in the docker-compose.yml

[sweatybridge]

I've merged a fix in cli 1.187.5 so it sends the http scheme to vector instead of tcp. Generally our local and self-service versions match hosted for reproducibility so upgrading vector to a new version may not be trivial.

But I will let @Ziinc decide on that.

[arikanf]

This problem still exists even though upgrading vector to 0.30.0 on macos sonoma with the latest docker desktop

✔ Network supabase_default Created 0.0s
✔ Container supabase-imgproxy Created 0.1s
✔ Container supabase-vector Created 0.1s
✔ Container supabase-db Created 0.0s
✔ Container supabase-analytics Created 0.0s
✔ Container supabase-kong Created 0.0s
✔ Container supabase-studio Created 0.0s
✔ Container realtime-dev.supabase-realtime Created 0.0s
✔ Container supabase-meta Created 0.0s
✔ Container supabase-rest Created 0.0s
✔ Container supabase-auth Created 0.0s
✔ Container supabase-edge-functions Created 0.0s
✔ Container supabase-storage Created 0.0s
Attaching to realtime-dev.supabase-realtime, supabase-analytics, supabase-auth, supabase-db, supabase-edge-functions, supabase-imgproxy, supabase-kong, supabase-meta, supabase-rest, supabase-storage, supabase-studio, supabase-vector
supabase-imgproxy | WARNING [2024-08-14T08:41:13Z] No keys defined, so signature checking is disabled
supabase-imgproxy | WARNING [2024-08-14T08:41:13Z] No salts defined, so signature checking is disabled
supabase-imgproxy | WARNING [2024-08-14T08:41:13Z] Exposing root via IMGPROXY_LOCAL_FILESYSTEM_ROOT is unsafe
supabase-vector | 2024-08-14T08:41:13.849447Z INFO vector::app: Log level is enabled. level="vector=info,codec=info,vrl=info,file_source=info,tower_limit=trace,rdkafka=info,buffers=info,lapin=info,kube=info"
supabase-vector | 2024-08-14T08:41:13.851080Z INFO vector::app: Loading configs. paths=["etc/vector/vector.yml"]
supabase-imgproxy | INFO [2024-08-14T08:41:13Z] Starting server at :5001
supabase-vector | 2024-08-14T08:41:13.865626Z WARN vector::config::loading: Transform "router._unmatched" has no consumers
supabase-vector | 2024-08-14T08:41:13.865993Z INFO source{component_kind="source" component_id=docker_host component_type=docker_logs component_name=docker_host}: vector::sources::docker_logs: Capturing logs from now on. now=2024-08-14T08:41:13.865927844+00:00
supabase-vector | 2024-08-14T08:41:13.866064Z INFO source{component_kind="source" component_id=docker_host component_type=docker_logs component_name=docker_host}: vector::sources::docker_logs: Listening to docker log events.
supabase-vector | 2024-08-14T08:41:13.951340Z INFO vector::topology::running: Running healthchecks.
supabase-vector | 2024-08-14T08:41:13.951437Z INFO vector::topology::builder: Healthcheck passed.
supabase-vector | 2024-08-14T08:41:13.951453Z INFO vector::topology::builder: Healthcheck passed.
supabase-vector | 2024-08-14T08:41:13.951460Z INFO vector::topology::builder: Healthcheck passed.
supabase-vector | 2024-08-14T08:41:13.951463Z INFO vector::topology::builder: Healthcheck passed.
supabase-vector | 2024-08-14T08:41:13.951465Z INFO vector::topology::builder: Healthcheck passed.
supabase-vector | 2024-08-14T08:41:13.951467Z INFO vector::topology::builder: Healthcheck passed.
supabase-vector | 2024-08-14T08:41:13.951490Z INFO vector::topology::builder: Healthcheck passed.
supabase-vector | 2024-08-14T08:41:13.951557Z INFO vector: Vector has started. debug="false" version="0.30.0" arch="aarch64" revision="38c3f0b 2023-05-22 17:38:48.655488673"
supabase-vector | 2024-08-14T08:41:13.951723Z ERROR source{component_kind="source" component_id=docker_host component_type=docker_logs component_name=docker_host}: vector::sources::docker_logs: Listing currently running containers failed. error=error trying to connect: Connection refused (os error 111)
supabase-vector | 2024-08-14T08:41:13.954824Z INFO vector::internal_events::api: API server running. address=0.0.0.0:9001 playground=http://0.0.0.0:9001/playground
supabase-vector | 2024-08-14T08:41:13.954866Z INFO vector_common::shutdown: All sources have finished.
supabase-vector | 2024-08-14T08:41:13.954868Z INFO vector_common::shutdown: All sources have finished.
supabase-vector | 2024-08-14T08:41:13.954880Z INFO vector::app: All sources have finished.
supabase-vector | 2024-08-14T08:41:13.954888Z INFO vector: Vector has stopped.
supabase-vector | 2024-08-14T08:41:13.956640Z INFO vector::topology::running: Shutting down... Waiting on running components. remaining_components="logflare_db, logflare_auth, logflare_storage, logflare_kong, logflare_realtime, logflare_rest, logflare_functions" time_remaining="59 seconds left"
supabase-vector exited with code 0
Gracefully stopping... (press Ctrl+C again to force)
dependency failed to start: container supabase-vector exited (0)

2024-08-14T08:41:13.951557Z INFO vector: Vector has started. debug="false" version="0.30.0" arch="aarch64" revision="38c3f0b 2023-05-22 17:38:48.655488673"
2024-08-14T08:41:13.951723Z ERROR source{component_kind="source" component_id=docker_host component_type=docker_logs component_name=docker_host}: vector::sources::docker_logs: Listing currently running containers failed. error=error trying to connect: Connection refused (os error 111)
2024-08-14T08:41:13.954824Z INFO vector::internal_events::api: API server running. address=0.0.0.0:9001 playground=http://0.0.0.0:9001/playground
2024-08-14T08:41:13.954866Z INFO vector_common::shutdown: All sources have finished.
2024-08-14T08:41:13.954868Z INFO vector_common::shutdown: All sources have finished.
2024-08-14T08:41:13.954880Z INFO vector::app: All sources have finished.
2024-08-14T08:41:13.954888Z INFO vector: Vector has stopped.
2024-08-14T08:41:13.956640Z INFO vector::topology::running: Shutting down... Waiting on running components. remaining_components="logflare_db, logflare_auth, logflare_storage, logflare_kong, logflare_realtime, logflare_rest, logflare_functions" time_remaining="59 seconds left"

[sweatybridge]

@arikanf are you using latest cli or self-hosting?

[arikanf]

self-hosting with docker

[sweatybridge]

Can you verify that /var/run/docker.sock is enabled under docker settings > advanced.

And also that docker socket is volume mounted into vector container? https://github.com/supabase/supabase/blob/master/docker/docker-compose.yml#423

[arikanf]

[ivanburlakov]

@sweatybridge Same issue when running supabase start on almost a barebones project using stable Supabase CLI v1.219.2. --debug shows vector version as 0.28.1. Maybe upgrading it will fix it, as was said here?

Rootless Docker, OS: Fedora Silverblue 41 with Supabase installed from Homebrew inside a Toolbx.

Successfully launched with supabase start -x vector

[sweatybridge]

@ivanburlakov we patched this on cli side to map tcp url scheme to http for vector 0.28.1 https://github.com/supabase/cli/blob/develop/internal/start/start.go#L302

Do you have docker socket exposed over tcp?