Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

test failed in CI: clickhouse-cluster #7554

Open
jgallagher opened this issue Feb 18, 2025 · 1 comment
Open

test failed in CI: clickhouse-cluster #7554

jgallagher opened this issue Feb 18, 2025 · 1 comment
Labels
Test Flake Tests that work. Wait, no. Actually yes. Hang on. Something is broken.

Comments

@jgallagher
Copy link
Contributor

This test failed on a CI run on #7552:

https://github.com/oxidecomputer/omicron/runs/37425204617

Log showing the specific test failure:

https://buildomat.eng.oxide.computer/wg/0/details/01JMDAAVAK414E40W2N1DH22TR/PjAIHOoTVWLU5DUfGYcAEU351gSQoeekCe5CE7JjhMA74Vc6/01JMDABK8JAFW0SDJAK0W1F0W3

Excerpt from the log showing the failure:

5759	2025-02-18T20:14:39.890Z	     Running `target/debug/clickhouse-cluster-dev`
5760	2025-02-18T20:14:39.902Z	Feb 18 20:14:39.900 INFO Setting up a ClickHouse cluster
5761	2025-02-18T20:14:39.905Z	Deploying keeper: /var/tmp/omicron_tmp/clickward_test/keeper-3
5762	2025-02-18T20:14:39.999Z	Deploying keeper: /var/tmp/omicron_tmp/clickward_test/keeper-2
5763	2025-02-18T20:14:40.003Z	Deploying keeper: /var/tmp/omicron_tmp/clickward_test/keeper-1
5764	2025-02-18T20:14:40.004Z	Deploying clickhouse server: /var/tmp/omicron_tmp/clickward_test/clickhouse-1
5765	2025-02-18T20:14:40.004Z	Deploying clickhouse server: /var/tmp/omicron_tmp/clickward_test/clickhouse-2
5766	2025-02-18T20:15:10.104Z	Error: failed to ping ClickHouse server
5767	2025-02-18T20:15:10.104Z	
5768	2025-02-18T20:15:10.104Z	Caused by:
5769	2025-02-18T20:15:10.196Z	    timed out after 30.101961239s
5770	2025-02-18T20:15:10.196Z	
5771	2025-02-18T20:15:10.196Z	Stack backtrace:
5772	2025-02-18T20:15:10.196Z	   0: <E as anyhow::context::ext::StdError>::ext_context
5773	2025-02-18T20:15:10.196Z	             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.95/src/backtrace.rs:27:14
5774	2025-02-18T20:15:10.196Z	   1: anyhow::context::<impl anyhow::Context<T,E> for core::result::Result<T,E>>::context
5775	2025-02-18T20:15:10.196Z	             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.95/src/context.rs:54:31
5776	2025-02-18T20:15:10.196Z	   2: {async_fn#0}
5777	2025-02-18T20:15:10.196Z	             at ./oximeter/test-utils/src/lib.rs:194:5
5778	2025-02-18T20:15:10.196Z	   3: {async_block#0}
5779	2025-02-18T20:15:10.196Z	             at ./dev-tools/clickhouse-cluster-dev/src/main.rs:60:42
5780	2025-02-18T20:15:10.196Z	   4: {closure#0}<clickhouse_cluster_dev::main::{async_block_env#0}>
5781	2025-02-18T20:15:10.197Z	             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.43.0/src/runtime/park.rs:284:63
5782	2025-02-18T20:15:10.197Z	   5: with_budget<core::task::poll::Poll<core::result::Result<(), anyhow::Error>>, tokio::runtime::park::{impl#4}::block_on::{closure_env#0}<clickhouse_cluster_dev::main::{async_block_env#0}>>
5783	2025-02-18T20:15:10.197Z	             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.43.0/src/runtime/coop.rs:107:5
5784	2025-02-18T20:15:10.197Z	   6: budget<core::task::poll::Poll<core::result::Result<(), anyhow::Error>>, tokio::runtime::park::{impl#4}::block_on::{closure_env#0}<clickhouse_cluster_dev::main::{async_block_env#0}>>
5785	2025-02-18T20:15:10.197Z	             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.43.0/src/runtime/coop.rs:73:5
5786	2025-02-18T20:15:10.197Z	   7: block_on<clickhouse_cluster_dev::main::{async_block_env#0}>
5787	2025-02-18T20:15:10.197Z	             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.43.0/src/runtime/park.rs:284:31
5788	2025-02-18T20:15:10.197Z	   8: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
5789	2025-02-18T20:15:10.197Z	             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.43.0/src/runtime/context/blocking.rs:66:9
5790	2025-02-18T20:15:10.197Z	   9: tokio::runtime::scheduler::multi_thread::MultiThread::block_on::{{closure}}
5791	2025-02-18T20:15:10.198Z	             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.43.0/src/runtime/scheduler/multi_thread/mod.rs:87:13
5792	2025-02-18T20:15:10.198Z	  10: tokio::runtime::context::runtime::enter_runtime
5793	2025-02-18T20:15:10.198Z	             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.43.0/src/runtime/context/runtime.rs:65:16
5794	2025-02-18T20:15:10.198Z	  11: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
5795	2025-02-18T20:15:10.198Z	             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.43.0/src/runtime/scheduler/multi_thread/mod.rs:86:9
5796	2025-02-18T20:15:10.198Z	  12: tokio::runtime::runtime::Runtime::block_on_inner
5797	2025-02-18T20:15:10.198Z	             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.43.0/src/runtime/runtime.rs:370:45
5798	2025-02-18T20:15:10.198Z	  13: block_on<clickhouse_cluster_dev::main::{async_block_env#0}>
5799	2025-02-18T20:15:10.198Z	             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.43.0/src/runtime/runtime.rs:342:13
5800	2025-02-18T20:15:10.198Z	  14: clickhouse_cluster_dev::main
5801	2025-02-18T20:15:10.198Z	             at ./dev-tools/clickhouse-cluster-dev/src/main.rs:69:5
5802	2025-02-18T20:15:10.198Z	  15: core::ops::function::FnOnce::call_once
5803	2025-02-18T20:15:10.198Z	             at /rustc/e71f9a9a98b0faf423844bf0ba7438f29dc27d58/library/core/src/ops/function.rs:250:5
5804	2025-02-18T20:15:10.199Z	  16: __rust_begin_short_backtrace<fn() -> core::result::Result<(), anyhow::Error>, core::result::Result<(), anyhow::Error>>
5805	2025-02-18T20:15:10.200Z	             at /rustc/e71f9a9a98b0faf423844bf0ba7438f29dc27d58/library/std/src/sys/backtrace.rs:154:18
5806	2025-02-18T20:15:10.200Z	  17: {closure#0}<core::result::Result<(), anyhow::Error>>
5807	2025-02-18T20:15:10.200Z	             at /rustc/e71f9a9a98b0faf423844bf0ba7438f29dc27d58/library/std/src/rt.rs:195:18
5808	2025-02-18T20:15:10.200Z	  18: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
5809	2025-02-18T20:15:10.200Z	             at /rustc/e71f9a9a98b0faf423844bf0ba7438f29dc27d58/library/core/src/ops/function.rs:284:13
5810	2025-02-18T20:15:10.200Z	  19: std::panicking::try::do_call
5811	2025-02-18T20:15:10.200Z	             at /rustc/e71f9a9a98b0faf423844bf0ba7438f29dc27d58/library/std/src/panicking.rs:557:40
5812	2025-02-18T20:15:10.200Z	  20: std::panicking::try
5813	2025-02-18T20:15:10.200Z	             at /rustc/e71f9a9a98b0faf423844bf0ba7438f29dc27d58/library/std/src/panicking.rs:520:19
5814	2025-02-18T20:15:10.200Z	  21: std::panic::catch_unwind
5815	2025-02-18T20:15:10.200Z	             at /rustc/e71f9a9a98b0faf423844bf0ba7438f29dc27d58/library/std/src/panic.rs:358:14
5816	2025-02-18T20:15:10.200Z	  22: std::rt::lang_start_internal::{{closure}}
5817	2025-02-18T20:15:10.200Z	             at /rustc/e71f9a9a98b0faf423844bf0ba7438f29dc27d58/library/std/src/rt.rs:174:48
5818	2025-02-18T20:15:10.200Z	  23: std::panicking::try::do_call
5819	2025-02-18T20:15:10.200Z	             at /rustc/e71f9a9a98b0faf423844bf0ba7438f29dc27d58/library/std/src/panicking.rs:557:40
5820	2025-02-18T20:15:10.200Z	  24: std::panicking::try
5821	2025-02-18T20:15:10.200Z	             at /rustc/e71f9a9a98b0faf423844bf0ba7438f29dc27d58/library/std/src/panicking.rs:520:19
5822	2025-02-18T20:15:10.200Z	  25: std::panic::catch_unwind
5823	2025-02-18T20:15:10.200Z	             at /rustc/e71f9a9a98b0faf423844bf0ba7438f29dc27d58/library/std/src/panic.rs:358:14
5824	2025-02-18T20:15:10.200Z	  26: std::rt::lang_start_internal
5825	2025-02-18T20:15:10.200Z	             at /rustc/e71f9a9a98b0faf423844bf0ba7438f29dc27d58/library/std/src/rt.rs:174:20
5826	2025-02-18T20:15:10.200Z	  27: std::rt::lang_start
5827	2025-02-18T20:15:10.200Z	             at /rustc/e71f9a9a98b0faf423844bf0ba7438f29dc27d58/library/std/src/rt.rs:194:17
5828	2025-02-18T20:15:10.201Z	  28: main
5829	2025-02-18T20:15:10.201Z	  29: <unknown>
5830	2025-02-18T20:15:10.201Z	  30: __libc_start_main
5831	2025-02-18T20:15:10.201Z	  31: _start
5832	2025-02-18T20:15:10.205Z	  SETUP FAIL [  38.303s] clickhouse-cluster: cargo run -p clickhouse-cluster-dev
5833	2025-02-18T20:15:10.205Z	
5834	2025-02-18T20:15:10.205Z	  Cancelling due to setup script failure
5835	2025-02-18T20:15:10.208Z	────────────
5836	2025-02-18T20:15:10.208Z	     Summary [  64.680s] 0/1802 tests run: 0 passed, 4 skipped
5837	2025-02-18T20:15:10.216Z	error: setup script failed
5838	2025-02-18T20:15:10.219Z	Command exited with non-zero status 105
@jgallagher jgallagher added the Test Flake Tests that work. Wait, no. Actually yes. Hang on. Something is broken. label Feb 18, 2025
@karencfv
Copy link
Contributor

Gargh, how annoying. There's not much information there to go on by. I'll add a bit more info to the logs so we can debug better if this happens again.

@karencfv karencfv mentioned this issue Feb 18, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Test Flake Tests that work. Wait, no. Actually yes. Hang on. Something is broken.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jgallagher @karencfv