Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] valkey search in VM ubuntu #34

Open
pesmith8a opened this issue Feb 4, 2025 · 12 comments
Open

[BUG] valkey search in VM ubuntu #34

pesmith8a opened this issue Feb 4, 2025 · 12 comments

Comments

@pesmith8a
Copy link

Running in Valkey 8 in Ubuntu 22 VM with 4 cores and 33 gb of ram.
Followed instructions on Github to compile modules using automated VS code method. libvalkeysearch.so compiled, but when started from valkey.conf the bug report below is generated.

I am a novice for compiling code as instructed from VS Code. so maybe I made a mistake somewhere.

Paul Smith

=== VALKEY BUG REPORT START: Cut & paste starting from here ===
923937:M 04 Feb 2025 15:21:37.814 # valkey 255.255.255 crashed by signal: 4, si_code: 2
923937:M 04 Feb 2025 15:21:37.814 # Crashed running the instruction at: 0x7781db62ecac

------ STACK TRACE ------
EIP:
/home/psmith/libvalkeysearch.so(+0xc2ecac)[0x7781db62ecac]

923937 valkey-server *
/lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7781de042520]
/usr/bin/valkey-server 127.0.0.1:6379(invalidFunctionWasCalled+0x0)[0x5855b9540650]
[0x7ffd7adef210]

1/1 expected stacktraces.

------ STACK TRACE DONE ------

------ REGISTERS ------
923937:M 04 Feb 2025 15:21:37.815 #
RAX:00007781dcd2a8f0 RBX:00007781dcc98630
RCX:0000000000000001 RDX:0000000000000001
RDI:00007781dcd2a8f0 RSI:00007781dcd2a8f0
RBP:00007ffd7adef140 RSP:00007ffd7adef140
R8 :00007781de45b100 R9 :00007781de17f7f0
R10:0000000000000022 R11:00007781de1e3110
R12:00007ffd7adf0138 R13:00005855e318f7d0
R14:00007781dcc98d88 R15:0000000000000000
RIP:00007781db62ecac EFL:0000000000010202
CSGSFS:002b000000000033
923937:M 04 Feb 2025 15:21:37.815 * hide-user-data-from-log is on, skip logging stack content to avoid spilling user data.

------ INFO OUTPUT ------

Server

redis_version:7.2.4
server_name:valkey
valkey_version:255.255.255
redis_git_sha1:2eac2ccd
redis_git_dirty:0
redis_build_id:67f51984ae421017
server_mode:standalone
os:Linux 6.8.0-52-generic x86_64
arch_bits:64
monotonic_clock:POSIX clock_gettime
multiplexing_api:epoll
gcc_version:11.4.0
process_id:923937
process_supervised:no
run_id:aef0bc8f01aed4e2e4530c0be7de9072cd203869
tcp_port:6379
server_time_usec:1738700497810938
uptime_in_seconds:0
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:10647249
executable:/usr/bin/valkey-server
config_file:/etc/valkey.conf
io_threads_active:0
availability_zone:

Clients

connected_clients:0
cluster_connections:0
maxclients:10000
client_recent_max_input_buffer:0
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
pubsub_clients:0
watching_clients:0
clients_in_timeout_table:0
total_watched_keys:0
total_blocking_keys:0
total_blocking_keys_on_nokey:0
paused_actions:none
paused_timeout_milliseconds:0

Memory

used_memory:898496
used_memory_human:877.44K
used_memory_rss:0
used_memory_rss_human:0B
used_memory_peak:898496
used_memory_peak_human:877.44K
used_memory_peak_perc:inf%
used_memory_overhead:216
used_memory_startup:0
used_memory_dataset:898280
used_memory_dataset_perc:99.98%
allocator_allocated:0
allocator_active:0
allocator_resident:0
allocator_muzzy:0
total_system_memory:35720462336
total_system_memory_human:33.27G
used_memory_lua:31744
used_memory_vm_eval:31744
used_memory_lua_human:31.00K
used_memory_scripts_eval:0
number_of_cached_scripts:0
number_of_functions:0
number_of_libraries:0
used_memory_vm_functions:33792
used_memory_vm_total:65536
used_memory_vm_total_human:64.00K
used_memory_functions:216
used_memory_scripts:216
used_memory_scripts_human:216B
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:-nan
allocator_frag_bytes:0
allocator_rss_ratio:-nan
allocator_rss_bytes:0
rss_overhead_ratio:-nan
rss_overhead_bytes:0
mem_fragmentation_ratio:-nan
mem_fragmentation_bytes:0
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_total_replication_buffers:0
mem_clients_slaves:0
mem_clients_normal:0
mem_cluster_links:0
mem_aof_buffer:0
mem_allocator:jemalloc-5.3.0
mem_overhead_db_hashtable_rehashing:0
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0

Persistence

loading:0
async_loading:0
current_cow_peak:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:0
rdb_bgsave_in_progress:0
rdb_last_save_time:1738700497
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_saves:0
rdb_last_cow_size:0
rdb_last_load_keys_expired:0
rdb_last_load_keys_loaded:0
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_rewrites:0
aof_rewrites_consecutive_failures:0
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0

Stats

total_connections_received:0
total_commands_processed:0
instantaneous_ops_per_sec:0
total_net_input_bytes:0
total_net_output_bytes:0
total_net_repl_input_bytes:0
total_net_repl_output_bytes:0
instantaneous_input_kbps:0.00
instantaneous_output_kbps:0.00
instantaneous_input_repl_kbps:0.00
instantaneous_output_repl_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:0
evicted_keys:0
evicted_clients:0
evicted_scripts:0
total_eviction_exceeded_time:0
current_eviction_exceeded_time:0
keyspace_hits:0
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
pubsubshard_channels:0
latest_fork_usec:0
total_forks:0
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
total_active_defrag_time:0
current_active_defrag_time:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:0
dump_payload_sanitizations:0
total_reads_processed:0
total_writes_processed:0
io_threaded_reads_processed:0
io_threaded_writes_processed:0
io_threaded_freed_objects:0
io_threaded_accept_processed:0
io_threaded_poll_processed:0
io_threaded_total_prefetch_batches:0
io_threaded_total_prefetch_entries:0
client_query_buffer_limit_disconnections:0
client_output_buffer_limit_disconnections:0
reply_buffer_shrinks:0
reply_buffer_expands:0
eventloop_cycles:0
eventloop_duration_sum:0
eventloop_duration_cmd_sum:0
instantaneous_eventloop_cycles_per_sec:0
instantaneous_eventloop_duration_usec:0
acl_access_denied_auth:0
acl_access_denied_cmd:0
acl_access_denied_key:0
acl_access_denied_channel:0

Replication

role:master
connected_slaves:0
replicas_waiting_psync:0
master_failover_state:no-failover
master_replid:72cc83a0485833346019257e6204d9620746ff0e
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:10485760
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

CPU

used_cpu_sys:0.001088
used_cpu_user:0.006531
used_cpu_sys_children:0.000000
used_cpu_user_children:0.000000
used_cpu_sys_main_thread:0.001088
used_cpu_user_main_thread:0.006531

Modules

Commandstats

Errorstats

Latencystats

Cluster

cluster_enabled:0

Keyspace

------ CLIENT LIST OUTPUT ------

------ MODULES INFO OUTPUT ------

------ CONFIG DEBUG OUTPUT ------
list-compress-depth 0
debug-context ""
slave-read-only yes
lazyfree-lazy-user-flush yes
lazyfree-lazy-user-del yes
lazyfree-lazy-expire yes
repl-diskless-load disabled
proto-max-bulk-len 512mb
sanitize-dump-payload no
lazyfree-lazy-eviction yes
replica-read-only yes
dual-channel-replication-enabled no
repl-diskless-sync yes
client-query-buffer-limit 1gb
activedefrag no
import-mode no
io-threads 1
lazyfree-lazy-server-del yes

------ FAST MEMORY TEST ------
*** Preparing to test memory region 5855b97c8000 (2355200 bytes)
*** Preparing to test memory region 5855e318d000 (135168 bytes)
*** Preparing to test memory region 7781dcd04000 (180224 bytes)
*** Preparing to test memory region 7781dd400000 (8388608 bytes)
*** Preparing to test memory region 7781ddcbd000 (1114112 bytes)
*** Preparing to test memory region 7781ddf30000 (4096 bytes)
*** Preparing to test memory region 7781de21c000 (53248 bytes)
*** Preparing to test memory region 7781de234000 (86016 bytes)
*** Preparing to test memory region 7781de254000 (8192 bytes)
*** Preparing to test memory region 7781de367000 (4096 bytes)
*** Preparing to test memory region 7781de45b000 (73728 bytes)
.O.O.O.O.O.O.O.O.O.O.O
Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.

------ DUMPING CODE AROUND EIP ------
Symbol: (null) (base: (nil))
Module: /home/psmith/libvalkeysearch.so (base 0x7781daa00000)
$ xxd -r -p /tmp/dump.hex /tmp/dump.bin
$ objdump --adjust-vma=(nil) -D -b binary -m i386:x86-64 /tmp/dump.bin

=== VALKEY BUG REPORT END. Make sure to include from START to END. ===
@madolson madolson transferred this issue from valkey-io/valkey Feb 4, 2025
@madolson
Copy link
Member

madolson commented Feb 4, 2025

@pesmith8a I assume this is because you need to run on Valkey 8.0, but moved the issue to Valkey-search to double check.

@yairgott
Copy link
Collaborator

yairgott commented Feb 4, 2025 •
edited
Loading

Thanks for sharing! Let me help troubleshoot this with you.

Valkeysearch is also compatible with Valkey 7.2. If I understand correctly, the trace suggests a potential glibc compatibility issue.

Could you share the following details:

  1. Did you compile the module on the same host where you're running it?
  2. gdb backtrace: you may find the following useful:
    gdb --ex=r --args ./valkey-server "--loadmodule $PATH/libvalkeysearch.so "
  3. glibc version- Run: ldd --version
  4. Minimum Required glibc for symbols- Run: readelf -s . libvalkeysearch.so | grep GLIBC_ | awk -F'@' '{print $2}' | sort -u
  5. Just to confirm, you haven’t modified MODULE.bazel or used special compilation directives, right?

@pesmith8a
Copy link
Author

pesmith8a commented Feb 5, 2025 via email

@yairgott
Copy link
Collaborator

yairgott commented Feb 5, 2025

Thanks for this information.

On bullet #3, It looks like you accidentally pasted the entire line rather than just ‘ ldd --version’

@yairgott
Copy link
Collaborator

yairgott commented Feb 5, 2025

Tomorrow I’ll try to reproduce based on the info you provided. To have a more complete picture, can you also provide your Valkey conf file.
Also, I wonder if you compiled Valkey with its out of the box build settings or you’ve applied some changes to it.

@pesmith8a
Copy link
Author

pesmith8a commented Feb 5, 2025 via email

@pesmith8a
Copy link
Author

Here is the valkey.conf that was used.

valkeyconf.txt

@yairgott
Copy link
Collaborator

yairgott commented Feb 6, 2025

hmm, I was able to run it successfully on a new Ubuntu 22 VM using the valkey configuration you provided.

uname -a: Linux instance-20250206-014702 6.8.0-1021-gcp #23~22.04.1-Ubuntu SMP Thu Jan 16 02:17:57 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
ldd --version:
ldd (Ubuntu GLIBC 2.35-0ubuntu3.8) 2.35
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.

Can you please share your glibc version, by running ldd --version ?

Maybe reclone both repos to fetch the latest and try to rebuild.

@pesmith8a
Copy link
Author

my version for ldd cmd was same as yours!
Am making new VM and starting process from start, with more care

@pesmith8a
Copy link
Author

Have been able to build the So file on Ubuntu 24 Virtual box VM, with cores and 64 gig of ram.
Using Valkey that is installed from apt install valkey in Ubuntu 24.
When put libvalkeysearch.so in valkey.conf the start of server gives errors as shown in attached valkey-server.log

Questions:

  1. What valkey minimum hardware and software requirements are their to run libvalkeysearch.so?
  2. I have been running in same VM that built the So, and it gives errors about Ram issues or hardware issues. I have ran
    sudo /usr/bin/valkey-server --test-memory 60000 and this has not shown any errors.
  3. Looks like are needing a cluster and can not run in one VM.
  4. Would like to find out what is the configuration that has been used in Testing so that can use close to tested configuration. Note: I do not have access to Cloud resources, and am running this on my Windows machine in Virtual box.

valkey-server.log

@yairgott
Copy link
Collaborator

yairgott commented Feb 11, 2025 •
edited
Loading

  1. The module consumes negligible RAM at startup. It was tested on x86-64 CPU architecture where the OS host is either Debian or Ubuntu.
  2. No, both Standalone and cluster are supported and unless you use a special startup flag ,--coordinator, standalone mode is used.

It seems to me that the issue is probably related to the fact that your VM runs on a Windows host which emulates different Linux functionalities. We haven't tested it with this setup yet. Could it be that your VM runs as 32-bit CPU architecture?

@pesmith8a
Copy link
Author

Thanks for the good information it helped very much. I will switch to a bare metal installation of Ubuntu 64. Will use Ubuntu 22 first ( due to it having jemalloc and Ubuntu 24 not having jemalloc.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@pesmith8a @yairgott @madolson