Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

prometheus mds metrics no longer available #485

Open
slapcat opened this issue Dec 22, 2024 · 3 comments
Open

prometheus mds metrics no longer available #485

slapcat opened this issue Dec 22, 2024 · 3 comments
Labels
bug Something isn't working

Comments

@slapcat
Copy link

slapcat commented Dec 22, 2024

Issue report

Somewhere between revision 1100 and 1139, some prometheus metrics went missing. The one I've noticed so far is the MDS metrics. I found this issue when one of my clusters stopped reporting those metrics. Seeing it was on a newer version, I upgraded the working clusters to the latest revision and they are now all exhibiting the same issue.

What version of MicroCeph are you using ?

microceph (reef/stable) 18.2.4+snapc9f2b08f92

What are the steps to reproduce this issue ?

  1. Install latest reef/stable release.
  2. Enable prometheus mgr module: ceph mgr module enable prometheus
  3. Curl endpoint to get metrics and check for mds metrics: curl localhost:9283/metrics| grep mds

What happens (observed behaviour) ?

Only the metadata of the MDS daemon is there:

# HELP ceph_mds_metadata MDS Metadata
# TYPE ceph_mds_metadata untyped
ceph_mds_metadata{ceph_daemon="mds.scooby",fs_id="1",hostname="scooby",public_addr="10.0.0.7:6809/1914232058",rank="0",ceph_version="ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable)"} 1.0

What were you expecting to happen ?

Before 1100 -> 1139 upgrade, all MDS metrics were available:

# HELP ceph_mds_metadata MDS Metadata
# TYPE ceph_mds_metadata untyped
ceph_mds_metadata{ceph_daemon="mds.scooby",fs_id="1",hostname="scooby",public_addr="10.0.0.7:6809/972755473",rank="0",ceph_version="ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)"} 1.0
# HELP ceph_mds_caps Capabilities
# TYPE ceph_mds_caps gauge
ceph_mds_caps{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_ceph_cap_op_flush_ack caps truncate notify
# TYPE ceph_mds_ceph_cap_op_flush_ack counter
ceph_mds_ceph_cap_op_flush_ack{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_ceph_cap_op_flushsnap_ack caps truncate notify
# TYPE ceph_mds_ceph_cap_op_flushsnap_ack counter
ceph_mds_ceph_cap_op_flushsnap_ack{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_ceph_cap_op_grant Grant caps
# TYPE ceph_mds_ceph_cap_op_grant counter
ceph_mds_ceph_cap_op_grant{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_ceph_cap_op_revoke Revoke caps
# TYPE ceph_mds_ceph_cap_op_revoke counter
ceph_mds_ceph_cap_op_revoke{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_ceph_cap_op_trunc caps truncate notify
# TYPE ceph_mds_ceph_cap_op_trunc counter
ceph_mds_ceph_cap_op_trunc{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_dir_commit Directory commit
# TYPE ceph_mds_dir_commit counter
ceph_mds_dir_commit{ceph_daemon="mds.scooby"} 3.0
# HELP ceph_mds_dir_fetch_complete Fetch complete dirfrag
# TYPE ceph_mds_dir_fetch_complete counter
ceph_mds_dir_fetch_complete{ceph_daemon="mds.scooby"} 43.0
# HELP ceph_mds_dir_fetch_keys Fetch keys from dirfrag
# TYPE ceph_mds_dir_fetch_keys counter
ceph_mds_dir_fetch_keys{ceph_daemon="mds.scooby"} 6.0
# HELP ceph_mds_dir_merge Directory merge
# TYPE ceph_mds_dir_merge counter
ceph_mds_dir_merge{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_dir_split Directory split
# TYPE ceph_mds_dir_split counter
ceph_mds_dir_split{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_exported_inodes Exported inodes
# TYPE ceph_mds_exported_inodes counter
ceph_mds_exported_inodes{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_forward Forwarding request
# TYPE ceph_mds_forward counter
ceph_mds_forward{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_handle_client_cap_release Client cap release msg
# TYPE ceph_mds_handle_client_cap_release counter
ceph_mds_handle_client_cap_release{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_handle_client_caps Client caps msg
# TYPE ceph_mds_handle_client_caps counter
ceph_mds_handle_client_caps{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_handle_client_caps_dirty Client dirty caps msg
# TYPE ceph_mds_handle_client_caps_dirty counter
ceph_mds_handle_client_caps_dirty{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_handle_inode_file_caps Inter mds caps msg
# TYPE ceph_mds_handle_inode_file_caps counter
ceph_mds_handle_inode_file_caps{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_imported_inodes Imported inodes
# TYPE ceph_mds_imported_inodes counter
ceph_mds_imported_inodes{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_inodes Inodes
# TYPE ceph_mds_inodes gauge
ceph_mds_inodes{ceph_daemon="mds.scooby"} 140992.0
# HELP ceph_mds_inodes_expired Inodes expired
# TYPE ceph_mds_inodes_expired gauge
ceph_mds_inodes_expired{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_inodes_pinned Inodes pinned
# TYPE ceph_mds_inodes_pinned gauge
ceph_mds_inodes_pinned{ceph_daemon="mds.scooby"} 105904.0
# HELP ceph_mds_inodes_with_caps Inodes with capabilities
# TYPE ceph_mds_inodes_with_caps gauge
ceph_mds_inodes_with_caps{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_load_cent Load per cent
# TYPE ceph_mds_load_cent gauge
ceph_mds_load_cent{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_openino_dir_fetch OpenIno incomplete directory fetchings
# TYPE ceph_mds_openino_dir_fetch counter
ceph_mds_openino_dir_fetch{ceph_daemon="mds.scooby"} 33.0
# HELP ceph_mds_process_request_cap_release Process request cap release
# TYPE ceph_mds_process_request_cap_release counter
ceph_mds_process_request_cap_release{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_reply_latency_sum Reply latency Total
# TYPE ceph_mds_reply_latency_sum counter
ceph_mds_reply_latency_sum{ceph_daemon="mds.scooby"} 0.70521454
# HELP ceph_mds_reply_latency_count Reply latency Count
# TYPE ceph_mds_reply_latency_count counter
ceph_mds_reply_latency_count{ceph_daemon="mds.scooby"} 73.0
# HELP ceph_mds_request Requests
# TYPE ceph_mds_request counter
ceph_mds_request{ceph_daemon="mds.scooby"} 75.0
# HELP ceph_mds_root_rbytes root inode rbytes
# TYPE ceph_mds_root_rbytes gauge
ceph_mds_root_rbytes{ceph_daemon="mds.scooby"} 597387365897.0
# HELP ceph_mds_root_rfiles root inode rfiles
# TYPE ceph_mds_root_rfiles gauge
ceph_mds_root_rfiles{ceph_daemon="mds.scooby"} 234472.0
# HELP ceph_mds_root_rsnaps root inode rsnaps
# TYPE ceph_mds_root_rsnaps gauge
ceph_mds_root_rsnaps{ceph_daemon="mds.scooby"} 55.0
# HELP ceph_mds_slow_reply Slow replies
# TYPE ceph_mds_slow_reply counter
ceph_mds_slow_reply{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_subtrees Subtrees
# TYPE ceph_mds_subtrees gauge
ceph_mds_subtrees{ceph_daemon="mds.scooby"} 3.0
# HELP ceph_mds_cache_ireq_enqueue_scrub Internal Request type enqueue scrub
# TYPE ceph_mds_cache_ireq_enqueue_scrub counter
ceph_mds_cache_ireq_enqueue_scrub{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_cache_ireq_exportdir Internal Request type export dir
# TYPE ceph_mds_cache_ireq_exportdir counter
ceph_mds_cache_ireq_exportdir{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_cache_ireq_flush Internal Request type flush
# TYPE ceph_mds_cache_ireq_flush counter
ceph_mds_cache_ireq_flush{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_cache_ireq_fragmentdir Internal Request type fragmentdir
# TYPE ceph_mds_cache_ireq_fragmentdir counter
ceph_mds_cache_ireq_fragmentdir{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_cache_ireq_fragstats Internal Request type frag stats
# TYPE ceph_mds_cache_ireq_fragstats counter
ceph_mds_cache_ireq_fragstats{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_cache_ireq_inodestats Internal Request type inode stats
# TYPE ceph_mds_cache_ireq_inodestats counter
ceph_mds_cache_ireq_inodestats{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_cache_num_recovering_enqueued Files waiting for recovery
# TYPE ceph_mds_cache_num_recovering_enqueued gauge
ceph_mds_cache_num_recovering_enqueued{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_cache_num_recovering_prioritized Files waiting for recovery with elevated priority
# TYPE ceph_mds_cache_num_recovering_prioritized gauge
ceph_mds_cache_num_recovering_prioritized{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_cache_num_recovering_processing Files currently being recovered
# TYPE ceph_mds_cache_num_recovering_processing gauge
ceph_mds_cache_num_recovering_processing{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_cache_num_strays Stray dentries
# TYPE ceph_mds_cache_num_strays gauge
ceph_mds_cache_num_strays{ceph_daemon="mds.scooby"} 6119.0
# HELP ceph_mds_cache_num_strays_delayed Stray dentries delayed
# TYPE ceph_mds_cache_num_strays_delayed gauge
ceph_mds_cache_num_strays_delayed{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_cache_num_strays_enqueuing Stray dentries enqueuing for purge
# TYPE ceph_mds_cache_num_strays_enqueuing gauge
ceph_mds_cache_num_strays_enqueuing{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_cache_recovery_completed File recoveries completed
# TYPE ceph_mds_cache_recovery_completed counter
ceph_mds_cache_recovery_completed{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_cache_recovery_started File recoveries started
# TYPE ceph_mds_cache_recovery_started counter
ceph_mds_cache_recovery_started{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_cache_strays_created Stray dentries created
# TYPE ceph_mds_cache_strays_created counter
ceph_mds_cache_strays_created{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_cache_strays_enqueued Stray dentries enqueued for purge
# TYPE ceph_mds_cache_strays_enqueued counter
ceph_mds_cache_strays_enqueued{ceph_daemon="mds.scooby"} 243.0
# HELP ceph_mds_cache_strays_migrated Stray dentries migrated
# TYPE ceph_mds_cache_strays_migrated counter
ceph_mds_cache_strays_migrated{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_cache_strays_reintegrated Stray dentries reintegrated
# TYPE ceph_mds_cache_strays_reintegrated counter
ceph_mds_cache_strays_reintegrated{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_log_ev Events
# TYPE ceph_mds_log_ev gauge
ceph_mds_log_ev{ceph_daemon="mds.scooby"} 116900.0
# HELP ceph_mds_log_evadd Events submitted
# TYPE ceph_mds_log_evadd counter
ceph_mds_log_evadd{ceph_daemon="mds.scooby"} 325.0
# HELP ceph_mds_log_evex Total expired events
# TYPE ceph_mds_log_evex counter
ceph_mds_log_evex{ceph_daemon="mds.scooby"} 953.0
# HELP ceph_mds_log_evexd Current expired events
# TYPE ceph_mds_log_evexd gauge
ceph_mds_log_evexd{ceph_daemon="mds.scooby"} 953.0
# HELP ceph_mds_log_evexg Expiring events
# TYPE ceph_mds_log_evexg gauge
ceph_mds_log_evexg{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_log_evtrm Trimmed events
# TYPE ceph_mds_log_evtrm counter
ceph_mds_log_evtrm{ceph_daemon="mds.scooby"} 953.0
# HELP ceph_mds_log_jlat_sum Journaler flush latency Total
# TYPE ceph_mds_log_jlat_sum counter
ceph_mds_log_jlat_sum{ceph_daemon="mds.scooby"} 58.250196456
# HELP ceph_mds_log_jlat_count Journaler flush latency Count
# TYPE ceph_mds_log_jlat_count counter
ceph_mds_log_jlat_count{ceph_daemon="mds.scooby"} 80.0
# HELP ceph_mds_log_replayed Events replayed
# TYPE ceph_mds_log_replayed counter
ceph_mds_log_replayed{ceph_daemon="mds.scooby"} 117528.0
# HELP ceph_mds_log_seg Segments
# TYPE ceph_mds_log_seg gauge
ceph_mds_log_seg{ceph_daemon="mds.scooby"} 129.0
# HELP ceph_mds_log_segadd Segments added
# TYPE ceph_mds_log_segadd counter
ceph_mds_log_segadd{ceph_daemon="mds.scooby"} 1.0
# HELP ceph_mds_log_segex Total expired segments
# TYPE ceph_mds_log_segex counter
ceph_mds_log_segex{ceph_daemon="mds.scooby"} 1.0
# HELP ceph_mds_log_segexd Current expired segments
# TYPE ceph_mds_log_segexd gauge
ceph_mds_log_segexd{ceph_daemon="mds.scooby"} 1.0
# HELP ceph_mds_log_segexg Expiring segments
# TYPE ceph_mds_log_segexg gauge
ceph_mds_log_segexg{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_log_segtrm Trimmed segments
# TYPE ceph_mds_log_segtrm counter
ceph_mds_log_segtrm{ceph_daemon="mds.scooby"} 1.0
# HELP ceph_mds_mem_cap Capabilities
# TYPE ceph_mds_mem_cap gauge
ceph_mds_mem_cap{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_mem_cap_plus Capabilities added
# TYPE ceph_mds_mem_cap_plus counter
ceph_mds_mem_cap_plus{ceph_daemon="mds.scooby"} 6.0
# HELP ceph_mds_mem_cap_minus Capabilities removed
# TYPE ceph_mds_mem_cap_minus counter
ceph_mds_mem_cap_minus{ceph_daemon="mds.scooby"} 6.0
# HELP ceph_mds_mem_dir Directories
# TYPE ceph_mds_mem_dir gauge
ceph_mds_mem_dir{ceph_daemon="mds.scooby"} 105299.0
# HELP ceph_mds_mem_dir_plus Directories opened
# TYPE ceph_mds_mem_dir_plus counter
ceph_mds_mem_dir_plus{ceph_daemon="mds.scooby"} 105299.0
# HELP ceph_mds_mem_dir_minus Directories closed
# TYPE ceph_mds_mem_dir_minus counter
ceph_mds_mem_dir_minus{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_mem_dn Dentries
# TYPE ceph_mds_mem_dn gauge
ceph_mds_mem_dn{ceph_daemon="mds.scooby"} 140992.0
# HELP ceph_mds_mem_dn_plus Dentries opened
# TYPE ceph_mds_mem_dn_plus counter
ceph_mds_mem_dn_plus{ceph_daemon="mds.scooby"} 140992.0
# HELP ceph_mds_mem_dn_minus Dentries closed
# TYPE ceph_mds_mem_dn_minus counter
ceph_mds_mem_dn_minus{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_mem_heap Heap size
# TYPE ceph_mds_mem_heap gauge
ceph_mds_mem_heap{ceph_daemon="mds.scooby"} 223424.0
# HELP ceph_mds_mem_ino Inodes
# TYPE ceph_mds_mem_ino gauge
ceph_mds_mem_ino{ceph_daemon="mds.scooby"} 140425.0
# HELP ceph_mds_mem_ino_plus Inodes opened
# TYPE ceph_mds_mem_ino_plus counter
ceph_mds_mem_ino_plus{ceph_daemon="mds.scooby"} 140750.0
# HELP ceph_mds_mem_ino_minus Inodes closed
# TYPE ceph_mds_mem_ino_minus counter
ceph_mds_mem_ino_minus{ceph_daemon="mds.scooby"} 325.0
# HELP ceph_mds_server_cap_acquisition_throttle Cap acquisition throttle counter
# TYPE ceph_mds_server_cap_acquisition_throttle counter
ceph_mds_server_cap_acquisition_throttle{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_server_cap_revoke_eviction Cap Revoke Client Eviction
# TYPE ceph_mds_server_cap_revoke_eviction counter
ceph_mds_server_cap_revoke_eviction{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_server_handle_client_request Client requests
# TYPE ceph_mds_server_handle_client_request counter
ceph_mds_server_handle_client_request{ceph_daemon="mds.scooby"} 75.0
# HELP ceph_mds_server_handle_client_session Client session messages
# TYPE ceph_mds_server_handle_client_session counter
ceph_mds_server_handle_client_session{ceph_daemon="mds.scooby"} 21.0
# HELP ceph_mds_server_handle_peer_request Peer requests
# TYPE ceph_mds_server_handle_peer_request counter
ceph_mds_server_handle_peer_request{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_server_req_create_latency_sum Request type create latency Total
# TYPE ceph_mds_server_req_create_latency_sum counter
ceph_mds_server_req_create_latency_sum{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_server_req_create_latency_count Request type create latency Count
# TYPE ceph_mds_server_req_create_latency_count counter
ceph_mds_server_req_create_latency_count{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_server_req_getattr_latency_sum Request type get attribute latency Total
# TYPE ceph_mds_server_req_getattr_latency_sum counter
ceph_mds_server_req_getattr_latency_sum{ceph_daemon="mds.scooby"} 0.069502791
# HELP ceph_mds_server_req_getattr_latency_count Request type get attribute latency Count
# TYPE ceph_mds_server_req_getattr_latency_count counter
ceph_mds_server_req_getattr_latency_count{ceph_daemon="mds.scooby"} 66.0
# HELP ceph_mds_server_req_getfilelock_latency_sum Request type get file lock latency Total
# TYPE ceph_mds_server_req_getfilelock_latency_sum counter
ceph_mds_server_req_getfilelock_latency_sum{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_server_req_getfilelock_latency_count Request type get file lock latency Count
# TYPE ceph_mds_server_req_getfilelock_latency_count counter
ceph_mds_server_req_getfilelock_latency_count{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_server_req_getvxattr_latency_sum Request type get virtual extended attribute latency Total
# TYPE ceph_mds_server_req_getvxattr_latency_sum counter
ceph_mds_server_req_getvxattr_latency_sum{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_server_req_getvxattr_latency_count Request type get virtual extended attribute latency Count
# TYPE ceph_mds_server_req_getvxattr_latency_count counter
ceph_mds_server_req_getvxattr_latency_count{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_server_req_link_latency_sum Request type link latency Total
# TYPE ceph_mds_server_req_link_latency_sum counter
ceph_mds_server_req_link_latency_sum{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_server_req_link_latency_count Request type link latency Count
# TYPE ceph_mds_server_req_link_latency_count counter
ceph_mds_server_req_link_latency_count{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_server_req_lookup_latency_sum Request type lookup latency Total
# TYPE ceph_mds_server_req_lookup_latency_sum counter
ceph_mds_server_req_lookup_latency_sum{ceph_daemon="mds.scooby"} 0.113742706
# HELP ceph_mds_server_req_lookup_latency_count Request type lookup latency Count
# TYPE ceph_mds_server_req_lookup_latency_count counter
ceph_mds_server_req_lookup_latency_count{ceph_daemon="mds.scooby"} 4.0
# HELP ceph_mds_server_req_lookuphash_latency_sum Request type lookup hash of inode latency Total
# TYPE ceph_mds_server_req_lookuphash_latency_sum counter
ceph_mds_server_req_lookuphash_latency_sum{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_server_req_lookuphash_latency_count Request type lookup hash of inode latency Count
# TYPE ceph_mds_server_req_lookuphash_latency_count counter
ceph_mds_server_req_lookuphash_latency_count{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_server_req_lookupino_latency_sum Request type lookup inode latency Total
# TYPE ceph_mds_server_req_lookupino_latency_sum counter
ceph_mds_server_req_lookupino_latency_sum{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_server_req_lookupino_latency_count Request type lookup inode latency Count
# TYPE ceph_mds_server_req_lookupino_latency_count counter
ceph_mds_server_req_lookupino_latency_count{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_server_req_lookupname_latency_sum Request type lookup name latency Total
# TYPE ceph_mds_server_req_lookupname_latency_sum counter
ceph_mds_server_req_lookupname_latency_sum{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_server_req_lookupname_latency_count Request type lookup name latency Count
# TYPE ceph_mds_server_req_lookupname_latency_count counter
ceph_mds_server_req_lookupname_latency_count{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_server_req_lookupparent_latency_sum Request type lookup parent latency Total
# TYPE ceph_mds_server_req_lookupparent_latency_sum counter
ceph_mds_server_req_lookupparent_latency_sum{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_server_req_lookupparent_latency_count Request type lookup parent latency Count
# TYPE ceph_mds_server_req_lookupparent_latency_count counter
ceph_mds_server_req_lookupparent_latency_count{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_server_req_lookupsnap_latency_sum Request type lookup snapshot latency Total
# TYPE ceph_mds_server_req_lookupsnap_latency_sum counter
ceph_mds_server_req_lookupsnap_latency_sum{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_server_req_lookupsnap_latency_count Request type lookup snapshot latency Count
# TYPE ceph_mds_server_req_lookupsnap_latency_count counter
ceph_mds_server_req_lookupsnap_latency_count{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_server_req_lssnap_latency_sum Request type list snapshot latency Total
# TYPE ceph_mds_server_req_lssnap_latency_sum counter
ceph_mds_server_req_lssnap_latency_sum{ceph_daemon="mds.scooby"} 0.000190177
# HELP ceph_mds_server_req_lssnap_latency_count Request type list snapshot latency Count
# TYPE ceph_mds_server_req_lssnap_latency_count counter
ceph_mds_server_req_lssnap_latency_count{ceph_daemon="mds.scooby"} 1.0
# HELP ceph_mds_server_req_mkdir_latency_sum Request type make directory latency Total
# TYPE ceph_mds_server_req_mkdir_latency_sum counter
ceph_mds_server_req_mkdir_latency_sum{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_server_req_mkdir_latency_count Request type make directory latency Count
# TYPE ceph_mds_server_req_mkdir_latency_count counter
ceph_mds_server_req_mkdir_latency_count{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_server_req_mknod_latency_sum Request type make node latency Total
# TYPE ceph_mds_server_req_mknod_latency_sum counter
ceph_mds_server_req_mknod_latency_sum{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_server_req_mknod_latency_count Request type make node latency Count
# TYPE ceph_mds_server_req_mknod_latency_count counter
ceph_mds_server_req_mknod_latency_count{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_server_req_mksnap_latency_sum Request type make snapshot latency Total
# TYPE ceph_mds_server_req_mksnap_latency_sum counter
ceph_mds_server_req_mksnap_latency_sum{ceph_daemon="mds.scooby"} 0.19603525
# HELP ceph_mds_server_req_mksnap_latency_count Request type make snapshot latency Count
# TYPE ceph_mds_server_req_mksnap_latency_count counter
ceph_mds_server_req_mksnap_latency_count{ceph_daemon="mds.scooby"} 1.0
# HELP ceph_mds_server_req_open_latency_sum Request type open latency Total
# TYPE ceph_mds_server_req_open_latency_sum counter
ceph_mds_server_req_open_latency_sum{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_server_req_open_latency_count Request type open latency Count
# TYPE ceph_mds_server_req_open_latency_count counter
ceph_mds_server_req_open_latency_count{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_server_req_readdir_latency_sum Request type read directory latency Total
# TYPE ceph_mds_server_req_readdir_latency_sum counter
ceph_mds_server_req_readdir_latency_sum{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_server_req_readdir_latency_count Request type read directory latency Count
# TYPE ceph_mds_server_req_readdir_latency_count counter
ceph_mds_server_req_readdir_latency_count{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_server_req_rename_latency_sum Request type rename latency Total
# TYPE ceph_mds_server_req_rename_latency_sum counter
ceph_mds_server_req_rename_latency_sum{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_server_req_rename_latency_count Request type rename latency Count
# TYPE ceph_mds_server_req_rename_latency_count counter
ceph_mds_server_req_rename_latency_count{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_server_req_renamesnap_latency_sum Request type rename snapshot latency Total
# TYPE ceph_mds_server_req_renamesnap_latency_sum counter
ceph_mds_server_req_renamesnap_latency_sum{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_server_req_renamesnap_latency_count Request type rename snapshot latency Count
# TYPE ceph_mds_server_req_renamesnap_latency_count counter
ceph_mds_server_req_renamesnap_latency_count{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_server_req_rmdir_latency_sum Request type remove directory latency Total
# TYPE ceph_mds_server_req_rmdir_latency_sum counter
ceph_mds_server_req_rmdir_latency_sum{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_server_req_rmdir_latency_count Request type remove directory latency Count
# TYPE ceph_mds_server_req_rmdir_latency_count counter
ceph_mds_server_req_rmdir_latency_count{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_server_req_rmsnap_latency_sum Request type remove snapshot latency Total
# TYPE ceph_mds_server_req_rmsnap_latency_sum counter
ceph_mds_server_req_rmsnap_latency_sum{ceph_daemon="mds.scooby"} 0.325743616
# HELP ceph_mds_server_req_rmsnap_latency_count Request type remove snapshot latency Count
# TYPE ceph_mds_server_req_rmsnap_latency_count counter
ceph_mds_server_req_rmsnap_latency_count{ceph_daemon="mds.scooby"} 1.0
# HELP ceph_mds_server_req_rmxattr_latency_sum Request type remove extended attribute latency Total
# TYPE ceph_mds_server_req_rmxattr_latency_sum counter
ceph_mds_server_req_rmxattr_latency_sum{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_server_req_rmxattr_latency_count Request type remove extended attribute latency Count
# TYPE ceph_mds_server_req_rmxattr_latency_count counter
ceph_mds_server_req_rmxattr_latency_count{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_server_req_setattr_latency_sum Request type set attribute latency Total
# TYPE ceph_mds_server_req_setattr_latency_sum counter
ceph_mds_server_req_setattr_latency_sum{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_server_req_setattr_latency_count Request type set attribute latency Count
# TYPE ceph_mds_server_req_setattr_latency_count counter
ceph_mds_server_req_setattr_latency_count{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_server_req_setdirlayout_latency_sum Request type set directory layout latency Total
# TYPE ceph_mds_server_req_setdirlayout_latency_sum counter
ceph_mds_server_req_setdirlayout_latency_sum{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_server_req_setdirlayout_latency_count Request type set directory layout latency Count
# TYPE ceph_mds_server_req_setdirlayout_latency_count counter
ceph_mds_server_req_setdirlayout_latency_count{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_server_req_setfilelock_latency_sum Request type set file lock latency Total
# TYPE ceph_mds_server_req_setfilelock_latency_sum counter
ceph_mds_server_req_setfilelock_latency_sum{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_server_req_setfilelock_latency_count Request type set file lock latency Count
# TYPE ceph_mds_server_req_setfilelock_latency_count counter
ceph_mds_server_req_setfilelock_latency_count{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_server_req_setlayout_latency_sum Request type set file layout latency Total
# TYPE ceph_mds_server_req_setlayout_latency_sum counter
ceph_mds_server_req_setlayout_latency_sum{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_server_req_setlayout_latency_count Request type set file layout latency Count
# TYPE ceph_mds_server_req_setlayout_latency_count counter
ceph_mds_server_req_setlayout_latency_count{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_server_req_setxattr_latency_sum Request type set extended attribute latency Total
# TYPE ceph_mds_server_req_setxattr_latency_sum counter
ceph_mds_server_req_setxattr_latency_sum{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_server_req_setxattr_latency_count Request type set extended attribute latency Count
# TYPE ceph_mds_server_req_setxattr_latency_count counter
ceph_mds_server_req_setxattr_latency_count{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_server_req_symlink_latency_sum Request type symbolic link latency Total
# TYPE ceph_mds_server_req_symlink_latency_sum counter
ceph_mds_server_req_symlink_latency_sum{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_server_req_symlink_latency_count Request type symbolic link latency Count
# TYPE ceph_mds_server_req_symlink_latency_count counter
ceph_mds_server_req_symlink_latency_count{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_server_req_unlink_latency_sum Request type unlink latency Total
# TYPE ceph_mds_server_req_unlink_latency_sum counter
ceph_mds_server_req_unlink_latency_sum{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_server_req_unlink_latency_count Request type unlink latency Count
# TYPE ceph_mds_server_req_unlink_latency_count counter
ceph_mds_server_req_unlink_latency_count{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_sessions_average_load Average Load
# TYPE ceph_mds_sessions_average_load gauge
ceph_mds_sessions_average_load{ceph_daemon="mds.scooby"} 58.0
# HELP ceph_mds_sessions_avg_session_uptime Average session uptime
# TYPE ceph_mds_sessions_avg_session_uptime gauge
ceph_mds_sessions_avg_session_uptime{ceph_daemon="mds.scooby"} 62.0
# HELP ceph_mds_sessions_session_add Sessions added
# TYPE ceph_mds_sessions_session_add counter
ceph_mds_sessions_session_add{ceph_daemon="mds.scooby"} 13.0
# HELP ceph_mds_sessions_session_count Session count
# TYPE ceph_mds_sessions_session_count gauge
ceph_mds_sessions_session_count{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_sessions_session_remove Sessions removed
# TYPE ceph_mds_sessions_session_remove counter
ceph_mds_sessions_session_remove{ceph_daemon="mds.scooby"} 13.0
# HELP ceph_mds_sessions_sessions_open Sessions currently open
# TYPE ceph_mds_sessions_sessions_open gauge
ceph_mds_sessions_sessions_open{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_sessions_sessions_stale Sessions currently stale
# TYPE ceph_mds_sessions_sessions_stale gauge
ceph_mds_sessions_sessions_stale{ceph_daemon="mds.scooby"} 0.0
# HELP ceph_mds_sessions_total_load Total Load
# TYPE ceph_mds_sessions_total_load gauge
ceph_mds_sessions_total_load{ceph_daemon="mds.scooby"} 58.0
ceph_objecter_op_active{ceph_daemon="mds.scooby"} 0.0
ceph_objecter_op_r{ceph_daemon="mds.scooby"} 196.0
ceph_objecter_op_rmw{ceph_daemon="mds.scooby"} 0.0
ceph_objecter_op_w{ceph_daemon="mds.scooby"} 1421.0
ceph_purge_queue_pq_executed{ceph_daemon="mds.scooby"} 243.0
ceph_purge_queue_pq_executing{ceph_daemon="mds.scooby"} 0.0
ceph_purge_queue_pq_executing_high_water{ceph_daemon="mds.scooby"} 16.0
ceph_purge_queue_pq_executing_ops{ceph_daemon="mds.scooby"} 0.0
ceph_purge_queue_pq_executing_ops_high_water{ceph_daemon="mds.scooby"} 16.0
ceph_purge_queue_pq_item_in_journal{ceph_daemon="mds.scooby"} 0.0
@sabaini
Copy link
Collaborator

sabaini commented Jan 10, 2025

This is likely due to reef having moved metrics to ceph-exporter which we will need to enable
CEPH-1069

@sabaini sabaini added the bug Something isn't working label Jan 10, 2025
@slapcat
Copy link
Author

slapcat commented Feb 27, 2025

Hi team - Is there any ETA on getting this fixed? Or at least a proposed workaround? I have absolutely no monitoring on cephfs now.

@syncronize-issues-to-jira Syncronize issues to Jira
Copy link

Thank you for reporting your feedback to us!

The internal ticket has been created: https://warthogs.atlassian.net/browse/CEPH-1183.

This message was autogenerated

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@sabaini @slapcat