pika-exporter：Add RocksDB column family metrics(#1566) #1696

chenbt-hz · 2023-07-08T11:20:29Z

新增以下的监控指标：
已zset为例，以下是info all的结果

zsets_cf-l0-file-count-limit-delays-with-ongoing-compaction: 0
zsets_cf-l0-file-count-limit-stops-with-ongoing-compaction: 0
zsets_compaction.L0.AvgSec: 0.043061
zsets_compaction.L0.CompCount: 1.000000
zsets_compaction.L0.CompMergeCPU: 0.001078
zsets_compaction.L0.CompSec: 0.043061
zsets_compaction.L0.CompactedFiles: 0.000000
zsets_compaction.L0.KeyDrop: 0.000000
zsets_compaction.L0.KeyIn: 0.000000
zsets_compaction.L0.MovedGB: 0.000000
zsets_compaction.L0.NumFiles: 1.000000
zsets_compaction.L0.RblobGB: 0.000000
zsets_compaction.L0.ReadGB: 0.000000
zsets_compaction.L0.ReadMBps: 0.000000
zsets_compaction.L0.RnGB: 0.000000
zsets_compaction.L0.Rnp1GB: 0.000000
zsets_compaction.L0.Score: 0.250000
zsets_compaction.L0.SizeBytes: 1182.000000
zsets_compaction.L0.WblobGB: 0.000000
zsets_compaction.L0.WnewGB: 0.000001
zsets_compaction.L0.WriteAmp: 1.000000
zsets_compaction.L0.WriteGB: 0.000001
zsets_compaction.L0.WriteMBps: 0.026177
zsets_compaction.Sum.AvgSec: 0.043061
zsets_compaction.Sum.CompCount: 1.000000
zsets_compaction.Sum.CompMergeCPU: 0.001078
zsets_compaction.Sum.CompSec: 0.043061
zsets_compaction.Sum.CompactedFiles: 0.000000
zsets_compaction.Sum.KeyDrop: 0.000000
zsets_compaction.Sum.KeyIn: 0.000000
zsets_compaction.Sum.MovedGB: 0.000000
zsets_compaction.Sum.NumFiles: 1.000000
zsets_compaction.Sum.RblobGB: 0.000000
zsets_compaction.Sum.ReadGB: 0.000000
zsets_compaction.Sum.ReadMBps: 0.000000
zsets_compaction.Sum.RnGB: 0.000000
zsets_compaction.Sum.Rnp1GB: 0.000000
zsets_compaction.Sum.Score: 0.000000
zsets_compaction.Sum.SizeBytes: 1182.000000
zsets_compaction.Sum.WblobGB: 0.000000
zsets_compaction.Sum.WnewGB: 0.000001
zsets_compaction.Sum.WriteAmp: 1.000000
zsets_compaction.Sum.WriteGB: 0.000001
zsets_compaction.Sum.WriteMBps: 0.026177
zsets_l0-file-count-limit-delays: 0
zsets_l0-file-count-limit-stops: 0
zsets_memtable-limit-delays: 0
zsets_memtable-limit-stops: 0
zsets_pending-compaction-bytes-delays: 0
zsets_pending-compaction-bytes-stops: 0
zsets_total-delays: 0
zsets_total-stops: 0

采集的数据大致如下：

....
# HELP pika_compaction 
#The all metrics of compaction_L<N>:
#compaction.L<N>.AvgSec	Average time spent per Compact.
#compaction.L<N>.CompCount	The number of times Compact has been accumulated.
#compaction.L<N>.CompMergeCPU	CPU time used in compression, in seconds.
#compaction.L<N>.CompSec	Compact cumulative time, in seconds.
#compaction.L<N>.CompactedFiles	Number of files that have completed compact.
#compaction.L<N>.KeyDrop	Number of keys deleted in compact.
#compaction.L<N>.KeyIn	Number of records compared during the compaction process.
#compaction.L<N>.MovedGB	During the compaction process, the number of bytes moved to level n+1.
#compaction.L<N>.NumFiles	Total number of sst files.
#compaction.L<N>.RblobGB	The size of data read from the blob file by the compaction, in GB.
#compaction.L<N>.ReadGB	Read size in GB.
#compaction.L<N>.ReadMBps	Read rate in MBps.
#compaction.L<N>.RnGB	When performing compact, read the size of the current layer file in GB.
#compaction.L<N>.Rnp1GB	When performing compact, read the size of the next level file in GB.
#compaction.L<N>.Score	Score, the higher the score, the higher the priority.
#compaction.L<N>.SizeBytes	Total size of SST in bytes.
#compaction.L<N>.WblobGB	The size of the blob file written during compaction, in GB.
#compaction.L<N>.WnewGB	WNP1- Rnp1.
#compaction.L<N>.WriteAmp	Total bytes written to level<N+1>/(total bytes read from level<N>).
#compaction.L<N>.WriteGB	The size of the table file written during the compaction period, in GB.
#compaction.L<N>.WriteMBps	Data write rate in compaction.
# TYPE pika_compaction gauge
pika_compaction{addr="127.0.0.1:9221",alias="",compaction_level="0",data_type="strings",info_type="AvgSec"} 0.048421
pika_compaction{addr="127.0.0.1:9221",alias="",compaction_level="0",data_type="strings",info_type="CompCount"} 2
pika_compaction{addr="127.0.0.1:9221",alias="",compaction_level="0",data_type="strings",info_type="CompMergeCPU"} 0.004067
pika_compaction{addr="127.0.0.1:9221",alias="",compaction_level="0",data_type="strings",info_type="CompSec"} 0.096843
pika_compaction{addr="127.0.0.1:9221",alias="",compaction_level="0",data_type="strings",info_type="CompactedFiles"} 0
pika_compaction{addr="127.0.0.1:9221",alias="",compaction_level="0",data_type="strings",info_type="KeyDrop"} 0
pika_compaction{addr="127.0.0.1:9221",alias="",compaction_level="0",data_type="strings",info_type="KeyIn"} 0
pika_compaction{addr="127.0.0.1:9221",alias="",compaction_level="0",data_type="strings",info_type="MovedGB"} 0
pika_compaction{addr="127.0.0.1:9221",alias="",compaction_level="0",data_type="strings",info_type="NumFiles"} 2
pika_compaction{addr="127.0.0.1:9221",alias="",compaction_level="0",data_type="strings",info_type="RblobGB"} 0
pika_compaction{addr="127.0.0.1:9221",alias="",compaction_level="0",data_type="strings",info_type="ReadGB"} 0
pika_compaction{addr="127.0.0.1:9221",alias="",compaction_level="0",data_type="strings",info_type="ReadMBps"} 0
pika_compaction{addr="127.0.0.1:9221",alias="",compaction_level="0",data_type="strings",info_type="RnGB"} 0
pika_compaction{addr="127.0.0.1:9221",alias="",compaction_level="0",data_type="strings",info_type="Rnp1GB"} 0
pika_compaction{addr="127.0.0.1:9221",alias="",compaction_level="0",data_type="strings",info_type="Score"} 0.5
pika_compaction{addr="127.0.0.1:9221",alias="",compaction_level="0",data_type="strings",info_type="SizeBytes"} 2358
pika_compaction{addr="127.0.0.1:9221",alias="",compaction_level="0",data_type="strings",info_type="WblobGB"} 0
pika_compaction{addr="127.0.0.1:9221",alias="",compaction_level="0",data_type="strings",info_type="WnewGB"} 2e-06
pika_compaction{addr="127.0.0.1:9221",alias="",compaction_level="0",data_type="strings",info_type="WriteAmp"} 1
pika_compaction{addr="127.0.0.1:9221",alias="",compaction_level="0",data_type="strings",info_type="WriteGB"} 2e-06
pika_compaction{addr="127.0.0.1:9221",alias="",compaction_level="0",data_type="strings",info_type="WriteMBps"} 0.02322
# HELP pika_compaction_Sum The all metrics of compaction_Sum.
# TYPE pika_compaction_Sum gauge
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="hashes",info_type="AvgSec"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="hashes",info_type="CompCount"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="hashes",info_type="CompMergeCPU"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="hashes",info_type="CompSec"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="hashes",info_type="CompactedFiles"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="hashes",info_type="KeyDrop"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="hashes",info_type="KeyIn"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="hashes",info_type="MovedGB"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="hashes",info_type="NumFiles"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="hashes",info_type="RblobGB"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="hashes",info_type="ReadGB"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="hashes",info_type="ReadMBps"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="hashes",info_type="RnGB"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="hashes",info_type="Rnp1GB"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="hashes",info_type="Score"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="hashes",info_type="SizeBytes"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="hashes",info_type="WblobGB"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="hashes",info_type="WnewGB"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="hashes",info_type="WriteAmp"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="hashes",info_type="WriteGB"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="hashes",info_type="WriteMBps"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="lists",info_type="AvgSec"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="lists",info_type="CompCount"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="lists",info_type="CompMergeCPU"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="lists",info_type="CompSec"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="lists",info_type="CompactedFiles"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="lists",info_type="KeyDrop"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="lists",info_type="KeyIn"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="lists",info_type="MovedGB"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="lists",info_type="NumFiles"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="lists",info_type="RblobGB"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="lists",info_type="ReadGB"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="lists",info_type="ReadMBps"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="lists",info_type="RnGB"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="lists",info_type="Rnp1GB"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="lists",info_type="Score"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="lists",info_type="SizeBytes"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="lists",info_type="WblobGB"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="lists",info_type="WnewGB"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="lists",info_type="WriteAmp"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="lists",info_type="WriteGB"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="lists",info_type="WriteMBps"} 0
....

src/storage/src/redis.cc

yaoyinnan · 2023-07-08T15:29:40Z

I think we should discuss which indicators are more important, and expose indicators that are more conducive to helping O&M personnel monitor Pika on grafana. For example, io_stalls can reflect disk I/O performance very well, but there are many compactions stats, so you need to consider whether to display or display part.

chenbt-hz · 2023-07-11T09:31:35Z

I think we should discuss which indicators are more important, and expose indicators that are more conducive to helping O&M personnel monitor Pika on grafana. For example, io_stalls can reflect disk I/O performance very well, but there are many compactions stats, so you need to consider whether to display or display part.

I discussed your suggestion with my DBA colleagues. Perhaps due to the lack of annotation information and experience in use, it is difficult for everyone to judge the specific practical significance of such indicators. This week, I will try to sort out the specific meaning of each indicator to facilitate discussion.

yaoyinnan

A modification suggestion is mainly to supplement the description information of the indicator.

I think this PR can be merged after modification. I discussed this with @AlexStocks. After the plan, you can start a new PR, the pika_exporter obtains the method of Metrics into the form of configurable to obtain a part of the information. Now obtained through the info all command. After, then it can be disassembled into multiple info XXX commands to obtain metrics.

What do you think of this plan?

tools/pika_exporter/README.md

tools/pika_exporter/exporter/metrics/rocksdb.go

tools/pika_exporter/Makefile

yaoyinnan · 2023-07-22T12:05:33Z

@chenbt-hz need to rebase

src/storage/src/redis.cc

tools/pika_exporter/exporter/metrics/parser.go

yaoyinnan

LGTM

…) (OpenAtomFoundation#1696) * 支持cfstats监控数据采集 * 修改监控指标注释;修复时间解析错误bug;添加mac-amd64编译 * 添加监控指标说明 * bug fix

cr-gpt bot reviewed Jul 8, 2023

View reviewed changes

src/storage/src/redis.cc Show resolved Hide resolved

chenbt-hz requested review from yaoyinnan, machinly and luky116 July 8, 2023 11:21

yaoyinnan reviewed Jul 13, 2023

View reviewed changes

tools/pika_exporter/README.md Outdated Show resolved Hide resolved

tools/pika_exporter/README.md Show resolved Hide resolved

tools/pika_exporter/exporter/metrics/rocksdb.go Outdated Show resolved Hide resolved

cr-gpt bot reviewed Jul 22, 2023

View reviewed changes

tools/pika_exporter/Makefile Outdated Show resolved Hide resolved

cr-gpt bot reviewed Jul 22, 2023

View reviewed changes

src/storage/src/redis.cc Show resolved Hide resolved

cr-gpt bot reviewed Jul 22, 2023

View reviewed changes

tools/pika_exporter/exporter/metrics/parser.go Show resolved Hide resolved

chenbt-hz added 3 commits July 22, 2023 20:51

支持cfstats监控数据采集

6dc3f91

修改监控指标注释;修复时间解析错误bug;添加mac-amd64编译

32af579

添加监控指标说明

acbb9fa

chenbt-hz force-pushed the exporter branch from f72656b to acbb9fa Compare July 22, 2023 12:56

yaoyinnan previously approved these changes Jul 22, 2023

View reviewed changes

yaoyinnan self-requested a review July 22, 2023 14:15

bug fix

9f3f475

chenbt-hz dismissed yaoyinnan’s stale review via 9f3f475 July 23, 2023 13:39

cr-gpt bot reviewed Jul 23, 2023

View reviewed changes

tools/pika_exporter/exporter/metrics/parser.go Show resolved Hide resolved

yaoyinnan approved these changes Jul 23, 2023

View reviewed changes

yaoyinnan requested a review from Mixficsol July 23, 2023 13:42

Mixficsol approved these changes Jul 24, 2023

View reviewed changes

Mixficsol merged commit c77a6d3 into OpenAtomFoundation:unstable Jul 24, 2023

yaoyinnan mentioned this pull request Jul 27, 2023

pika观测性支持 #1218

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pika-exporter：Add RocksDB column family metrics(#1566) #1696

pika-exporter：Add RocksDB column family metrics(#1566) #1696

chenbt-hz commented Jul 8, 2023 •

edited

Loading

yaoyinnan commented Jul 8, 2023

chenbt-hz commented Jul 11, 2023

yaoyinnan left a comment

yaoyinnan commented Jul 22, 2023

yaoyinnan left a comment

pika-exporter：Add RocksDB column family metrics(#1566) #1696

pika-exporter：Add RocksDB column family metrics(#1566) #1696

Conversation

chenbt-hz commented Jul 8, 2023 • edited Loading

yaoyinnan commented Jul 8, 2023

chenbt-hz commented Jul 11, 2023

yaoyinnan left a comment

Choose a reason for hiding this comment

yaoyinnan commented Jul 22, 2023

yaoyinnan left a comment

Choose a reason for hiding this comment

chenbt-hz commented Jul 8, 2023 •

edited

Loading