Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add RocksDB column family metrics #1566

Closed
yaoyinnan opened this issue Jun 1, 2023 · 2 comments · Fixed by #1696
Closed

Add RocksDB column family metrics #1566

yaoyinnan opened this issue Jun 1, 2023 · 2 comments · Fixed by #1696
Assignees
@chenbt-hz @yaoyinnan

Comments

@yaoyinnan
Copy link
Contributor

In the process of adding RocksDB metrics in #1560, the RocksDB column family metrics were not obtained, and this part of metrics needs to be added.
@yaoyinnan yaoyinnan self-assigned this Jun 1, 2023
@yaoyinnan
Copy link
Contributor Author

Ref: https://github.com/gukaifeng/gukaifeng.cn/blob/main/_posts/RocksDB%20--%20DB__Properties%20%E4%B8%AD%E7%9A%84%E5%B1%9E%E6%80%A7%E8%AF%A6%E8%A7%A3.md

kCFStats 包含的是 kCFStatsNoFileHistogram 和 kCFFileHistogram 的 Map 格式的信息。

根据参考的文档 kCFStatsNoFileHistogram 应该如下：

** Compaction Stats [default] **
Level    Files      Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB)
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  L0      6/5       2.12 GB   1.4      0.0     0.0      0.0  199099.8 199099.8       0.0   1.0      0.0    269.5 756504.74         250346.23    554853    1.363       0      0       0.0       0.0
  L1     38/38      1.12 GB   0.0 249127.6 199097.7  50029.9  248731.1 198701.2       0.0   1.2    134.9    134.7 1890421.55         452407.19    200791    9.415     93G   156M       0.0       0.0
  L2     60/0     979.08 MB   1.0 213539.6 198700.1  14839.5   86964.5  72125.0       0.0   0.4     13.0      5.3 16830463.52         691433.34   4870877    3.455     89G   580M       0.0       0.0
  L3    309/2       4.05 GB   1.0  28156.2 19799.1   8357.2   23635.3  15278.1   52325.9   1.2      9.3      7.8 3109912.67         407217.85    268347   11.589     29G  2260M       0.0       0.0
  L4    982/15     16.19 GB   1.0  48411.4 25620.9  22790.4   33402.8  10612.4   41983.0   1.3      9.5      6.6 5221179.63         716863.16    758257    6.886     52G    13G       0.0       0.0
  L5    205/2      29.18 GB   0.5 146746.1 52595.2  94151.0   94151.0      0.0       0.0   1.8     10.1      6.5 14814832.53        2756096.66    494153   29.980    182G    58G       0.0       0.0
 Sum   1600/62     53.61 GB   0.0 685981.0 495813.0 190168.0  685984.5 495816.4   94308.9   3.4     16.5     16.5 42623314.64        5274364.44   7147278    5.964    446G    74G       0.0       0.0
 Int      0/0       0.00 KB   0.0      0.4     0.4      0.1       0.7      0.6       0.0   2.0     10.5     17.3     41.22              6.14         3   13.740    449K    43K       0.0       0.0

** Compaction Stats [default] **
Priority Files      Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB)
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Low      0/0       0.00 KB   0.0 685981.0 495813.0 190168.0  486884.7 296716.6       0.0   0.0     16.8     11.9 41866809.91        5024018.21   6592425    6.351    446G    74G       0.0       0.0
High      0/0       0.00 KB   0.0      0.0     0.0      0.0  199099.8 199099.8       0.0   0.0      0.0    269.5 756504.74         250346.23    554853    1.363       0      0       0.0       0.0

Blob file count: 0, total size: 0.0 GB

Uptime(secs): 4728090.8 total, 14.0 interval
Flush(GB): cumulative 199099.803, interval 0.356
AddFile(GB): cumulative 0.000, interval 0.000
AddFile(Total Files): cumulative 0, interval 0
AddFile(L0 Files): cumulative 0, interval 0
AddFile(Keys): cumulative 0, interval 0
Cumulative compaction:  669.906709 TB write,  148.57 MB/s write, 685980.985360 GB read,  148.57 MB/s read, 42623314.6 seconds
Interval   compaction:    0.694616 GB write,   50.67 MB/s write,    0.424185 GB read,   30.94 MB/s read,    41.2 seconds
Stalls(count): 2273408 level0_slowdown, 1603632 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 10738 slowdown for pending_compaction_bytes, 0 memtable_compaction, 2 memtable_slowdown, interval 9 total count

包含一些 io_stalls，如 level0_numfiles、level0_slowdown、memtable_compaction、memtable_slowdown、stop_for_pending_compaction_bytes、slowdown_for_pending_compaction_bytes。

但是测试时发现获取到的 kCFStatsNoFileHistogram 是：

** Compaction Stats [default] **
Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB)
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Sum      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0      0.00              0.00         0    0.000       0      0       0.0       0.0
 Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0      0.00              0.00         0    0.000       0      0       0.0       0.0

** Compaction Stats [default] **
Priority    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB)
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0

Uptime(secs): 5.4 total, 5.4 interval
Flush(GB): cumulative 0.000, interval 0.000
AddFile(GB): cumulative 0.000, interval 0.000
AddFile(Total Files): cumulative 0, interval 0
AddFile(L0 Files): cumulative 0, interval 0
AddFile(Keys): cumulative 0, interval 0
Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 0, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 0, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 0, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 0, total-stops: 0

最后一行变为 Write Stall (count)：

Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 0, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 0, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 0, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 0, total-stops: 0

需要重新判断需要输出的 IO stat 指标。

此外，正常获得指标的时候是从 kStats 和 kCFStats 的 Map 中读取。

kCFStats 指标如下：

{cf-l0-file-count-limit-delays-with-ongoing-compaction: 0}
{cf-l0-file-count-limit-stops-with-ongoing-compaction: 0}
{compaction.Sum.AvgSec: 0.000000}
{compaction.Sum.CompCount: 0.000000}
{compaction.Sum.CompMergeCPU: 0.000000}
{compaction.Sum.CompSec: 0.000000}
{compaction.Sum.CompactedFiles: 0.000000}
{compaction.Sum.KeyDrop: 0.000000}
{compaction.Sum.KeyIn: 0.000000}
{compaction.Sum.MovedGB: 0.000000}
{compaction.Sum.NumFiles: 0.000000}
{compaction.Sum.RblobGB: 0.000000}
{compaction.Sum.ReadGB: 0.000000}
{compaction.Sum.ReadMBps: 0.000000}
{compaction.Sum.RnGB: 0.000000}
{compaction.Sum.Rnp1GB: 0.000000}
{compaction.Sum.Score: 0.000000}
{compaction.Sum.SizeBytes: 0.000000}
{compaction.Sum.WblobGB: 0.000000}
{compaction.Sum.WnewGB: 0.000000}
{compaction.Sum.WriteAmp: 0.000000}
{compaction.Sum.WriteGB: 0.000000}
{compaction.Sum.WriteMBps: 0.000000}
{l0-file-count-limit-delays: 0}
{l0-file-count-limit-stops: 0}
{memtable-limit-delays: 0}
{memtable-limit-stops: 0}
{pending-compaction-bytes-delays: 0}
{pending-compaction-bytes-stops: 0}
{total-delays: 0}
{total-stops: 0}

@AlexStocks
Copy link
Contributor

AlexStocks commented Jul 7, 2023 •
edited by luky116
Loading 

        * 目前未显示 CFStats Metrics，需要支持，但测试发现获取不到，还需要人跟进
        * 0610 分配给 baitao
        * 0624 问题明确，继续跟进
        * 0701 继续跟进
        * 0708 baitao 已提交 pr

@yaoyinnan yaoyinnan mentioned this issue Jul 8, 2023
Merged
Mixficsol pushed a commit that referenced this issue Jul 24, 2023
@chenbt-hz
pika-exporter：Add RocksDB column family metrics(#1566) (#1696)
c77a6d3 
* 支持cfstats监控数据采集

* 修改监控指标注释;修复时间解析错误bug;添加mac-amd64编译

* 添加监控指标说明

* bug fix
bigdaronlee163 pushed a commit to bigdaronlee163/pika that referenced this issue Jun 8, 2024
@chenbt-hz
pika-exporter：Add RocksDB column family metrics(OpenAtomFoundation#1566…
8b4a71c 
…) (OpenAtomFoundation#1696)

* 支持cfstats监控数据采集

* 修改监控指标注释;修复时间解析错误bug;添加mac-amd64编译

* 添加监控指标说明

* bug fix
cheniujh pushed a commit to cheniujh/pika that referenced this issue Sep 24, 2024
@chenbt-hz
pika-exporter：Add RocksDB column family metrics(OpenAtomFoundation#1566…
a85d46a 
…) (OpenAtomFoundation#1696)

* 支持cfstats监控数据采集

* 修改监控指标注释;修复时间解析错误bug;添加mac-amd64编译

* 添加监控指标说明

* bug fix
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@chenbt-hz chenbt-hz

@yaoyinnan yaoyinnan

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

pika-exporter：Add RocksDB column family metrics(#1566) chenbt-hz/pika
3 participants
@AlexStocks @chenbt-hz @yaoyinnan