(feat) add thread pool metric report

[fengjiachun]

Motivation:

add thread pool metric report on kill -s SIGUSR2 $pid

Modification:

• thread pool's metrics Handle NodeImpl disruptor event error #338
• some test case may hang on travis Some test cases may hang on travis #333
• shutdown order Handle NodeImpl disruptor event error #338
• fix: forget to stop TestCluster on test failed, this may be the root cause by 'hang on travis'

Result:

thread pool's metrics example:

`

19-11-14 10:40:12 ==============================================================

-- Timers ----------------------------------------------------------------------
threadPool.JRAFT_CLOSURE_EXECUTOR
count = 1574
mean rate = 17.69 calls/second
1-minute rate = 16.24 calls/second
5-minute rate = 11.43 calls/second
15-minute rate = 10.13 calls/second
min = 0.06 milliseconds
max = 2.63 milliseconds
mean = 0.21 milliseconds
stddev = 0.13 milliseconds
median = 0.17 milliseconds
75% <= 0.22 milliseconds
95% <= 0.40 milliseconds
98% <= 0.48 milliseconds
99% <= 0.61 milliseconds
99.9% <= 1.60 milliseconds
threadPool.JRaft-RPC-Processor
count = 2
mean rate = 0.02 calls/second
1-minute rate = 0.11 calls/second
5-minute rate = 0.31 calls/second
15-minute rate = 0.37 calls/second
min = 21.48 milliseconds
max = 31.71 milliseconds
mean = 26.59 milliseconds
stddev = 5.12 milliseconds
median = 31.71 milliseconds
75% <= 31.71 milliseconds
95% <= 31.71 milliseconds
98% <= 31.71 milliseconds
99% <= 31.71 milliseconds
99.9% <= 31.71 milliseconds
threadPool.rheakv-cli-rpc-executor
count = 2
mean rate = 0.18 calls/second
1-minute rate = 0.20 calls/second
5-minute rate = 0.20 calls/second
15-minute rate = 0.20 calls/second
min = 0.18 milliseconds
max = 2.56 milliseconds
mean = 1.30 milliseconds
stddev = 1.19 milliseconds
median = 0.18 milliseconds
75% <= 2.56 milliseconds
95% <= 2.56 milliseconds
98% <= 2.56 milliseconds
99% <= 2.56 milliseconds
99.9% <= 2.56 milliseconds
threadPool.rheakv-kv-store-rpc-executor
count = 36
mean rate = 3.18 calls/second
1-minute rate = 2.93 calls/second
5-minute rate = 2.83 calls/second
15-minute rate = 2.81 calls/second
min = 0.09 milliseconds
max = 82.06 milliseconds
mean = 3.10 milliseconds
stddev = 13.20 milliseconds
median = 0.15 milliseconds
75% <= 0.32 milliseconds
95% <= 15.53 milliseconds
98% <= 82.06 milliseconds
99% <= 82.06 milliseconds
99.9% <= 82.06 milliseconds
`

[fengjiachun]

@killme2008 可以看下这个 pr，除了 ci 超过 50 分钟，暂时没再发现 ci hang 住的情况

[killme2008]

new FileOutputStream(file, true) 也应该抽成局部变量吧，保证关闭，用逗号隔开

[fengjiachun]

这个没问题的，PrintStream 在关闭时也会关闭它内部的 outputstream，会顺带把 FileOutputStream 关掉

[killme2008]

不应该依赖实现的，从代码角度，应该确保每个打开的都关闭

[fengjiachun]

算是利用了一个java IO Stream 的一个规范，在其他场景，我不应该这么做。

java 的 IO Stream 设计为装饰着模式，一层包一层，外层的 stream 有责任且必须关闭自己以及任何和此流相关的所有系统资源

以下内容来自 OutputStream close 接口的注释说明：
Closes this output stream and releases any system resources associated with this stream.

[fengjiachun]

这样做有一个好处，不用考虑各个被嵌套的流的关闭顺序，比如不用考虑先 flush BufferedOutputStream 再 close FileOutputStream 这种先后逻辑

[killme2008]

这个位置为什么调整？ 需要放到 thread 里异步跑吗

[fengjiachun]

afterShutdown 原本在 fsm 的 doShutdown 里面，这有一个问题，就是 doShutdown 被调用后 fsm 实质上并没有真的被关闭，afterShutdown 里负责关闭 rocksdb，这样 rocksdb 会被提前关掉，从而导致 fsm 在还有往 rocksdb 写入的情况下 rocksdb 关掉导致的异常