engine: tag peers based on usefulness #191
Conversation
This patch tracks two usefulness metrics: short-term usefulness and long-term usefulness. Short-term usefulness is sampled frequently and highly weights new observations. Long-term usefulness is sampled less frequently and highly weights on long-term trends. In practice, we do this by keeping two EWMAs. If we see an interaction within the sampling period, we record the score, otherwise, we record a 0. The short-term one has a high alpha and is sampled every shortTerm period. The long-term one has a low alpha and is sampled every longTermRatio*shortTerm period. To calculate the final score, we sum the short-term and long-term scores then adjust it ±25% based on our debt ratio. Peers that have historically been more useful to us than we are to them get the highest score.
| package decision | ||
|
|
||
| func ewma(old, new, alpha float64) float64 { | ||
| return new*alpha + (1-alpha)*old |
There was a problem hiding this comment.
@Stebalien I know I am late to this PR but if you are willing to shoulder the import, I highly recommend using gonum/floats for the multiplications as it is significantly faster than stdlib.
There was a problem hiding this comment.
I'd rather keep this case simple as I don't think this is going to be a bottleneck.
@Stebalien I didn't see anywhere that exposed the values as metrics for them to be able to be collected and exposed in grafana? |
The connection manager exposes them but we'd need to find some way to pipe them out to grafana. Once we do that, I'd like to pipe dht routing tables and bitswap ledgers out as well so we can compare values and see how accurate our measurements are. |
engine: tag peers based on usefulness This commit was moved from ipfs/go-bitswap@fef4be2
This patch tracks two usefulness metrics: short-term usefulness and long-term usefulness. Short-term usefulness is sampled frequently and highly weights new observations. Long-term usefulness is sampled less frequently and highly weights on long-term trends.
In practice, we do this by keeping two EWMAs. If we see an interaction within the sampling period, we record the score, otherwise, we record a 0. The short-term one has a high alpha and is sampled every shortTerm period. The long-term one has a low alpha and is sampled every longTermRatio*shortTerm period.
To calculate the final score, we sum the short-term and long-term scores then adjust it ±25% based on our debt ratio. Peers that have historically been more useful to us than we are to them get the highest score.
The number of hard-coded constants makes me really uncomfortable. I'm wondering if we should:
What I'd like to do is try this on the gateways and cluster and see what happens. We'll probably need to somehow feed connection manager information into grafana to get a good picture of whether or not this is working.