sort ingesters when generating token lookup table fix #6513 #6535
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 2 commits
January 23, 2025 14:36
Signed-off-by: Daniel Sabsay <[email protected]>
Signed-off-by: Daniel Sabsay <[email protected]>
dsabsay
force-pushed
the
flaky-ring-consistency-test-fix
branch
from
2b87236 to
28860c7
Compare
|
Converting to draft because after rebasing |
|
Okay, repro is behaving as expected again (failed twice in a row, passed twice in a row with this fix). This is ready for review. |
|
Ok.. this LGTM.. This function is only called when something in the ring changes so it should not have any perf problem. |
alanprot
approved these changes
Jan 30, 2025
|
But do we ever expect same token to be shared by multliple members of the ring? I believe the chance is pretty low, but in theory it would cause shards to be slightly uneven. |
|
@justinjung04 I believe the same token is shared by multiple members only when |
CharlieTLe
added a commit
to CharlieTLe/cortex
that referenced
this pull request
Feb 13, 2025
Use 128 tokens instead of 10,000. Use ringDesc when generating testValues and creating a ring of ingesters. Fixes: cortexproject#6513 Related: cortexproject#6535 Signed-off-by: Charlie Le <[email protected]>
yeya24
approved these changes
Feb 18, 2025
alanprot
pushed a commit
that referenced
this pull request
Mar 4, 2025
Use 128 tokens instead of 10,000. Use ringDesc when generating testValues and creating a ring of ingesters. Fixes: #6513 Related: #6535 Signed-off-by: Charlie Le <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does:
Sort ingesters when generating token lookup table in ring to fix consistency issues.
Since map iteration order is undefined, successive calls to
getTokensInfo()can return different results when tokens are owned by multiple instances in the ring. I don't know enough about the ring to say whether that is expected to occur, but in the tests it does and this fixes that test.See the linked issue for more details and here for a reproducible test: https://github.com/dsabsay/cortex/tree/flaky-ring-consistency-test-repro
I validated the test locally by running my reproducer (linked above) twice (failed both times), applying this patch and running the same test (passed both times). I don't think it's worth merging the reproducer as it takes 5mins to run utilizing 16 cores fully.
Which issue(s) this PR fixes:
Fixes #6513
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]