Remove unnecessary logs and add Ender metrics #2133
Conversation
WalkthroughThe recent changes enhance observability and performance tracking within the indexing system, particularly with the Kafka producer and message processing. New metrics tracking has been integrated for batch sizes and send durations, while logging has been refined for clarity and insight. Overall, these updates aim to streamline operations, maintain essential functionalities, and support efficient monitoring and debugging processes. Changes
Sequence Diagram(s)sequenceDiagram
participant Producer
participant Kafka
participant Consumer
Producer->>Kafka: Send batch of messages
Kafka->>Producer: Acknowledge receipt
Producer->>Producer: Track metrics (size, duration)
Consumer->>Kafka: Fetch messages
Kafka->>Consumer: Deliver messages
Consumer->>Consumer: Process messages
Consumer->>Consumer: Log processing details
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (invoked as PR comments)
Additionally, you can add CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files selected for processing (4)
- indexer/packages/kafka/src/batch-kafka-producer.ts (3 hunks)
- indexer/services/ender/src/lib/batched-handlers.ts (1 hunks)
- indexer/services/ender/src/lib/on-message.ts (1 hunks)
- indexer/services/ender/src/lib/sync-handlers.ts (1 hunks)
Files skipped from review due to trivial changes (2)
- indexer/services/ender/src/lib/batched-handlers.ts
- indexer/services/ender/src/lib/sync-handlers.ts
Additional comments not posted (4)
indexer/packages/kafka/src/batch-kafka-producer.ts (3)
1-1: Import ofstatsis appropriate.The addition of
statsis necessary for the new metrics functionality.
69-69: Initialization ofstartTimeis correct.Capturing the start time before sending the batch is necessary for timing metrics.
87-88: Metrics tracking withstats.gaugeandstats.timingis well-implemented.These additions provide valuable insights into batch processing performance.
indexer/services/ender/src/lib/on-message.ts (1)
76-83: Logging enhancement inonMessageis comprehensive.The
logger.infocall captures essential details for monitoring message processing.
There was a problem hiding this comment.
Actionable comments posted: 0
Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files selected for processing (3)
- indexer/packages/kafka/src/batch-kafka-producer.ts (3 hunks)
- indexer/services/ender/src/lib/batched-handlers.ts (2 hunks)
- indexer/services/ender/src/lib/candles-generator.ts (2 hunks)
Files skipped from review as they are similar to previous changes (1)
- indexer/services/ender/src/lib/batched-handlers.ts
Additional comments not posted (3)
indexer/packages/kafka/src/batch-kafka-producer.ts (2)
1-1: Import changes look good.The addition of the
statsmodule is appropriate for the new metrics functionality.
Line range hint
69-88: Metrics integration looks good.The
stats.gaugeandstats.timingcalls effectively enhance observability by tracking batch size and send duration.Consider verifying the impact of these metrics on performance, especially under high load.
Run the following script to check for any potential performance issues related to the new metrics:
Verification successful
No Immediate Performance Concerns Found for Metrics Integration
The search did not reveal any performance-related comments or TODOs in the codebase concerning the new metrics in
batch-kafka-producer.ts. However, consider conducting manual performance testing under high load conditions to ensure these metrics do not introduce latency.
- File:
indexer/packages/kafka/src/batch-kafka-producer.ts- Line: 82-91
Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Verify the impact of metrics on performance. # Test: Search for any performance-related comments or TODOs in the codebase. rg --type ts 'performance|TODO'Length of output: 24444
indexer/services/ender/src/lib/candles-generator.ts (1)
Line range hint
537-554: Timing metrics integration looks good.The addition of
stats.timingeffectively tracks the execution time of thegetOrderbookMidPriceMapfunction, aiding in performance monitoring.Consider verifying the accuracy of the timing metrics to ensure they reflect the actual execution time.
Run the following script to check for any discrepancies in the timing metrics:
|
@mergify backport indexer/release/v6.x |
❌ No backport have been created
GitHub error: |
|
@mergify backport release/indexer/v6.x |
✅ Backports have been created
|
(cherry picked from commit 3c13930)
Co-authored-by: Christopher-Li <[email protected]>
Changelist
[Describe or list the changes made in this PR]
Test Plan
[Describe how this PR was tested (if applicable)]
Author/Reviewer Checklist
state-breakinglabel.indexer-postgres-breakinglabel.PrepareProposalorProcessProposal, manually add the labelproposal-breaking.feature:[feature-name].backport/[branch-name].refactor,chore,bug.Summary by CodeRabbit
New Features
Bug Fixes
Refactor