Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Investigate performance issues #587

Open
edgarrmondragon opened this issue Feb 25, 2025 · 0 comments
Open

Investigate performance issues #587

edgarrmondragon opened this issue Feb 25, 2025 · 0 comments
Assignees
@edgarrmondragon
Labels
help wanted Extra attention is needed question Further information is requested

Comments

@edgarrmondragon
Copy link
Member

edgarrmondragon commented Feb 25, 2025 •
edited
Loading

I debugged this ELT (a log file with 13+ GB of content), and there are no traces of redundancy, every record appears only once.

The target-postgres batch_processing_time is an average of 20 s to each batch, and it sums to ~6 minutes of total COPY operation time, which is acceptable.

The tap-postgres average extraction time interval between records is ~0.001 s, what leads (roughly) to a total of 90 minutes of data reading.

If anyone has ideas, evidence (or even better, a flame graph 😉) as to what's causing this performance problem, they're more than welcome!

Related

Links
@edgarrmondragon edgarrmondragon self-assigned this Feb 25, 2025
@edgarrmondragon edgarrmondragon added bug Something isn't working help wanted Extra attention is needed question Further information is requested labels Feb 25, 2025
@edgarrmondragon edgarrmondragon pinned this issue Feb 25, 2025
@edgarrmondragon edgarrmondragon removed the bug Something isn't working label Feb 25, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@edgarrmondragon edgarrmondragon

Labels
help wanted Extra attention is needed question Further information is requested
Projects
MeltanoLabs Overview
Status: Todo
Milestone
No milestone
Development

No branches or pull requests
1 participant
@edgarrmondragon