Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PurlDB-requested package index scans can fail for larger packages with large scans and many scans at once #1143

Open
3 tasks
keshav-space opened this issue Apr 1, 2024 · 0 comments

Comments

@keshav-space
Copy link
Member

keshav-space commented Apr 1, 2024

send_scan_project_results in purldb-scan-queue-worker isn't able to properly send the large scan results.

This issue fixed the critical problems:

And we can still improve this, using some of these approaches:

  • Send compressed scan results: I was able to compress the ~350MB scan result to under 25MB using zstd or gzip ... xz is way too slow otherwise. For a start using the built in Gzip compression of HTTP clients and servers should help a lot.
  • Adapt timeout to payload: The 60-second DEFAULT_TIMEOUT is not sufficient for sending scan results, we should have a variable timeout depending on the size of the scan results.
  • Consider paginated upload in multiple chunks or streamed upload using JSON lines.
@pombredanne pombredanne changed the title send_scan_project_results is failing for larger scans PurlDB-requested package index scans i failing for larger packages with large scans Apr 3, 2024
@pombredanne pombredanne changed the title PurlDB-requested package index scans i failing for larger packages with large scans PurlDB-requested package index scans is failing for larger packages with large scans Apr 3, 2024
@pombredanne pombredanne changed the title PurlDB-requested package index scans is failing for larger packages with large scans PurlDB-requested package index scans can fail for larger packages with large scans and many scans at once May 8, 2024
@pombredanne pombredanne removed this from 04-purl2sym May 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant