The indexing of (very) large packages never completes

[keshav-space]

/api/scan_queue/update_status takes too long to return for large scans

It appears like update_status is doing too many things which takes it forever to return, in some case over an hour. Here is a design, breaking the scan upload process in two parts:

• First simply collect the scan results (possibly stored in a local temp file), and return immediately, possibly updating the status to some intermediate status. This should accept compressed data.
• Push the index_package step to Redis queue (we are already using the Redis queue for PURL watch feature) to process asynchronously and update the status when completed

Also see:

• PurlDB-requested package index scans can fail for larger packages with large scans and many scans at once scancode.io#1143