Feat/cram stream

[AssafSternberg]

Aim: Add stream_cram method to MODO, that allows to stream both local and remote CRAM files.##

PR elements:

1. Refactor of the region parser to allow a wider range of range request formats:

The old region parser did not allow the user to ask for an entire chromosome (e.g., "chr1"), a one side open-ended region (e.g., "chr1:10000" or "chr1:-10500"), or un-mapped reads (e.g., "*"). The refactored parser deals with such requests. Due to the possibility of start and/or end to be None, the output was changed to tuple[chrom: str, start: str, end: str].

2. Refactor slice_cram() to handle the updated output of parse_region():

The old version of slice_cram() worked only with a "reference:start-end" region format, expecting start and end as integers. The New version handles start and end being strings or None.

3. Add slice_remote_cram() to handle remote cram files:

The new function streams or writes to a local file the region of a remote CRAM file. The function uses the helper functions:

• filename_from_url(): extracts the filename from the given URL.
• hstget_command(): formats a valid htsget command line request.

4. Add htsget_url function:

Currently not used, but might be useful in the future. Similar to htsget_command(), it returns a valid htsget URL that can be used in command line use (e.g., 'samtools view -H http://localhost/htsget/reads/ex/demo1?referenceName=chr1&start=10000&end=10500&format=CRAM')

5. Add stream_cram method to MODO

The method format is identical for local and remote files. First, it checks that the CRAM file actually exists in the MODO. If it does, based on the existence of s3_endpoint it either calls slice_cram() or slice_remote_cram(). Currently local files are always streamed. For remote files, if an output filename is given, the data is written to a local file. If not, the data is streamed.

[almutlue]

Thanks a lot for the PR. I left some questions and suggestions on mainly minor things, but I have one main point:

• using an os.system call to run htseq makes the code more complicated and inconsistent. I think we should use python instead, as this is what the htseq cli calls anyways. We could either use the htseq-api with output= sys.stdout.buffer(as they do in https://github.com/ga4gh/htsget/blob/ac3ba43fd017fc1db173d323a5de2987a354fdb6/htsget/cli.py#L72) or simply write our own requests? I did not manage to fully unravel their code, but it is rather short and looks like little extra to the request call. The code looks slightly smelly to me (not addressed todos from >2 years, hard coded values, https://github.com/ga4gh/htsget/blob/master/htsget/io.py). So maybe it is worth to implement by ourselves?

[AssafSternberg]

Thanks for the great comments and suggestions @almutlue !

Do you @cmdoret have any further comments ... or disagree with any of the above resolutions ... or can I go ahead and merge the branch with main (and if yes, which one should I use 'create a merge commit', 'squash and merge', or 'rebase and merge'?)

[almutlue]

Question: Do we need another example dataset? Can we remove test files before merging?

[almutlue]

Thanks Assaf.
This looks good to me and works on my side as well.
I needed to specify a local reference file to make it work. It would be nice, if it could use the remote reference as default, but I think that is for a new PR.