BUG - cannot download ICGC data in AWS EC2 instance

[ggabernet]

Description

Hi, we followed the instructions provided in https://docs.icgc.org/download/guide/#installation-of-the-score-client on how to install the score client in an EC2 instance and download the data with both a manifest file and an object ID, and could not get to make it work.

score-client-3.0.1/bin/score-client download --manifest ~/manifest.aws-virginia.1581368724158.tsv --output-dir /data

ERROR: Command error: org.springframework.web.client.ResourceAccessException: I/O error on GET request for "http://localhost:5431/download/ping": Connect to localhost:5431 [localhost/127.0.0.1] failed: Connection refused (Connection refused); nested exception is org.apache.http.conn.HttpHostConnectException: Connect to localhost:5431 [localhost/127.0.0.1] failed: Connection refused (Connection refused)

We also tried directly getting the url for one of the object IDs:

score-client-3.0.1/bin/score-client url --object-id dd0a01e9-1f89-5e71-81d4-a1e5bef3da08 Resolving URL for object: dd0a01e9-1f89-5e71-81d4-a1e5bef3da08 ERROR: Command error: I/O error on GET request for "http://localhost:5431/download/dd0a01e9-1f89-5e71-81d4-a1e5bef3da08": Connect to localhost:5431 [localhost/127.0.0.1] failed: Connection refused (Connection refused); nested exception is org.apache.http.conn.HttpHostConnectException: Connect to localhost:5431 [localhost/127.0.0.1] failed: Connection refused (Connection refused)

Please check the log for detailed error messages

Expected Behaviour

That the files in the manifest are downloaded.

Actual Behaviour

Got error message:
Resolving URL for object: dd0a01e9-1f89-5e71-81d4-a1e5bef3da08 ERROR: Command error: I/O error on GET request for "http://localhost:5431/download/dd0a01e9-1f89-5e71-81d4-a1e5bef3da08": Connect to localhost:5431 [localhost/127.0.0.1] failed: Connection refused (Connection refused); nested exception is org.apache.http.conn.HttpHostConnectException: Connect to localhost:5431 [localhost/127.0.0.1] failed: Connection refused (Connection refused)

Possible Fix

Steps to Reproduce

1. Start an EC2 instance with Amazon Linux2 CentOS, instance type c4.2xlarge (N. Virginia)
2. Follow instructions on https://docs.icgc.org/download/guide/#installation-of-the-score-client on how to install and configure and run the score client

• we have DACO access including cloud computing
• we have added the access token for AWS download to the config file of the score-client as suggested

3. Run command (with valid manifest)
   score-client-3.0.1/bin/score-client download --manifest ~/manifest.aws-virginia.1581368724158.tsv --output-dir /data

Your Environment

• EC2 instance with Amazon Linux2 CentOS
  NAME="Amazon Linux" VERSION="2" ID="amzn" ID_LIKE="centos rhel fedora" VERSION_ID="2" PRETTY_NAME="Amazon Linux 2" ANSI_COLOR="0;33" CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2" HOME_URL="https://amazonlinux.com/"

[cindyfu]

I also encountered the same problem.

[rtisma]

@ggabernet @cindyfu thank you for creating the ticket. We are aware of this issue, and a release is planned this week that will fix this issue. For the time being, you can explicitly specify the aws profile via --profile aws. So your command would look like:

score-client-3.0.1/bin/score-client --profile aws download --manifest ~/manifest.aws-virginia.1581368724158.tsv --output-dir /data

[ggabernet]

@rtisma thanks, this worked fine now.

[rosibaj]

@ggabernet Glad to hear its working! We have also released a new version of the score-client (3.1) that should fix this issue!

You can download the latest client here - https://docs.icgc.org/software/download/#latest-release-tarball

[ggabernet]

That's awesome news, thanks!