Downloading node is super slow #562

SimenB · 2016-12-09T13:04:14Z

X-post: nodejs/nodejs.org#1061

Our builds are currently failing because of this. Sorry if it's the wrong repo

EloB · 2016-12-09T13:10:05Z

I've tried to reinstall node and now I can't work because if this :(

SimenB · 2016-12-09T13:11:42Z

It usually completes in the end, but not fast enough for CI

jbergstroem · 2016-12-09T13:36:42Z

I'll have a look and see if something is up with cloudflare/digitalocean.

EloB · 2016-12-09T13:43:25Z

@jbergstroem https://www.cloudflarestatus.com/ they have problem.

SimenB · 2016-12-09T13:45:47Z

Either this or nodejs/nodejs.org#1061 should probably be closed, to keep it in one place 😄

targos · 2016-12-09T13:47:00Z

Not sure it's a cloudflare issue. This is the error I get:

SimenB · 2016-12-09T13:48:01Z

I did get some TLS errors initially, before it just plain timed out

PeterDaveHello · 2016-12-09T13:52:29Z

I wonder if there is any chance that we can build/provide mirror service in case we have some familiar problem again?

jbergstroem · 2016-12-09T13:54:02Z

Traffic is within 5% of what we can expect around this time of day. I didn't see any out of the ordinary errors in our nginx error log and our ssl certs are valid.

agsmorodin · 2016-12-09T13:54:28Z

Issue still exists.
Here is my speed now, Amsterdam location:

wget https://nodejs.org/dist/v0.12.2/node-v0.12.2-linux-x64.tar.gz
--2016-12-09 13:43:57--  https://nodejs.org/dist/v0.12.2/node-v0.12.2-linux-x64.tar.gz
Resolving nodejs.org (nodejs.org)... 104.20.22.46, 104.20.23.46, 2400:cb00:2048:1::6814:162e, ...
Connecting to nodejs.org (nodejs.org)|104.20.22.46|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 9487774 (9.0M) [application/gzip]
Saving to: 'node-v0.12.2-linux-x64.tar.gz.5

 6% [=========>                                                                                                                                                         ] 638,731     14.6KB/s   in 3m 5s  

2016-12-09 13:47:11 (3.36 KB/s) - Connection closed at byte 638731. Retrying.

--2016-12-09 13:47:12--  (try: 2)  https://nodejs.org/dist/v0.12.2/node-v0.12.2-linux-x64.tar.gz
Connecting to nodejs.org (nodejs.org)|104.20.22.46|:443... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 9487774 (9.0M), 8849043 (8.4M) remaining [application/gzip]
Saving to: 'node-v0.12.2-linux-x64.tar.gz.5
1,539,008   5.44KB/s  eta 43m 48s^C

jbergstroem · 2016-12-09T13:55:12Z

(I'm not saying we're not experiencing issues, just taking notes from what I can measure)

malaire · 2016-12-09T13:57:21Z

Same problem when downloading from Finland -- all larger (few MB or more) downloads fail with Partial Content.

jbergstroem · 2016-12-09T14:04:45Z

I'm getting pretty awful packet loss to the origin host; investigating.

mhdawson · 2016-12-09T14:09:58Z

I can confirm that even just trying to load benchmarking.nodejs.org is taking a really long time from Ottawa/Canada

jbergstroem · 2016-12-09T14:19:46Z

Definitely something going on at our host. I've gotten in touch with support but will try to debug as much as possible (which as you know is a great experience over ssh with packet loss).

Update:

root disk io: just fine
block storage io: all good
networking in/out: broken
cpu/ram: nothing out of the ordinary (except higher interrupts waiting for network)
open tcp connections: low enough to at least rule out starvation attempts

We have rsync jobs almost two hours old (backup, benchmarking, rsync dist) which would point to network degradation at least at that point. Bandwidth history seems to be pretty consistent though..

bennycode · 2016-12-09T15:17:26Z

I saw that Travis is downloading Node.js versions via nvm. Even if they are pre-installed on the VM.

Here is what is preinstalled (Build system information):

Here is the log entry from nvm:

I don't know why it's doing that (downloading Node.js v6.9.1 again) because we have set an explicit version (which should be pre-installed) in our .travis.yml:

language: node_js
node_js:
  - 6.9.1

Build Log:

https://travis-ci.org/wireapp/wire-webapp/builds/182614458

PeterDaveHello · 2016-12-09T15:19:33Z

@bennyn I think nvm shows the reason as the checksum not matched.

PeterDaveHello · 2016-12-09T15:20:47Z

@jbergstroem is there any chance to provide an alternative or mirror site so that we can tolerance the problem until it's fixed? Thanks.

jbergstroem · 2016-12-09T15:26:45Z

@PeterDaveHello you can try using http://unencrypted.nodejs.org. It is a mirror but shouldn't be treated as such when nodejs.org is unavailable (read: this is not the official mirror, just an unencrypted version for old http clients)

jbergstroem · 2016-12-09T15:27:00Z

Update: The issue is identified, I'm working with DO support to resolve it.

matthewp · 2016-12-09T15:30:57Z

There's a second issue I think should be addressed. If a .travis.yml has a Node version set as 4 or 7 or whatever else and the download fails, Travis shouldn't continue on defaulting to 0.10 as it currently is doing. It should fail immediately, as it can't build what the user intended.

EDIT: whoops, look like that's a new issue for that now. Awesome!

SimenB · 2016-12-09T15:33:13Z

@matthewp that's an issue with travis, or maybe nvm. Not nodejs.org

matthewp · 2016-12-09T15:33:39Z

Correct! Sorry for the noise.

jbergstroem · 2016-12-09T16:01:24Z

@PeterDaveHello ouch. I'm updating it now -- give it a few minutes.

PeterDaveHello · 2016-12-09T16:02:32Z

@jbergstroem is it possible to provide an official method to mirror it?

jbergstroem · 2016-12-09T16:02:56Z

@PeterDaveHello unencrypted.nodejs.org is in preparations of being both a http and rsync mirror. We will officially announce it when done.

PeterDaveHello · 2016-12-09T16:03:36Z

@jbergstroem that's awesome, thanks for the info! Maybe you can consider still enable https on it but just not force https on it, so one more choice, and safer.

bear · 2016-12-09T16:03:42Z

@jbergstroem CircleCI SRE/Ops person here ... after you have recovered from the event please do reach out to me (us) if we can be of any assistance with mirrors

jbergstroem · 2016-12-09T16:04:13Z

Progress update: we will have to move the origin host to finally resolve the network issues. Should not take more than 20 minutes.

PeterDaveHello · 2016-12-09T16:07:35Z

@bear it'll be good to have a local mirror inside Circle CI 👍

jbergstroem · 2016-12-09T16:12:16Z

@bear: thanks for reaching out.

For those of you that are interested in mirroring; unencrypted.nodejs.org will provide both rsync and http endpoints in a not-too-far future. We will announce this as part of migratating nodejs.org to enforce HSTS (and ssl only).

jbergstroem · 2016-12-09T16:29:55Z

20 minutes has passed, so another update: Host is back up; we're still working with support to resolve networking issues - but hopefully this'll come to an end very shortly.

jbergstroem · 2016-12-09T16:53:53Z

Update: we're back and catching up. I will keep a close look at performance and will finally close issues once everything is back. Expect a followup blog post / issue once we've collected all information.

mikeal · 2016-12-09T18:25:06Z

I wonder if there are some Cloudflare settings we can use to make the caching more aggressive and not run into this in the future.

For our downloads, all of the content is "append only" right? We never replace resources that we put up there so we could make the caching super aggressive.

/cc @terinjokes

dragon788 · 2016-12-09T18:35:19Z

@jbergstroem You should check out Mosh, https://mosh.org/. At least that way your ssh session will feel responsive instead of utter tripe when on an unreliable connection.

jbergstroem · 2016-12-09T18:40:43Z

@mikeal: there's a few things we can do to improve a situation where our origin runs into availability issues:

change the way we generate http logs so we can have cloudflare face all downloads. this requires a few things, most importantly changing our account type at cf
set up a different ruleset for all downloads throuh cloudflare and add redundancy, for instance from unencrypted.nodejs.org once that treated as official.
further explore distributed mirrors and organize them similarly to how other operating system/package managers works.

@mikeal you're correct -- each node.js artifact is treated as read-only once generated (we have checksums for all of them), which means that cf could fully front us once 1. above is accomplished.

jbergstroem · 2016-12-09T18:43:13Z

Going to close this issue now, since we've resolved all outstanding issues. Please let us know if you find anything not working. Thanks for having patience with us!

mikeal · 2016-12-09T18:55:09Z

changing our account type at cf

Which account type would we need? I thought we had a comp'd enterprise account?

jbergstroem · 2016-12-09T18:57:06Z

@mikeal: it used to be enterprise, now is business.

terinjokes · 2016-12-09T20:53:07Z

@mikeal I'm happy to chat with you offline and see what we can do.

…

On Dec 9, 2016 10:57 AM, "Johan Bergström" ***@***.***> wrote: @mikeal <https://github.com/mikeal>: it used to be enterprise, now is business. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#562 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAQsZQ0pOaZsSg4Ih8qmui6P1dc_FrH7ks5rGaSEgaJpZM4LI6ur> .

soberstadt · 2016-12-12T14:16:39Z

Thank you for your hard work! It was a pretty minor inconvenience for me, my Travis builds were not working during the downtime. I assuming that most others were experiencing simular pain. Moving forward, I added this to my .travis.yml file to cache my node version to avoid the build dependency:

cache:
  directories:
    - node_modules
    - ../../../.nvm/versions

mikeal · 2016-12-12T21:43:29Z

I spoke with Terin about this. With our current account there are more aggressive caching rules we can set. @terinjokes can give use the exact syntax.

jbergstroem · 2016-12-13T01:02:44Z

@terinjokes perhaps drop by our irc channel and have a chat? I'm jbergstroem@freenode, channel is #node-build.

jbergstroem · 2016-12-13T01:53:19Z

@mikeal see my comment above as why we don't cache "everything".

PeterDaveHello · 2017-04-09T08:44:47Z

@jbergstroem Hi! do you still plan to provide official mirror mechanism? Looks like http://unencrypted.nodejs.org/dist/ is down now? Since nodejs.org was down again few days ago and I don't know if there is any ETA on it, I can only use other stupid method to mirror it right now, to prevent the next server down and provide better performance 😉

To those who needs a mirror, you can try: http://ftp.yzu.edu.tw/nodejs.org-dist,
and export to environment variable for node version manager ✨

export NVM_NODEJS_ORG_MIRROR="http://ftp.yzu.edu.tw/nodejs.org-dist"

jbergstroem · 2017-04-09T18:21:36Z

@PeterDaveHello it's not gone; its mirrored at http://unencrypted.nodejs.org/download/. The landing page I created to say it was unencrypted was temporarily 404'ing (not sure we should even keep it).

mohamedalighouma · 2018-08-09T19:40:20Z

the same here

jbergstroem mentioned this issue Dec 9, 2016

The website is super slow, breaking builds using nvm nodejs/nodejs.org#1061

Closed

targos mentioned this issue Dec 9, 2016

Refactor networks mljs/feedforward-neural-networks#7

Merged

MylesBorins mentioned this issue Dec 9, 2016

The installation package could not be opened nodejs/node#10195

Closed

bnoordhuis mentioned this issue Dec 9, 2016

Something is wrong with the installers and download page nodejs/node#10198

Closed

Turbo87 mentioned this issue Dec 9, 2016

Fail build if nvm install fails travis-ci/travis-ci#7001

Closed

pzrq mentioned this issue Dec 9, 2016

COMPASS-521 Collection List hangs after applying a query mongodb-js/compass#668

Merged

chgibb mentioned this issue Dec 9, 2016

Merge Dev to Master chgibb/PHAT#55

Closed

PeterDaveHello mentioned this issue Dec 9, 2016

Mirror nodejs/nodejs.org#1063

Closed

MylesBorins mentioned this issue Dec 9, 2016

Node Download Page is DOWN nodejs/nodejs.org#1064

Closed

brew mentioned this issue Dec 9, 2016

Move 404 and 500 templates to /views dir okfn/opendatasurvey#861

Merged

jbergstroem closed this as completed Dec 9, 2016

BanzaiMan mentioned this issue Dec 11, 2016

npm Github reference: Permission denied travis-ci/travis-ci#7006

Closed

valscion mentioned this issue Mar 17, 2017

nodejs.org download site is down? Can't download Node #653

Closed

Downloading node is super slow #562

Downloading node is super slow #562

Comments

SimenB commented Dec 9, 2016

EloB commented Dec 9, 2016

SimenB commented Dec 9, 2016 • edited Loading

jbergstroem commented Dec 9, 2016

EloB commented Dec 9, 2016

SimenB commented Dec 9, 2016

targos commented Dec 9, 2016

SimenB commented Dec 9, 2016

PeterDaveHello commented Dec 9, 2016

jbergstroem commented Dec 9, 2016

agsmorodin commented Dec 9, 2016

jbergstroem commented Dec 9, 2016

malaire commented Dec 9, 2016

jbergstroem commented Dec 9, 2016

mhdawson commented Dec 9, 2016

jbergstroem commented Dec 9, 2016 • edited Loading

bennycode commented Dec 9, 2016

PeterDaveHello commented Dec 9, 2016

PeterDaveHello commented Dec 9, 2016

jbergstroem commented Dec 9, 2016

jbergstroem commented Dec 9, 2016

matthewp commented Dec 9, 2016 • edited Loading

SimenB commented Dec 9, 2016

matthewp commented Dec 9, 2016

jbergstroem commented Dec 9, 2016

PeterDaveHello commented Dec 9, 2016

jbergstroem commented Dec 9, 2016

PeterDaveHello commented Dec 9, 2016 • edited Loading

bear commented Dec 9, 2016

jbergstroem commented Dec 9, 2016

PeterDaveHello commented Dec 9, 2016

jbergstroem commented Dec 9, 2016

jbergstroem commented Dec 9, 2016

jbergstroem commented Dec 9, 2016

mikeal commented Dec 9, 2016

dragon788 commented Dec 9, 2016

jbergstroem commented Dec 9, 2016

jbergstroem commented Dec 9, 2016

mikeal commented Dec 9, 2016

jbergstroem commented Dec 9, 2016

terinjokes commented Dec 9, 2016 via email

soberstadt commented Dec 12, 2016

mikeal commented Dec 12, 2016

jbergstroem commented Dec 13, 2016

jbergstroem commented Dec 13, 2016

PeterDaveHello commented Apr 9, 2017 • edited Loading

jbergstroem commented Apr 9, 2017

mohamedalighouma commented Aug 9, 2018

SimenB commented Dec 9, 2016 •

edited

Loading

jbergstroem commented Dec 9, 2016 •

edited

Loading

matthewp commented Dec 9, 2016 •

edited

Loading

PeterDaveHello commented Dec 9, 2016 •

edited

Loading

PeterDaveHello commented Apr 9, 2017 •

edited

Loading