Add io.js v1.x

[yosuke-furukawa]

I would like to compare with io.js v1.x and io.js v2.x.

[thlorenz]

Nice, however this seems a slightly different concern.
I wanted to help people choose between staying with node 0.10 or upgrading to either 0.12 or latest io.js.

However upgrading to io.js 1.x is not really what people should worry about.
Should we fork this to have benchmarks just focusing on io.js versions to track improvements/detoriations between releases?

We could fork into the iojs organization if you want and maintain that version there.
Ideally we'd run something like this on every build, and generate graphs as well.

[yosuke-furukawa]

@thlorenz

Should we fork this to have benchmarks just focusing on io.js versions to track improvements/detoriations between releases?

Yes!! I have a motivation to benchmark continuously on every release and it should be discussed on nodejs/benchmark WG.

And io.js v1.x is more stable than v2.x. some npm modules does not work on latest.
So we would be better to introduce io.js v1.x as an option to upgrade.

How do you feel about that ??

[thlorenz]

Ok, I'll try to find some time tonight to run these for 1.x so they'll be included.
They take a long time and that's the main hurdle :)

If I can't run the Linux ones at least will have to wait for 1.5 months as I'm gonna be travelling, leaving my Linux desktop at home ;)

On that note we need to set up an instance that has consistent performance to run these on continuously for different releases.

[thlorenz]

Updated the data to include 1.8.1 of iojs (manually changed the script with what you added).
Currently not happy with it overall consistency of results -- rerunning on Linux since I had some daemons running during the benchmarks.

Getting this right seems really hard.

• A) you run benchmarks for ALL versions back to back (and even then the machine conditions may change in between versions - i.e. CPU gets hot and slows down a bit)
• B) you have to make sure the conditions are exactly the same when you run another version as it was when you got data for the existing versions

We should probably move this discussion over into the Benchmarking WG and figure out a way how to get consistent results.
We could try to summon Brendan Gregg in there as I'm sure he's got lots of insight to share.

Closing this PR, but please add a link to where the discussion was moved once we do.
Thanks.

[thlorenz]

Updated Linux data and it seems more realistic now.

[brendanashworth]

Regarding an io.js WG, @rvagg has mentioned one before and I'm sure many developers (including me) would be interested in that sort of thing.