Faster sha256 computation

[dgageot]

The computation of the sh256 digest is now done concurrently to the tarball creation.
This gives a 30% boost on my tests (with a hot cache)

Signed-off-by: David Gageot [email protected]

[ekcasey]

@dgageot How are you you testing this when you see a 30% performance. When I did a quick and dirty benchmark the times were slightly faster without the ConcurrentHasher.

[dgageot]

On a Mac, I created a 1.2Gb tarball from my go-build folder. I created the archive multiple times and ignored that first result. I'll try to run a better benchmark.

[dgageot]

I ran the tests again, here's what I got:

I'm taring a copy of my go env GOCACHE folder. The resulting tar file is 1.2Gb. I ran the tests on master and on this branch, on my mac. I ran the tests both in cold cache and hot cache conditions. I used purge to approximate cold cache conditions.

master branch

Run	Cold cache	Hot cache
1	16.5	5.8
2	16.2	5.7
3	16	5.8

with this PR

Run	Cold cache	Hot cache
1	12.9	3.4
2	13.2	3.1
3	13.2	3.3

[ekcasey]

@dgageot I was able to reproduce your results and performance gains. There is one tradeoff worth noting. While this implementation ran faster in real time on my workstation the actual CPU time required by this implementation is higher. Therefore, in CPU bound environments, this change might actually degrade performance.

That being said I think this is probably the right tradeoff.

[ekcasey]

I think we should add a minimal unit test ensuring that ConcurrentHasher.Sum returns identical to the underlying Hash.Sum. It looks correct right now, but a test will prevent someone from accidentally breaking it in the future.

[ekcasey]

Would you mind moving the hasher implementation into a new file (e.g. hash.go)? I think that small organizational change would make both the hashing and the tarring logic more readable.

[dgageot]

@ekcasey This should be better now. Thanks for your time!