linux/make-bootstrap-tools: Add support for a real XZ and HTTPS in curl #8081

wkennington · 2015-05-31T02:30:24Z

This change makes bootstrap-tools.tar.xz 500k larger which should be inconsequential given that bootstrap-tools is already 26MB. It also reduces the overall size of the bootstrap-tools tarball by dynamically determining what libraries it needs to include at build time. Note this is the first of 2 PR's

wkennington · 2015-05-31T02:37:40Z

This is the first set of patches in two PR's, the steps for migration are:

Merge this pr into master
Merge master into staging
Build a bootstrap-tools tarball in staging
Replace the bootstrap-tools tarballs in staging and then apply the second patchset when building stdenv.
Merge staging back into master once everything builds.

wkennington · 2015-05-31T03:10:09Z

cc @edolstra @vcunat

vcunat · 2015-05-31T07:35:32Z

pkgs/stdenv/darwin/make-bootstrap-tools.nix

      cp -d ${openssl}/lib/*.dylib $out/lib

+      # Add ca-certs for curl verification


I thought bootstrap only uses fetchurl, which doesn't use certificates AFAIK (--insecure).

Look at the next PR.

On Sun, May 31, 2015, 00:35 Vladimír Čunát [email protected] wrote:

In pkgs/stdenv/darwin/make-bootstrap-tools.nix
#8081 (comment):

cp -d ${openssl}/lib/*.dylib $out/lib

# Add ca-certs for curl verification

I thought bootstrap only uses fetchurl, which doesn't use certificates
AFAIK (--insecure).

—
Reply to this email directly or view it on GitHub
https://github.com/NixOS/nixpkgs/pull/8081/files#r31387744.

vcunat · 2015-05-31T07:45:10Z

In general, I believe we should replace (most) usages of bz2 by xz. It's supposed to be significantly superior in both compression ratio and resources needed for decompression. We might even drop bzip2 from the bootstrap tarball, but there would be little gain (~150 kB).

wkennington · 2015-05-31T07:46:21Z

Agreed but we have to change all of those sources first. I think it's worth
just leaving it in.

On Sun, May 31, 2015, 00:45 Vladimír Čunát [email protected] wrote:

In general, I believe we should replace (most) usages of bz2 by xz. It's
supposed to be significantly superior in both compression ratio and
resources needed for decompression. We might even drop bzip2 from the
bootstrap tarball, but there would be little gain (~150 kB).

—
Reply to this email directly or view it on GitHub
#8081 (comment).

wkennington · 2015-06-01T01:55:51Z

This will now need to be merged into staging as the fixes for grep require a mass rebuild.

dezgeg · 2015-06-03T09:12:38Z

I built this and the 2nd PR on my ARMv7 board. Here's one more patch that's needed for the cross tools to build correctly: dezgeg@6d0f474.

edolstra · 2015-07-07T13:32:45Z

This PR is unreviewable like this, since it contains 158 commits, most of which seem unrelated to the bootstrap tools...

wkennington · 2015-07-07T15:58:30Z

It was a bad rebase after I Cherry picked in some stuff. I'll fix this soon.

On Tue, Jul 7, 2015, 06:33 Eelco Dolstra [email protected] wrote:

This PR is unreviewable like this, since it contains 158 commits, most of
which seem unrelated to the bootstrap tools...

—
Reply to this email directly or view it on GitHub
#8081 (comment).

…ols inside of bootstrap tools This makes it easier to add tools in the future and helps to minimize the amount of included libraries to the minimum without breaking executables.

… overhead

This will open up options for sourcing tarballs from repositories like github as they only serve tarballs over ssl. It will also allow us to eventually verify all tls connections made by fetchurl.

…archs

…raries

domenkozar · 2015-09-09T18:21:03Z

This won't make it into 15.09

bjornfor · 2015-10-20T18:59:32Z

Any news here? (I'm looking for HTTPS in fetchurl for bootstrapping perl-cross.)

vcunat · 2015-10-20T19:15:02Z

IMO, using/supporting HTTPS down to bootstrapping level is rather superfluous. All fetches have checked hashes, so the only use case is when you update by invalidating hash and using the one from the error message, but we would better improve that workflow instead....

wkennington · 2015-10-20T19:19:08Z

No, checking certs may be superfluous but it is becoming the trend to only
host tarballs over https. Just look at any project hosted on github as an
example. If they don't do any third party hosting it will only be available
over https. This was a problem for supporting http2 in curl as the project
needed was only hosted on GitHub but the bootstrap couldn't pull the
binaries directly.
On Tue, Oct 20, 2015 at 12:15 PM Vladimír Čunát [email protected]
wrote:

IMO, using/supporting HTTPS down to bootstrapping level is rather
superfluous. All fetches have checked hashes, so the only use case is when
you update by invalidating hash and using the one from the error message,
but we would better improve that workflow instead....

—
Reply to this email directly or view it on GitHub
#8081 (comment).

bjornfor · 2015-10-20T19:22:53Z

Yes, exactly. perl-cross tarballs are only available over HTTPS on github. (My long-term plan is to build a small cross-system with nix, boot it in Qemu and native compile from there.)

vcunat · 2015-10-20T20:23:24Z

Oh, I see. I know some people want to completely abolish unencrypted transfers.

wkennington mentioned this pull request May 31, 2015

linux/stdenv: Fixups to accomodate boostrap-tools changes and ssl in fetchurl #8082

Closed

vcunat reviewed May 31, 2015
View reviewed changes

wkennington force-pushed the master.bootstrap1 branch 2 times, most recently from 8f1c2f4 to 127184d Compare May 31, 2015 21:37

vcunat added the 1.severity: mass-rebuild This PR causes a large number of packages to rebuild label Jun 2, 2015

vcunat modified the milestones: 15.10, 15.06 Jun 11, 2015

dezgeg mentioned this pull request Jun 20, 2015

gnugrep: fix cross-compilation (buildInputs -> nativeBuildInputs) #8433

Closed

wkennington force-pushed the master.bootstrap1 branch from 1893551 to 92810ee Compare June 20, 2015 21:43

vcunat mentioned this pull request Jun 22, 2015

ppl: build error #8455

Closed

wkennington force-pushed the master branch from bfbcef6 to 369febb Compare July 15, 2015 09:01

wkennington force-pushed the master.bootstrap1 branch from 92810ee to 25b2cc5 Compare July 16, 2015 21:24

wkennington added 5 commits July 16, 2015 14:35

linux/make-bootstrap-tools: Only copy needed runtime libraries for to…

491b684

…ols inside of bootstrap tools This makes it easier to add tools in the future and helps to minimize the amount of included libraries to the minimum without breaking executables.

linux/make-boostrap-tools: Add a real version of XZ as it has minimal…

b9cceff

… overhead

linux/make-bootstrap-tools: Add support for ssl to curl

d320ecd

This will open up options for sourcing tarballs from repositories like github as they only serve tarballs over ssl. It will also allow us to eventually verify all tls connections made by fetchurl.

linux/make-bootstrap-tools-cross: Port the patches

e46b130

busyboxBootstrap: Build on hydra

54e74bc

wkennington added 5 commits July 16, 2015 14:35

coreutils: Just use the default as it doesn't much space

0df632e

busyboxBootstrap: Use glibc instead of musl as it is broken for many …

495fbc0

…archs

make-bootstrap-tools-cross: Fix the method for determining shared lib…

477c83b

…raries

make-bootstrap-tools-cross: Fix the method for determining shared lib…

2e3773a

…raries

make-bootstrap-tools-cross: Add strangely missing rpath for libmpc

b32f310

wkennington force-pushed the master.bootstrap1 branch from 25b2cc5 to b32f310 Compare July 16, 2015 21:35

domenkozar modified the milestones: 16.03, 15.09 Sep 9, 2015

Mathnerd314 mentioned this pull request Nov 7, 2015

Standardize all-packages.nix arguments and stdenv attributes #10874

Closed

mcmtroffaes mentioned this pull request Nov 27, 2015

bootstrap: enable https for bootstrap curl using wolfssl #11287

Closed

wkennington closed this Dec 17, 2015

wkennington deleted the master.bootstrap1 branch December 17, 2015 19:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

linux/make-bootstrap-tools: Add support for a real XZ and HTTPS in curl #8081

linux/make-bootstrap-tools: Add support for a real XZ and HTTPS in curl #8081

wkennington commented May 31, 2015

wkennington commented May 31, 2015

wkennington commented May 31, 2015

vcunat May 31, 2015

wkennington May 31, 2015

vcunat commented May 31, 2015

wkennington commented May 31, 2015

wkennington commented Jun 1, 2015

dezgeg commented Jun 3, 2015

edolstra commented Jul 7, 2015

wkennington commented Jul 7, 2015

domenkozar commented Sep 9, 2015

bjornfor commented Oct 20, 2015

vcunat commented Oct 20, 2015

wkennington commented Oct 20, 2015

bjornfor commented Oct 20, 2015

vcunat commented Oct 20, 2015

		cp -d ${openssl}/lib/*.dylib $out/lib

		# Add ca-certs for curl verification

linux/make-bootstrap-tools: Add support for a real XZ and HTTPS in curl #8081

linux/make-bootstrap-tools: Add support for a real XZ and HTTPS in curl #8081

Conversation

wkennington commented May 31, 2015

wkennington commented May 31, 2015

wkennington commented May 31, 2015

vcunat May 31, 2015

Choose a reason for hiding this comment

wkennington May 31, 2015

Choose a reason for hiding this comment

vcunat commented May 31, 2015

wkennington commented May 31, 2015

wkennington commented Jun 1, 2015

dezgeg commented Jun 3, 2015

edolstra commented Jul 7, 2015

wkennington commented Jul 7, 2015

domenkozar commented Sep 9, 2015

bjornfor commented Oct 20, 2015

vcunat commented Oct 20, 2015

wkennington commented Oct 20, 2015

bjornfor commented Oct 20, 2015

vcunat commented Oct 20, 2015