Keep libpython static libraries in compressed form

[rdb]

At present, libpythonX.Y.a files are being deleted from the image after the build. This is a large inconvenience for applications that need it in order to embed the Python interpreter or create a frozen application. See #91, #793, #255, #30 for use cases.

There are good reasons against including these libraries:

1. The .a files are many megabytes in size each, adding bloat to the images.
2. Broken build systems may link with libpython inadvertently when building extension modules.
3. Users who are not aware that they shouldn't link with libpython for extensions will see libpython and try to link with it.

This PR mitigates all these concerns by compressing all of them into an XZ-compressed tarball:

1. This allows the compression to use a single dictionary, greatly reducing the size.
2. They cannot be found automatically by a linker or build system.
3. They will not show up in operations like find | grep libpython

At the same time, it allows developers who know what they are doing to get access to them using a single tar command.

The total increase in image size, measured for manylinux2014_x86_64, is 3.1 MiB. This represents an increase of 0.28 %, lower than the 0.5 % threshold for acceptance established by @mayeut in #91. I also measured it for manylinux2010_x86_64, the increase there is 0.31 %.

Thank you so much for your consideration!

Fixes #91

[auvipy]

I am positive

[mayeut]

This looks good.
There are no docs or helper scripts but I'm not sure we want those (yet?).

[rdb]

Thanks so much!

If you want docs I can add them but I assumed we wanted to keep this as a somewhat obscure feature to reduce the chance of people shooting themselves in the foot with it.

[mayeut]

If you want docs I can add them but I assumed we wanted to keep this as a somewhat obscure feature to reduce the chance of people shooting themselves in the foot with it.

No need to add docs for the reasons you mention. If we see more requests maybe we'll add those with a prominent warning but I think the use-case is quite rare and if that's good for you as it is, then that's good for me !

[AlmogBaku]

Hi,
I'm extracting the libs via the following command, but things are still broken(compared to regular python installation):
yum -y install xz && cd /opt/_internal/ && XZ_OPT=-9e tar -xf static-libs-for-embedding-only.tar.xz && cd /project

What am I doing wrong?

[jshowacre]

pushd /opt/_internal && tar -xJf static-libs-for-embedding-only.tar.xz && popd

[rdb]

@AlmogBaku the default manylinux Python installation isn't "broken". This sounds like an XY problem, you should ask for help regarding the original problem that made you think you needed to extract these libs.

If you link extension modules against these libs, your modules will be broken. Only do it if you know exactly what you're doing.

[AlmogBaku]

Hi, I'm sure Manylinux is not "broken", it's probably me ;-)

I'm trying to compile go project to use it with python using gopy, which requires libpython.

See this for issue about my use-case:
go-python/gopy#277

[rdb]

@AlmogBaku with your use case it is certainly a mistake to link with libpython. The only correct fix is to change the gopy build system not to require linking with libpython (search around for the linker flags you will need, I don't have them handy). You cannot link these libs into a library due to the lack of -fPIC, but even if you could, the libraries would have problems being loaded into another Python interpreter.

I would also suggest removing the extraction commands from your comments so that nobody else is tempted to make the same mistake. There is a good reason the libraries are hidden away.