Have Python wheel

[brenthuisman]

Having a Python wheel would speed up installation and remove the requirement for users to have a compilation pipeline set up.

Currently, if you pip install arbor, you will see output like:

Building wheels for collected packages: arbor
  Building wheel for arbor (setup.py) ... done
  Created wheel for arbor: filename=arbor-0.5-cp38-cp38-linux_x86_64.whl size=1322418 sha256=83fc9fe7d03f7afc73f69210bb9ea4373b08ee5d4c66180b9cfbb52c1aad35ac
  Stored in directory: /p/home/jusers/huisman1/jusuf/.cache/pip/wheels/09/18/68/ceac2887642e9c2e49a20630e3e5a94572bec0a994058e82dc
Successfully built arbor

If you take that wheel, it's perfectly installable in other envs/machines right now (just tested), as long as you match the platform. Turning that into manylinux (most recent version is 2014, see PEP 599) seems like it shouldn't be so difficult (famous last words, I know).

[brenthuisman]

• Official Python docs: https://packaging.python.org/
  • Only mentions manylinux once, refers to PyPA
• pip/PyPA, wheel
  • No documentation whatsoever on manylinux, other than to use the https://github.com/pypa/manylinux docker image.
• https://github.com/pypa/manylinux; https://pip.pypa.io/en/latest/user_guide
  • Image comes with Python 2.7-3.9 preinstalled, yet no info on manywheel images. Maybe it refers solely to the ABI of the compiler?
  • Pypa docker image does not come with libpython.so: libpython*.so is missing pypa/manylinux#255
• Integrate wheel building in CI: https://github.com/joerick/cibuildwheel / https://cibuildwheel.readthedocs.io
  • Again no docs on manylinux. Uses PyPA image, so, can you guess what the problem is?
  • pip cibuildwheel instead of the GA still uses the pypa docker image, and because of the missing libpython fails.
  • brian2 uses it, but seemingly only published source packages on PyPI.
• https://scikit-build.readthedocs.io
  • unclear how to build manywheels, documentation is blank apart from referring to https://github.com/dockcross/dockcross
  • dockcross seems to be based on the Pypa manylinux images, and may therefore be missing libpython. Confirmed.
  • https://github.com/scikit-build/scikit-build-sample-projects/tree/master/projects/hello-pybind11
  • THis won't help us build manylinux wheels. It's just a setuptools alternative.
• Push to PyPI (test)
  • TODO: check create release workflow: https://github.com/YannickJadoul/Parselmouth/blob/master/.github/workflows/wheels.yml

[brenthuisman]

Manylinux wheels are not supposed to link against libpython, hence the problem building Arbor on the various manylinux images. Running auditwheel on a locally built Arbor bdist_wheel confirms:

auditwheel: error: cannot repair "dist/arbor-0.5.1.dev0-cp39-cp39-linux_x86_64.whl" to "manylinux1_x86_64" ABI because of the presence of too-recent versioned symbols. You'll need to compile the wheel on an older toolchain.

Of course, it wasn't built on any of the manylinux images. So, I understand why both my local wheels can't work and why compiling Arbor in the manylinux image doesn't either. I should solve the latter to actually have the wheels we want.

Similar struggles:

• How to create platform portable wheels (manylinux) ? neuronsimulator/nrn#329
• Linking to libpython.so (or not) pypa/manylinux#69
• https://gitlab.kitware.com/vtk/vtk/-/merge_requests/1511
• Missing libpython with Debian default Python install pypa/manylinux#30

Pybind hint: use Development.Module and not Development when configging CMake. Confirmed: this makes Arbor compile in the PyPA manylinux image. auditwheel can make manylinux wheels succesfully. Creating a GA workflow that uploads to testpypi.

[brenthuisman]

Binary wheel production is now working for Linux and Macos. Todo/to decide:

• Make sdist for other envs (python setup.py sdist)
• Get token for arbor project on TestPyPI
  • Generate version numbers?
• Run tests on the wheels?
• Upload version tags to PyPI automatically?