GH-45576: [Python] Support building C++ library automatically

[mgorny]

What changes are included in this PR?

Add a PYARROW_BUILD_ARROW_CPP environment variable and a corresponding --build-arrow-cpp option that enables automatically building the Arrow C++ libraries as part of setup.py invocation and using them to build PyArrow afterwards. This is a prototype / proof-of-concept.

Are these changes tested?

This change introduces an additional build path. I've tested it locally.

Are there any user-facing changes?

It introduces a new option, primarily aimed at downstream packagers — though I suppose some users building from source may also want to use it. If the approach is sound, updating https://arrow.apache.org/docs/developers/python.html#build-and-test may make sense.

• GitHub Issue: [Python] Support building libarrow (C++) automatically as part of the Python package #45576

[github-actions]

⚠️ GitHub issue #45576 has been automatically assigned in GitHub to PR creator.

[tiran]

I have tested @mgorny 's PR in our downstream build pipeline. It was as intended and makes our downstream builds much easier to maintain.

[pitrou]

I'm frankly not sure we want to support anything like this. Our setup.py is already tedious to maintain and we probably don't want to add any complexity there. @raulcd @jorisvandenbossche Thoughts?

[raulcd]

I think the current plan, or at least the latest conversations, were to move our build-backend to scikit-build-core. Adding more custom options to our setup.py seems that will make that migration even more complex.
I understand the appeal but this will still fail for people building from source from pypi for example and only helps downstream package maintainers that want to build pyarrow from source cloning the repo, right? which at that point requires a couple commands to build Arrow C++:

cmake -S arrow/cpp -B arrow/cpp/build \
      -DCMAKE_INSTALL_PREFIX=$ARROW_HOME \
      --preset ninja-release-python
cmake --build arrow/cpp/build --target install

@tiran when you say your downstream builds are much easier to maintain, could you share a link on how are you currently building Arrow C++? Is that you would like to get rid of the manual cmake commands?

[pitrou]

I think the current plan, or at least the latest conversations, were to move our build-backend to scikit-build-core.

Yes, IMHO that's very much where we want to go.

[mgorny]

I understand the appeal but this will still fail for people building from source from pypi for example

Well, that's definitely something that could be fixed as a followup, or at least to some degree.

were to move our build-backend to scikit-build-core

Well, I'm all for that. I suppose a similar solution (i.e. ability to build Arrow C++ libraries from the Python build) could also be implemented at CMake level, and actually that's how I initially wanted to do it — but I've given up since it required a lot of added complexity. I've noted the problems in #45576. I'm not saying it's impossible, just would probably involve more changes to CMake files and more work, and I'm not sure if that is desired.