RPATH with Python and virtualenv #2076
Comments
|
@rjeschmi What kind of specific errors are you running into with I think the only way around this is to avoid using Interestingly, Spack is actually considering switching to |
|
@pescobar You should be aware of this... ( |
|
@rjeschmi thanks for pointing this. This is a blocker for me because my users do heavy usage of virtualenvs. I am thinking if I should totally stop using rpath or just use the rpath skip-out option to build python. I guess a similar issue can happen if users install R packages to their home folder and R has been linked with rpath, isn't it? |
|
@pescobar after talking online with you I don't think it changes the behaviour for you vs regular easybuild right now. I don't think it will effect any other packages either. It is just a relocation peculiarity and if you have LD_LIBRARY_PATH set, it will still work fine (just RPATH isn't doing the work) |
|
@rjeschmi thanks for the explanation in the irc channel ;) As you mention, this issue would not affect my specific use case because I still define |
|
@rjeschmi Do you see any reason to just stop using Especially w.r.t. Spack considering to move to |
|
Relevant to this discussion is @ocaisa's comment in easybuilders/easybuild#282 (comment) |
|
The only reason to use $ORIGIN is if you are going to support relocation. I don't mind it in theory, but it should be a further experimental option. |
|
Isn't the installation step itself usually a relocation? What if the application requires loading of it's own .so libraries (like |
|
So, that would mean building Qt with |
|
No it wouldn't most likely...but that's assuming that your choice of location is actually the correct one |
|
@tgamblin Any thoughts on this on the benefits over RPATH'ing |
|
To me, one for the other is a straight swap so I say go with hard-coding as the default but allow people to switch to |
|
I'd just use prefix here. If you look at the RPATH on the Python binary, it has tons of absolute paths to dependencies anyway, so $ORIGIN isn't buying you relocatability. You'd have to change all those other non-ORIGIN RPATHs too if you wanted to move your EB installation. I am not sure we'll move to relative RPATHs soon because it's hard to get right. You have to know a lot about how your build is laid out -- e.g., how many levels deep in the prefix is the library? It looks like you assume one-level deep libs here ( Because it's so hard to do this in a general way, Spack probably won't move to relative RPATHs any time soon. I am thinking we could allow relocation more easily with patchelf than with $ORIGIN. See spack/spack#445. |
|
Thanks a lot for the feedback @tgamblin! 👍 |
|
Yeah you have the advantage too that $ORIGIN/../libs will probably fit in the same space as $prefix (but less likely the other way) so patchelf should be a lot easier that direction. Also the patchelf should be relatively easy. Definitely worth thinking about long_fake_prefix that is easy to search for as an option with a patchelf finish step (and this is all easier if the prefix is just there in the RPATH). |
|
I believe parchelf works fine with variable-length/longer RPATHs. It's older cousin chrpath doesn't. |
|
I agree with @tgamblin that using the absolute paths is better for libs that are installed in deeper hierarchies (I'm thinking especially shared objects for python packages here). However, I'd suggest to include $ORIGIN itself as well. E.g. ParaView installs libraries in lib/paraview-$version, which wouldn't be found otherwise. |
|
I had success making this work by symlinking to lib/lib64 directories some time back, but this was done with a tool that wrapped virtualenv, not in virtualenv itself... |
|
Drive-by comment: I think |
One use case I'd like to see work is to have a version of Python that works properly with virtualenv without
after compiling Python with --rpath I can see that RPATH is set with readelf:
[rob@ottbioinfo venv]$ ./py-eb/bin/python --version
Python 2.7.5
[rob@ottbioinfo venv]$ ldd py-eb/bin/python
linux-vdso.so.1 => (0x00007ffc017b3000)
libpython2.7.so.1.0 => /lib64/libpython2.7.so.1.0 (0x00007efdd4b32000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007efdd492d000)
libm.so.6 => /lib64/libm.so.6 (0x00007efdd462b000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007efdd440f000)
libutil.so.1 => /lib64/libutil.so.1 (0x00007efdd420b000)
libc.so.6 => /lib64/libc.so.6 (0x00007efdd3e49000)
/lib64/ld-linux-x86-64.so.2 (0x00007efdd4f02000)
The text was updated successfully, but these errors were encountered: