Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"dyld: Library not loaded: /lib/libpython3.13.dylib" in 3.13.0a4 MacOS non-framework build with custom prefix #115737

Closed
native-api opened this issue Feb 20, 2024 · 11 comments
Labels
build The build process and cross-build OS-mac type-bug An unexpected behavior, bug, or error

Comments

@native-api
Copy link
Contributor

native-api commented Feb 20, 2024

Bug report

Bug description:

The above error was discovered in Pyenv CI check (logs are on the link).

See pyenv/pyenv#2903 (comment) for diagnostics and probable cause.

CPython versions tested on:

3.13

Operating systems tested on:

macOS

Linked PRs

@native-api native-api added the type-bug An unexpected behavior, bug, or error label Feb 20, 2024
@Eclips4 Eclips4 added OS-mac build The build process and cross-build labels Feb 20, 2024
@ronaldoussoren
Copy link
Contributor

How can we reproduce this? In particular, what is the command-line for configure and any additional customization of the build process?

@ned-deily
Copy link
Member

From a quick look at the linked pyenv issue, it looks like this may be related to the changes for iOS frameworks builds (gh-114099). @freakboy3742?

@ronaldoussoren
Copy link
Contributor

I'd still like to know how to reproduce this without having to hunt through the pyenv GitHub. The linked issue seems to use -rpath somehow, which is not something we explicitly support.

@freakboy3742
Copy link
Contributor

@ned-deily As with @ronaldoussoren, I'm trying to work out what the reproduction instructions are... I'll dig in and see if I can reverse engineer the problem.

@native-api
Copy link
Contributor Author

native-api commented Feb 21, 2024

Attached a Bash trace log from the Pyenv invocation which contains configure/make calls and ennvar assignments. That's what I usually use to diagnose Pyenv issues.
out.txt
If you need a more distilled view, I'll be able to compose it when I return from work in about 10 hours.

Setting rpaths is required in MacOS to find dylibs outside of the default search path -- in Homebrew and under the custom installation prefix.

@freakboy3742
Copy link
Contributor

@native-api Thanks for that detail - based on that, I've been able work out the reproduction case is:

./configure --prefix=$(pwd)/root --enable-shared --libdir=$(pwd)/root/lib && make -j4 all install && $(pwd)/root/bin/python3.13 -m test.pythoninfo

@freakboy3742
Copy link
Contributor

FWIW: it looks like the problem slipped in because while I did test non-framework builds, and non-framework builds in a non-default location, I didn't check a shared (i.e. ,--enable-shared) non-framework, non-default location build.

@ned-deily
Copy link
Member

shared (i.e. ,--enable-shared) non-framework, non-default location build

Sorry, I should have thought to suggest testing that one as well originally. I generally avoid --enable-shared builds on macOS as they often lead to problems like this but they are used.

@native-api
Copy link
Contributor Author

native-api commented Feb 21, 2024

We have enabled --enable-shared by default by user requests. The rationale was it is required for products that link against libpython and it's the setting in official CPython Docker images.

@ned-deily
Copy link
Member

ned-deily commented Feb 21, 2024

To be sure, there's nothing inherently bad about using --enable-shared on macOS, especially when running from an installed Python. You just need to be vigilant during development and testing when running from a Python build directory to make sure the interpreter is not using the old installed shared library rather than the newly built one.

I've commented on the PR on how we can simplify the proposed fix.

That said, I stumbled across another potential problem here with --enable-shared. In the reproduction case above, there is:

--prefix=$(pwd)/root --enable-shared --libdir=$(pwd)/root/lib

Now it turns out that specifying that value for libdir is a no-op since the default value of libdir is $prefix/lib so there's no need to use libdir in this case. However, if you were to use a different value for libdir, then the macOS install_name, i.e. the path to the shared library that is baked into the python executable by the linker, is hardwired to be $prefix/lib so attempting to execute python will immediately segfault. It would be a simple change here to fix the install_name for the enable-shared case to use libdir, which is where the shared library is installed, and, while that does prevent the segfault, the installed Python is still not very usable as the standard library extension modules are also installed into libdir but the default sys.path doesn't take that into account so imports of them fail. This isn't a new issue it seems and has probably been around for almost forever but, as far as I recall, no one has every complained about it so it is probably best to leave well enough alone and not attempt to change this behavior here.

@ned-deily
Copy link
Member

Thanks for the report and the followup info. The simplified fix is merged into main for release in 3.13.0a5.

woodruffw pushed a commit to woodruffw-forks/cpython that referenced this issue Mar 4, 2024
diegorusso pushed a commit to diegorusso/cpython that referenced this issue Apr 17, 2024
freakboy3742 added a commit to freakboy3742/cpython that referenced this issue Aug 5, 2024
freakboy3742 added a commit to freakboy3742/cpython that referenced this issue Aug 5, 2024
freakboy3742 added a commit to freakboy3742/cpython that referenced this issue Aug 5, 2024
freakboy3742 added a commit to freakboy3742/cpython that referenced this issue Aug 5, 2024
freakboy3742 added a commit to freakboy3742/cpython that referenced this issue Sep 6, 2024
freakboy3742 added a commit to freakboy3742/cpython that referenced this issue Sep 9, 2024
freakboy3742 added a commit to freakboy3742/cpython that referenced this issue Sep 9, 2024
freakboy3742 added a commit to freakboy3742/cpython that referenced this issue Sep 9, 2024
freakboy3742 added a commit to freakboy3742/cpython that referenced this issue Sep 9, 2024
freakboy3742 added a commit to freakboy3742/cpython that referenced this issue Oct 9, 2024
freakboy3742 added a commit to freakboy3742/cpython that referenced this issue Dec 13, 2024
freakboy3742 added a commit to freakboy3742/cpython that referenced this issue Dec 13, 2024
freakboy3742 added a commit to freakboy3742/cpython that referenced this issue Dec 13, 2024
freakboy3742 added a commit to freakboy3742/cpython that referenced this issue Dec 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
build The build process and cross-build OS-mac type-bug An unexpected behavior, bug, or error
Projects
None yet
Development

No branches or pull requests

5 participants