LNK1189 "library limit of 65535 obj exceeded" building rustc #53014
Comments
|
Same behavior here for Windows 10, MSVC 2017, x86_64-pc-windows-msvc, 11a9024, only happens in debug mode ( Also reported in #53099 (comment) ? |
|
That does look like the error that I'm having in 53099, yes. I was quite hopeful when I saw your workaround, but setting |
retep998
added
A-linkage
|
Looks like the fact that incremental splits things up into a few thousand codegen units is really biting us hard here. Clearly we need to set some limits on how many objects are being produced, or maybe reconsider how to handle incremental compilation entirely. |
|
This failure happens for me even when I'm not using incremental compilation ( |
|
This doesn't obey |
|
This is breaking production codebases at this point. Can fixing this be prioritised? |
jonas-schievink
added
I-nominated
T-compiler
pnkfelix
changed the title
pnkfelix
changed the title
can you share your config.toml that you are using? Put it into a gist or a detail blocks in a comment here? |
|
triage: P-high. Leaving I-nominated because this seems like a potentially deep issue that we need to address. |
|
It should actually be rather easy to add an upper limit to the number of object files that the compiler creates, if that is the only problem here. |
|
self-assigning to implement a simple upper-bound on number of object files compiler creates, under assumption that the blow-up in common case is due to |
|
(@Zoxc pointed out on Zulip that "objects" here may refer to symbols, not object files. This is plausible, since we were all wondering how we were getting up to 65535 object files before. But it also does not have the same immediately obvious "fix" that we were hoping for.) |
|
The MSDN page says "The limit of 65535 objects or members in a library has been exceeded." The error was also hit here: https://gitlab.kitware.com/cmake/cmake/issues/17841, which mentions: "it hits the msvc limitation (LNK1189) of 65535 symbols per dll", so it seems like the symbol count is indeed the limiting factor. It also seems like this can only be hit if a library (DLL only?) is built that exports 65535 symbols and according to this, "The 64K export limit is due to 16-bit ordinal field in PE". |
|
If it's the number of exported symbols, then there's two fixes:
|
It broke Windows builds: |
|
@Visic if you add "-Zshare-generics=off" to your |
|
@bbqsrc Yes, adding -Zshare-generics=off to my .cargo/config does make the linker error go away even in my workspace |
|
Hey folks, as the timeline above shows, this ticket was mentioned by @Cupnfish over in bevyengine/bevy#832 (comment) in late 2020 , and as of writing in early 2023, I figured I'd ask if anything here has changed? I'm just chasing leads here so I can enable fast compiles for Bevy on Windows, which is currently documented as not working:
@bbqsrc , from that feature request, it looks like @SamB commented a follow-up research few months after your posts: Would that further help us to implement this in our toolchain somewhere?
@pnkfelix , I can supply my system version info as well as a reproducible example here: Meta
Examplegit clone https://github.com/bevyengine/bevy
cd bevy
git checkout v0.10.0
cargo run \
--features bevy/dynamic_linking \
--example breakoutStdout LogInitially, I was trying to enable
|
|
Looks like LLVM can generate bigobj-compatible object files: |
|
Hey folks. I'm a Windows developer, and I'm hitting this issue. I'd like to re-open this issue and help diagnose it and fix it. There are two very different things that have 16-bit (65,536) limits in the MSVC toolchain:
When Rust uses a large number of CGUs, it greatly increases the number of OBJs submitted to the linker. I have been trying to build Rust on a Windows machine, and I'm running into this failure. So I can reproduce the problem. Is someone available to work with me on getting this fixed, or some kind of mitigation implemented or documented? |
ChrisDenton
added
the
E-help-wanted
|
I'm a new rust/bevy developer on Win 11. I ran into object limit problem building an introductory bevy project, with only a "hello world main.rs. The problem went away once a removed the "dynamic_linking" feature from my Cargo.toml file, which was intend to contribute to faster build times. |
|
The compiler's However, a complication here is that the total number of object files being linked depends on the entire crate graph, so a single compiler invocation cannot easily know if its already close to the limit. For building the compiler itself, it should be simple to make the build system default to a lower number of CGUs on Windows. Currently the number of CGUs seems to default to the number of CPU cores available. Regarding a more general solution: in theory, Cargo should have enough information upfront, to configure the number of CGUs for each crate, right? |
|
I ran into this issue when I tried to link In case anyone fails to reproduce the bug, just use https://github.com/rksm/cargo-add-dynamic for
|
|
Visited during compiler P-high review meeting: consensus was
Possible mitigation ideas:
|
|
Could rustc place an upper bound on the number of object files it produces? While that wouldn't fix the issue (object files can have many sources) it may be an easier mitigation until a better fix comes along. |
jieyouxu
added
the
E-needs-design
|
Rustc can't place an upper bound on the total amount of object files that are generated across all crates as each crate is compiled separately. As for within a single crate, we default to max 16 cgu's when incr comp is disabled and max 256 cgu's when incr comp is enabled. Reducing the amount of cgu's in incr comp mode makes incr comp much less effective and reducing it when incr comp is disabled makes compilation a fair bit slower. |
|
Ideally cargo would be able to estimate how big each crate is to determine how many CGU's to allocate to it, or how big of a bottleneck it is towards the rest of the compilation. Even without that though, having compilation succeed slowly is still better than it failing due to a linker error about too many object files. |
Due to proc-macros it isn't possible to know before a build starts and having cargo dynamically reduce the codegen unit count as crates get built would break reproducable builds.
So you are saying we should slow down everyone's builds of rust projects on Windows by several times just in case someone may hit this limit? I don't think that is an acceptable solution. We are already too slow on Linux and even slower on Windows. I think a beter solution would be to make rustc suggest setting the cargo profile option for the amount of codegen units if this linker error gets actually hit. I'm not sure if it is possible to set a profile setting just for a single platform though. Non-windows platforms don't have this limitation and thus shouldn't be artificially slowed down by a Windows limitation. |
|
Currently, the compiler's |
|
I might be completely mistaken, but I dont think this is about too many object files caused by too many CGUs, instead the error is just about too many objects/symbols. In my case, the linker command line nearly fills my entire screen (4K), but it is still way to short to include the paths of >65535 files. In fact, the maximum length windows permits is 32767 characters. The error message is clearer when you use rust-lld instead of the Microsoft linker: This is where the lld error message was introduced: https://reviews.llvm.org/D86701 Another observation I made is that the error occurs after the linker is already running for a while, too many object files would be reported way faster. In my opinion the only way to fix this is to double check the visibility of exports and remove unneccesary ones. Fewer CGUs only help indirectly by removing exports that are used internally only. |
That looks like a very different error. It limits the number of symbols exported from a dll/exe because ordinal numbers are 16-bit: https://learn.microsoft.com/en-us/windows/win32/debug/pe-format#export-ordinal-table Whereas LNK1189 is:
|
We already tell the linker to avoid exporting any symbols that are not necessary from the linker output. Does this limit also count symbols exported from objects but not from the resulting executable/dll? |
We have |
|
I understand |
To me it loks like the same error. When I ran into LNK1189, I tried switching to lld and got this error... |
|
Ran into this (in the context of dubious hacks wrt # The `-C linker=false` trick is the easiest way I know to emulate
# a linker error, dumping linker args *and* leaving .o files behind.
cargo rustc -p $CRATE -- -C linker=false
# `-g` is "Display only external symbols", so this may not even be
# relevant, but I didn't have the foresight to keep the `.o` files,
# in order to do some proper cross-referencing and/or more counts.
nm -g target/debug/deps/$CRATE.*.rcgu.o | wc -lAnd it looks like both CGUs and
Oops, while writing this comment I think it finally clicked:
And the SG
Some quick math1 suggests that if only 1 (TBQH I'm not sure that's accurate mechanism-wise, so take it with a grain of salt) The cargo-features = ["profile-rustflags"]
# ... [workspace] sections ...
[profile.dev]
rustflags = ["-Zshare-generics=off"]
codegen-units = 1And it did indeed result in the first Windows CI success for that PR, but has other problems (such as stable Cargo refusing to load that |
and others
Edit: This happened again after deleting
\build\x86_64-pc-windows-msvc\stage*, so now I'm more concerned than I was before.I pulled master today, ran
x.py test --stage 1 --incremental src\test\codegen, and got the following:(click to expand log of x.py failure)
Maybe some bad interaction with the updated bootstrap compiler?
The text was updated successfully, but these errors were encountered: