Intermittent "Permission denied" on Windows #22628

Manishearth · 2015-02-21T12:47:16Z

Example: http://buildbot.rust-lang.org/builders/auto-win-64-nopt-t/builds/2223

error: linking with `gcc` failed: exit code: 1
note: "gcc" '"-Wl,--enable-long-section-names"' '"-fno-use-linker-plugin"' '"-Wl,--nxcompat"' '"-static-libgcc"' '"-m64"' '"-L"' '"C:\\bot\\slave\\auto-win-64-nopt-t\\build\\obj\\x86_64-pc-windows-gnu\\stage2\\bin\\rustlib\\x86_64-pc-windows-gnu\\lib"' '"-o"' '"x86_64-pc-windows-gnu\\stage2\\test\\stdtest-x86_64-pc-windows-gnu.exe"' '"x86_64-pc-windows-gnu\\stage2\\test\\stdtest-x86_64-pc-windows-gnu.o"' '"-Wl,--gc-sections"' '"C:\\bot\\slave\\auto-win-64-nopt-t\\build\\obj\\x86_64-pc-windows-gnu\\stage2\\bin\\rustlib\\x86_64-pc-windows-gnu\\lib\\libtest-4e7c5e5c.rlib"' '"C:\\bot\\slave\\auto-win-64-nopt-t\\build\\obj\\x86_64-pc-windows-gnu\\stage2\\bin\\rustlib\\x86_64-pc-windows-gnu\\lib\\libgetopts-4e7c5e5c.rlib"' '"C:\\bot\\slave\\auto-win-64-nopt-t\\build\\obj\\x86_64-pc-windows-gnu\\stage2\\bin\\rustlib\\x86_64-pc-windows-gnu\\lib\\libterm-4e7c5e5c.rlib"' '"C:\\bot\\slave\\auto-win-64-nopt-t\\build\\obj\\x86_64-pc-windows-gnu\\stage2\\bin\\rustlib\\x86_64-pc-windows-gnu\\lib\\libserialize-4e7c5e5c.rlib"' '"C:\\bot\\slave\\auto-win-64-nopt-t\\build\\obj\\x86_64-pc-windows-gnu\\stage2\\bin\\rustlib\\x86_64-pc-windows-gnu\\lib\\liblog-4e7c5e5c.rlib"' '"C:\\bot\\slave\\auto-win-64-nopt-t\\build\\obj\\x86_64-pc-windows-gnu\\stage2\\bin\\rustlib\\x86_64-pc-windows-gnu\\lib\\libstd-4e7c5e5c.rlib"' '"C:\\bot\\slave\\auto-win-64-nopt-t\\build\\obj\\x86_64-pc-windows-gnu\\stage2\\bin\\rustlib\\x86_64-pc-windows-gnu\\lib\\libcollections-4e7c5e5c.rlib"' '"C:\\bot\\slave\\auto-win-64-nopt-t\\build\\obj\\x86_64-pc-windows-gnu\\stage2\\bin\\rustlib\\x86_64-pc-windows-gnu\\lib\\librand-4e7c5e5c.rlib"' '"C:\\bot\\slave\\auto-win-64-nopt-t\\build\\obj\\x86_64-pc-windows-gnu\\stage2\\bin\\rustlib\\x86_64-pc-windows-gnu\\lib\\liballoc-4e7c5e5c.rlib"' '"C:\\bot\\slave\\auto-win-64-nopt-t\\build\\obj\\x86_64-pc-windows-gnu\\stage2\\bin\\rustlib\\x86_64-pc-windows-gnu\\lib\\liblibc-4e7c5e5c.rlib"' '"C:\\bot\\slave\\auto-win-64-nopt-t\\build\\obj\\x86_64-pc-windows-gnu\\stage2\\bin\\rustlib\\x86_64-pc-windows-gnu\\lib\\libunicode-4e7c5e5c.rlib"' '"C:\\bot\\slave\\auto-win-64-nopt-t\\build\\obj\\x86_64-pc-windows-gnu\\stage2\\bin\\rustlib\\x86_64-pc-windows-gnu\\lib\\libcore-4e7c5e5c.rlib"' '"-L"' '"x86_64-pc-windows-gnu\\rt"' '"-L"' '"C:\\bot\\slave\\auto-win-64-nopt-t\\build\\obj\\x86_64-pc-windows-gnu\\llvm\\Release+Asserts\\lib"' '"-L"' '"C:\\bot\\slave\\auto-win-64-nopt-t\\build\\obj\\x86_64-pc-windows-gnu\\stage2\\bin\\rustlib\\x86_64-pc-windows-gnu\\lib"' '"-L"' '"C:\\bot\\slave\\auto-win-64-nopt-t\\build\\obj\\.rust\\bin\\x86_64-pc-windows-gnu"' '"-L"' '"C:\\bot\\slave\\auto-win-64-nopt-t\\build\\obj\\bin\\x86_64-pc-windows-gnu"' '"-Wl,--whole-archive"' '"-Wl,-Bstatic"' '"-Wl,--no-whole-archive"' '"-Wl,-Bdynamic"' '"-lws2_32"' '"-luserenv"' '"-lkernel32"' '"-lws2_32"' '"-luserenv"' '"-lcompiler-rt"'
note: C:/program files/mingw-w64/x86_64-4.9.1-win32-seh-rt_v3-rev1/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/4.9.1/../../../../x86_64-w64-mingw32/bin/ld.exe: cannot open output file x86_64-pc-windows-gnu\stage2\test\stdtest-x86_64-pc-windows-gnu.exe: Permission denied
collect2.exe: error: ld returned 1 exit status

It doesn't seem to happen in the same place at every time, but basically an exe (usually stdtest) can't be opened for whatever reason.

cc @retep998 @klutzy

The text was updated successfully, but these errors were encountered:

klutzy · 2015-02-21T13:58:52Z

I've thought it's basically this: for some reason stdtest hanged at previous test, then during the second build linker fails to open exe since it's still running.
(The comment only said run-pass tests but while investigating it I also found one some ***test hanged.)

klutzy · 2015-02-21T14:20:59Z

Ping @alexcrichton: I think you can access to buildbot directly. Could you dump thread/stack status of hanged process (when this occurs again) using Process Explorer? This article explains how to see list of threads and stack trace of each thread.

Manishearth · 2015-02-21T20:46:37Z

Some various failures this caused (most of these today!):

This "intermittent" is really becoming a problem, can we prioritize this?

Manishearth · 2015-02-21T21:46:31Z

Seems like 99b39cc should have fixed it, but d0de2b4 broke it.

We tried b6044c7 but it caused a panic

Edit: I'm talking about the wrong failure here.

brson · 2015-02-21T21:59:03Z

@Manishearth suspects d0de2b4 may caused the recent spike.

brson · 2015-02-21T23:41:40Z

I added a patch to buildbot that attempts to kill everything named stdtest before running make check, and changed the max builds per windows slave to 1 rust-lang-deprecated/rust-buildbot@9911005

brson · 2015-02-22T00:01:34Z

This is how mariadb creates buildbot jobs https://mariadb.com/kb/en/mariadb/buildbot-setup-buildbot-setup-for-windows/

alexcrichton · 2015-02-22T10:00:40Z

@klutzy is correct in the diagnosis. The fact that stdtest hangs is pretty new to me and I haven't had the chance to investigate it much.

Note that the fix for all the tests in @klutzy's linked comment was to join on all threads. I had a patch to use scoped but it should have instead been spawn + join. @Manishearth I see a few linked commits to this issue which add calls to sleep, but I believe it is better solved with a join.

Could you dump thread/stack status of hanged process (when this occurs again) using Process Explorer? This article explains how to see list of threads and stack trace of each thread.

I took a peek, and unfortunately all I could deduce was that a thread was blocked in accept. I was unable to get any indication of which accept it was blocked in (no symbols were available).

and changed the max builds per windows slave to 1 rust-lang-deprecated/rust-buildbot@9911005

@brson, why change the max builds? I'm under the impression that the error does not come from this but rather the fact that something just hangs for too long. Killing stdtest at the start sounds like a good idea though. Note that this was my impetus for opening #22448.

Manishearth · 2015-02-22T10:02:32Z

@alexcrichton We can't join() when the child has panicked. That's why I added a sleep()

Manishearth · 2015-02-22T10:05:26Z

@alexcrichton Max builds has to be one, we're indiscriminately killing all stdtest processes without knowing which build they came from. Peter is working on a job object based solution so that we don't have to do that.

alexcrichton · 2015-02-22T10:10:57Z

@alexcrichton We can't join() when the child has panicked. That's why I added a sleep()

It depends how you spawned the thread. If the thread was spawned with scoped then it will propagate the panic but if spawned with spawn you will catch the panic with a Result.

Max builds has to be one, we're indiscriminately killing all stdtest processes without knowing which build they came from

Hm ok.

retep998 · 2015-02-22T12:22:01Z

I have created a simple Rust wrapper application that uses job objects to ensure all existing processes are completely dead.
https://github.com/retep998/dojob-rs/blob/master/src/main.rs

Manishearth · 2015-02-23T17:08:31Z

http://buildbot.rust-lang.org/builders/auto-win-32-nopt-t/builds/3431

note: C:/program files (x86)/mingw-w64/i686-4.9.1-win32-dwarf-rt_v3-rev1/mingw32/bin/../lib/gcc/i686-w64-mingw32/4.9.1/../../../../i686-w64-mingw32/bin/ld.exe: cannot open output file i686-pc-windows-gnu\test\run-pass\unwind-resource.stage2-i686-pc-windows-gnu.exe: Permission denied

So the diagnosis by @klutzy / @alexcrichton is correct; this test sometimes hangs and stays alive for the next build. (Though stdtest might be a separate problem)

We need to integrate dojob somehow ... @brson ?

If these tests exit while a thread is panicking it often leads to situations like rust-lang#22628.

alexcrichton · 2015-02-23T17:54:40Z

@Manishearth no the fix for that problem specifically is #22723

Manishearth · 2015-02-23T17:55:44Z

Yeah, I'm talking about both failures here (stdtest, and the unsafe one).

…t, r=alexcrichton If these tests exit while a thread is panicking it often leads to situations like rust-lang#22628.

Manishearth · 2015-02-24T15:23:40Z

http://buildbot.rust-lang.org/builders/auto-win-32-nopt-t/builds/3449

rm: cannot remove ‘i686-pc-windows-gnu/stage0/bin/rustlib/i686-pc-windows-gnu/lib/syntax-4e7c5e5c.0.o’: Device or resource busy

brson · 2015-02-24T16:00:42Z

@alexcrichton max_builds=1 because the script indiscrimanently kills anything called stdtest.

@retep998 wants me to wrap the make calls with dojob.rs, which should make buildbot kill the whole process tree. Seems like a good idea. I hope that would let us remove the taskkill hack.

retep998 · 2015-03-05T00:48:39Z

Has anyone seen this issue lately since dojob was implemented?

alexcrichton · 2015-03-05T01:06:32Z

I have not, thanks @retep998!

Manishearth mentioned this issue Feb 21, 2015

Register new snapshots #22584

Merged

Manishearth mentioned this issue Feb 21, 2015

Spurious test failures #19120

Closed

13 tasks

kmcallister added O-windows Operating system: Windows A-infrastructure labels Feb 21, 2015

Manishearth added a commit to Manishearth/rust that referenced this issue Feb 21, 2015

Fix for intermittent Windows failures, see rust-lang#22628

a5fcca3

Manishearth mentioned this issue Feb 21, 2015

Fix for intermittent Windows failures, see #22628 #22648

Closed

klutzy mentioned this issue Feb 23, 2015

Unoptimized stdtest sometimes causes jemalloc assertion error #22710

Closed

Manishearth mentioned this issue Feb 23, 2015

Add cfgs (everywhere) in liblibc for NaCl targets. #21581

Merged

alexcrichton added a commit to alexcrichton/rust that referenced this issue Feb 23, 2015

test: Make two tests less flaky on windows

2408436

If these tests exit while a thread is panicking it often leads to situations like rust-lang#22628.

alexcrichton mentioned this issue Feb 23, 2015

test: Make two tests less flaky on windows #22723

Merged

Manishearth added a commit to Manishearth/rust that referenced this issue Feb 23, 2015

Rollup merge of rust-lang#22723 - alexcrichton:less-flaky-windows-tes…

d6d3377

…t, r=alexcrichton If these tests exit while a thread is panicking it often leads to situations like rust-lang#22628.

klutzy mentioned this issue Feb 24, 2015

sync::mpsc::select::test::stress is flaky on Windows #22771

Closed

brson mentioned this issue Feb 24, 2015

Automation metabug #17356

Closed

65 tasks

alexcrichton closed this as completed Mar 5, 2015

SirOlaf mentioned this issue May 13, 2024

Testament: Prevent intermittent permission denied errors on Windows nim-lang/Nim#23605

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Intermittent "Permission denied" on Windows #22628

Intermittent "Permission denied" on Windows #22628

Manishearth commented Feb 21, 2015

klutzy commented Feb 21, 2015

klutzy commented Feb 21, 2015

Manishearth commented Feb 21, 2015

Manishearth commented Feb 21, 2015

brson commented Feb 21, 2015

brson commented Feb 21, 2015

brson commented Feb 22, 2015

alexcrichton commented Feb 22, 2015

Manishearth commented Feb 22, 2015

Manishearth commented Feb 22, 2015

alexcrichton commented Feb 22, 2015

retep998 commented Feb 22, 2015

Manishearth commented Feb 23, 2015

alexcrichton commented Feb 23, 2015

Manishearth commented Feb 23, 2015

Manishearth commented Feb 24, 2015

brson commented Feb 24, 2015

retep998 commented Mar 5, 2015

alexcrichton commented Mar 5, 2015

Intermittent "Permission denied" on Windows #22628

Intermittent "Permission denied" on Windows #22628

Comments

Manishearth commented Feb 21, 2015

klutzy commented Feb 21, 2015

klutzy commented Feb 21, 2015

Manishearth commented Feb 21, 2015

Manishearth commented Feb 21, 2015

brson commented Feb 21, 2015

brson commented Feb 21, 2015

brson commented Feb 22, 2015

alexcrichton commented Feb 22, 2015

Manishearth commented Feb 22, 2015

Manishearth commented Feb 22, 2015

alexcrichton commented Feb 22, 2015

retep998 commented Feb 22, 2015

Manishearth commented Feb 23, 2015

alexcrichton commented Feb 23, 2015

Manishearth commented Feb 23, 2015

Manishearth commented Feb 24, 2015

brson commented Feb 24, 2015

retep998 commented Mar 5, 2015

alexcrichton commented Mar 5, 2015