mmap(2) issues: prefault, atomic mmap

[GabeAl]

WSL 1709 Ubuntu 16.04.3

I noticed that the behavior of both the MAP_POPULATE flag in mmap and the MADV_WILLNEED on "real linux" will pre-fault the entire file into the mapped memory region, the only difference between the two being MAP_POPULATE operates at the time the mapping is established, and MADV_WILLNEED operating at the time of the first memory access into the mapped region.

In effect this is the rough equivalent to reading the entire file into memory (i.e. fread) but without an internal buffer to hold the contents (the page table is completely pre-faulted and the virtual pointer into the page table is returned).

On Windows, the blocking pre-fault behavior is not seen. This causes atrocious disk I/O performance especially on spindle devices that are reliant on this behavior, as pages are faulted in only in the accessed region (that is, the "vanilla" mmap behavior sans pre-faulting). I saw that a few million binary searches on a 9GB mmap'd file took hours on WSL but seconds on docker and "real ubuntu 16.04" on the same machine.

A naive workaround is to pre-populate the file into pagecache sequentially via other means (such as calling md5 or wc -l on the file before trying to mmap it).

Example:

char *Raw = mmap(0,size,PROT_READ,MAP_SHARED | MAP_POPULATE, fp, 0);
madvise(Raw,size,MADV_WILLNEED); 
// access values of Raw at random (or in a binary search pattern)

(either the MAP_POPULATE or the MADV_WILLNEED hint should suffice on their own but both are shown here.)
Be sure the file (I use files >8GB but anything big will do) isn't already paged into RAM or you won't see the behavior. I use RAMMap to drop page caches in Windows to reproduce what you'd see if you're reading the file for the first time from disk.

Second bug (and this is an actual "bug" not just a non-implementation of prefault): the mmap operation is not atomic as it is on real linux and as specified by POSIX. This behavior is seen when using MAP_FIXED to re-map an area of memory back to the same pointer returned via previous mmap command (without MAP_FIXED on address 0). Forcing the operation to be atomic via thread sync is a workaround that fixes the behavior.

Example 2:

char *Raw = mmap(0, fsz+16, PROT_READ, MAP_SHARED, fp, 0);
#pragma omp parallel
//#pragma omp critical
Raw = mmap(Raw, fsz+16, PROT_READ, MAP_SHARED|MAP_FIXED, fp, 0);
// Use Raw[value]...

If the omp critical line is uncommented (aka synchronization is forced), behavior is as expected. Otherwise SIGSEGV 11 is issued on WSL yet this should be [and is] fine on "real linux" due to guaranteed atomicity of the operation.

Although MAP_POPULATE is linux-exclusive, the roughly equivalent MADV_WILLNEED, as well as the atomicity of the mmap + fixed operation, work across kernel versions and distros, as well as BSD and OSX, so it's pretty POSIX-y.

[jstarks]

Good feedback. The second issue in particular is concerning (and surprising given our relatively simple locking scheme).

Do you know of any packaged software that reproduces these issues, or are you observing this in private projects?

[jstarks]

I have a fix for the atomic mmap issue. It will be in a future insiders build. I have not fixed the prefetch-related issues yet.

[Brian-Perkins]

The atomic mmap issue should be fixed in 17083 Insider Build

[GabeAl]

Thanks! Any ETA on the prefetch-related issues?