[Debian gocryptfs 2.3-1] Intermittent "Operation not permitted" errors when rsync'ing to a gocryptfs mount

[glhughes123]

I have a target directory mounted with gocryptfs and I'm using rsync to copy files from an (unencrypted) source directory to it. I'm getting intermittent errors from rsync when it uses mkstemp and mkdir to create temporary files and directories during the sync process. For example:

rsync: [receiver] mkstemp "/backups/ares/home/greg2/foo.incomplete/Music/Music/Media.localized/Music/Compilations/The Raw and the Cooked/.01 She Drives Me Crazy.m4a.EXulcp" failed: Operation not permitted (1)
rsync: [generator] recv_generator: mkdir "/backups/ares/home/greg2/foo.incomplete/Music/Music/Media.localized/Music/Blondie/Autoamerican (Remastered 2001)" failed: Operation not permitted (1)

These issues happen seemingly at random and to different sets of files/directories each time I run the copy (same set of source files, always to an empty directory).

FWIW, I don't see these issues with cp, nor do I see the issues using rsync writing to an encfs mount.

I'm using:

• Debian bookworm (12.9)
• backports kernel (6.11.10+bpo-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.11.10-1~bpo12+1 (2024-12-19))
• default gocryptfs package (gocryptfs 2.3; go-fuse 2.1.0+git20220822.58a7e14; 2023-04-09 go1.19.8 linux/amd64)

This also repros with a local build of the latest gocryptfs source (commit 1464f9d).

[glhughes123]

After some further testing it appears that this issue occurs when:

• the -allow_other flag is specified, and
• the directory is mounted by root, and
• the rsync is performed by a non-root user

The non-root user is the owner of the encrypted folder, the mount point, and the source folder/files.

I cannot repro this issue if either the rsync is run by root (regardless of who mounts the directory) or if the mount is performed by the non-root user (with or without the -allow_other flag).

[rfjakob]

Hi, thanks for the report! What backing filesystems are in use at source and target?

[glhughes123]

The encrypted target folder is on an EXT4 filesystem. The FS is on an LVM partition that is LUKS encrypted on a Linux MD array (RAID6).

The issue occurs both when the source filesystem is the same as that of the encrypted target folder and when the source filesystem is remote (APFS on a Mac).

[glhughes123]

FWIW, this seemed like a race to me when I was playing around with it. That makes me think it's worth mentioning a couple things:

• IIRC, turning on debug logging spew to the console (FUSE, gocryptfs) seemed to eliminate or at least reduce the frequency of the issue.
• The target is backed by SSDs w/ thread and stripe cache tweaks in the RAID config so writes are pretty fast / high IOPS.

[glhughes123]

Noticed the discussion in #893 so wanted to add that in my case the system running gocryptfs is an Intel Xeon w7-3465x (Sapphire Rapids).

[rfjakob]

So, golang/sys@d0df966 broke it, and the Debian x/sys/unix package https://packages.debian.org/source/bookworm/golang-golang-x-sys is new enough to have this commit.

If you build gocryptfs v2.3.0 from source the problem should disappear. Can you confirm?

[glhughes123]

Oh great, thanks for finding the root cause.

Building v2.4.0 from source eliminates the problem so I'm inclined to go with that unless there's a specific reason to prefer v2.3.0 over v2.4.0.

[rfjakob]

I suggest using v2.5.1, released just now.

[glhughes123]

Built v2.5.1 from source and it seems to be working as expected.

[3Galley]

Hello
If i could add
Im seeing a similar behavior of some files not being able to be edited or overwritten if they are in the encrypted volume when testing 2.50 and 2.51.
For example, a word document with content if edited while directly saving on to the encrypted volume will save a 0byte file. Thus nothing is actually saved. I get no errors from Word itself. But now the file is empty
If using Excel, for example. It will save the file correctly the first time the document is created. it completes the save fine.
When modifying the file and attempting to save gives 3 error messages:
"Your changes could not be saved to
'Book1.xlsx' because of a sharing violation. Try saving to a different file."
"Sorry, we couldn't find /Users/alain/ Desktop/mount/Book1.xlsx.sb-
cd720ce3-RM4XYY/0E495000.MACTF.
Is it possible it was moved, renamed or deleted?"
"Your changes could not be saved to
'Book1.xIsx', but were saved to a temporary document named
'OE495000.MACTF'. Close the existing document, then open the temporary document and save it under a new name."
All the while the actual document dissapeared from the encrypted volume, only its temporary file (~$Book1.xlsx)
Ill attach some screenshots if it helps

So far the only one that works without issues on my mac is the 2.3.1 from MacPorts
The 2.5.1 I compiled it from the github after installing the pre requirements and built using the bash > build-without-openssl.bash

Would be awesome to use the latest version on my mac :)
Let me know what else i could provide to help squash this bug

---

gocryptfs v2.5.1-1-g5169c47 without_openssl; go-fuse v2.5.0; 2025-01-25 go1.23.5 darwin/arm64

debug output cli args: ["gocryptfs" "-fg" "-notifypid=8638" "--debug" "crypt" "mount"]

Password: 

Decrypting master key

cryptocore.New: key=32 bytes, aeadType=AES-GCM-256-Go, IVBitLen=128, useHKDF=true

contentenc.New: plainBS=4096

CryptoCore.Wipe: Only nil'ing stdlib refs

cryptocore.New: key=32 bytes, aeadType=XChaCha20-Poly1305-Go, IVBitLen=192, useHKDF=true

contentenc.New: plainBS=4096

nametransform.New: longNameMax=0, raw64=true, badname=[]

frontendArgs: {

"Cipherdir": "/Users/alain/Desktop/crypt",

"PlaintextNames": false,

"LongNames": true,

"PreserveOwner": false,

"ForceOwner": null,

"ConfigCustom": false,

"NoPrealloc": false,

"Exclude": null,

"ExcludeWildcard": null,

"ExcludeFrom": null,

"Suid": false,

"KernelCache": false,

"SharedStorage": false,

"OneFileSystem": false,

"DeterministicNames": false

}

DetectQuirks: Fstypename="apfs"

OpenDir ".": invalid entry ".DS_Store": illegal base64 data at input byte 0

decryptName "zero": decoded length 3 is not a multiple of 16

OpenDir ".": invalid entry "zero": bad message

ino81414905: FUSE Read: offset=0 length=6148

doRead: off=0 len=6148 -> off=18 len=8272 skip=0

ReadAt offset=18 bytes (0 blocks), want=8272, got=6228

ino81414905: Read: errno=0, returning 6148 bytes

ino81414905: FUSE Read: offset=0 length=6148

doRead: off=0 len=6148 -> off=18 len=8272 skip=0

ReadAt offset=18 bytes (0 blocks), want=8272, got=6228

ino81414905: Read: errno=0, returning 6148 bytes

ino81414905: FUSE Read: offset=0 length=6148

doRead: off=0 len=6148 -> off=18 len=8272 skip=0

ReadAt offset=18 bytes (0 blocks), want=8272, got=6228

ino81414905: Read: errno=0, returning 6148 bytes

ino81414905: FUSE Read: offset=0 length=6148

doRead: off=0 len=6148 -> off=18 len=8272 skip=0

ReadAt offset=18 bytes (0 blocks), want=8272, got=6228

ino81414905: Read: errno=0, returning 6148 bytes

Filesystem mounted and ready.

System Version: macOS 14.7.2 (23H311)

  Kernel Version: Darwin 23.6.0

Underlying FS: APFS

[rfjakob]

The situation with ms office on mac is, unfortunately, another can of worms. I have even seen it failing on smb shares.

…

On Sun, 26 Jan 2025, 02:30 3Galley, ***@***.***> wrote: Hello If i could add Im seeing a similar behavior of some files not being able to be edited or overwritten if they are in the encrypted volume when testing 2.50 and 2.51. For example, a word document with content if edited while directly saving on to the encrypted volume will save a 0byte file. Thus nothing is actually saved. I get no errors from Word itself. But now the file is empty If using Excel, for example. It will save the file correctly the first time the document is created. it completes the save fine. When modifying the file and attempting to save gives 3 error messages: "Your changes could not be saved to 'Book1.xlsx' because of a sharing violation. Try saving to a different file." "Sorry, we couldn't find /Users/alain/ Desktop/mount/Book1.xlsx.sb- cd720ce3-RM4XYY/0E495000.MACTF. Is it possible it was moved, renamed or deleted?" "Your changes could not be saved to 'Book1.xIsx', but were saved to a temporary document named 'OE495000.MACTF'. Close the existing document, then open the temporary document and save it under a new name." All the while the actual document dissapeared from the encrypted volume, only its temporary file (~$Book1.xlsx) Ill attach some screenshots if it helps So far the only one that works without issues on my mac is the 2.3.1 from MacPorts The 2.5.1 I compiled it from the github after installing the pre requirements and built using the bash > build-without-openssl.bash Would be awesome to use the latest version on my mac :) Let me know what else i could provide to help squash this bug ------------------------------ gocryptfs v2.5.1-1-g5169c47 without_openssl; go-fuse v2.5.0; 2025-01-25 go1.23.5 darwin/arm64 debug output cli args: ["gocryptfs" "-fg" "-notifypid=8638" "--debug" "crypt" "mount"] Password: Decrypting master key cryptocore.New <http://cryptocore.new/>: key=32 bytes, aeadType=AES-GCM-256-Go, IVBitLen=128, useHKDF=true contentenc.New <http://contentenc.new/>: plainBS=4096 CryptoCore.Wipe: Only nil'ing stdlib refs cryptocore.New <http://cryptocore.new/>: key=32 bytes, aeadType=XChaCha20-Poly1305-Go, IVBitLen=192, useHKDF=true contentenc.New <http://contentenc.new/>: plainBS=4096 nametransform.New <http://nametransform.new/>: longNameMax=0, raw64=true, badname=[] frontendArgs: { "Cipherdir": "/Users/alain/Desktop/crypt", "PlaintextNames": false, "LongNames": true, "PreserveOwner": false, "ForceOwner": null, "ConfigCustom": false, "NoPrealloc": false, "Exclude": null, "ExcludeWildcard": null, "ExcludeFrom": null, "Suid": false, "KernelCache": false, "SharedStorage": false, "OneFileSystem": false, "DeterministicNames": false } DetectQuirks: Fstypename="apfs" OpenDir ".": invalid entry ".DS_Store": illegal base64 data at input byte 0 decryptName "zero": decoded length 3 is not a multiple of 16 OpenDir ".": invalid entry "zero": bad message ino81414905: FUSE Read: offset=0 length=6148 doRead: off=0 len=6148 -> off=18 len=8272 skip=0 ReadAt offset=18 bytes (0 blocks), want=8272, got=6228 ino81414905: Read: errno=0, returning 6148 bytes ino81414905: FUSE Read: offset=0 length=6148 doRead: off=0 len=6148 -> off=18 len=8272 skip=0 ReadAt offset=18 bytes (0 blocks), want=8272, got=6228 ino81414905: Read: errno=0, returning 6148 bytes ino81414905: FUSE Read: offset=0 length=6148 doRead: off=0 len=6148 -> off=18 len=8272 skip=0 ReadAt offset=18 bytes (0 blocks), want=8272, got=6228 ino81414905: Read: errno=0, returning 6148 bytes ino81414905: FUSE Read: offset=0 length=6148 doRead: off=0 len=6148 -> off=18 len=8272 skip=0 ReadAt offset=18 bytes (0 blocks), want=8272, got=6228 ino81414905: Read: errno=0, returning 6148 bytes Filesystem mounted and ready. System Version: macOS 14.7.2 (23H311) Kernel Version: Darwin 23.6.0 Underlying FS: APFS Screenshot.2025-01-25.at.7.26.08.PM.png (view on web) <https://github.com/user-attachments/assets/f255e75b-b54e-4778-8f38-f4d7ec169f8f> Screenshot.2025-01-25.at.7.26.18.PM.png (view on web) <https://github.com/user-attachments/assets/e73048eb-f30c-43d1-9b75-667b8e78bbfa> Screenshot.2025-01-25.at.7.26.29.PM.png (view on web) <https://github.com/user-attachments/assets/6bdec327-6cf8-414b-90d7-eb192f4f2750> Screenshot.2025-01-25.at.7.27.38.PM.png (view on web) <https://github.com/user-attachments/assets/94cd865a-b7f4-4ba8-9405-8de4adaca837> Screenshot.2025-01-25.at.7.27.58.PM.png (view on web) <https://github.com/user-attachments/assets/548cd7bf-aa7d-423c-a97c-1ada23297193> — Reply to this email directly, view it on GitHub <#892 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACGA7YPWU53EPPFZJMNHUD2MQ3CVAVCNFSM6AAAAABVLA6Y6SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMJUGE3DGOBZGA> . You are receiving this because you commented.Message ID: ***@***.***>

[rfjakob]

@glhughes123 is there already a ticket in https://bugs.debian.org/ ?

[rfjakob]

(Plan is to release an easily-backportable fix for new x/sys versions, would be nice to get it into Debian's gocryptfs package.)

[glhughes123]

@glhughes123 is there already a ticket in https://bugs.debian.org/ ?

@rfjakob, I did not file a bug. I'm not aware of one that currently exists. I searched for bug reports against what I believe to be relevant go packages (e.g. golang-golang-x-sys-dev) but didn't see any relevant bugs.