recover from missing or corrupt gocryptfs.diriv

[mabod]

Now that I learned how to restore gocryptfs.conf just with the master key I am wondering how this is going for gocryptfs.diriv. I am not sure what gocryptfs.diriv is good for, but without it I can not write to the decrypted folder. How can I recover from a missing or corrupt gocryptfs.diriv?

[rfjakob]

The diriv files are 16 bytes of random data used for randomizing the encrypted file names ( see https://nuetzlich.net/gocryptfs/forward_mode_crypto/#file-names ).

If you lose one diriv file, you lose all file names in this directory, as they can no longer be decrypted. You can recover all file content by creating a new random diriv in the affected encrypted directory:

dd if=/dev/urandom of=gocryptfs.diriv bs=16 count=1

Then creating a bunch of empty files in the decrypted directory:

touch file{1..10}

This gives you 10 valid file names in the encrypted directory. You can then rename each of your old files over one of the new empty files.

[mabod]

How is that supposed to work with a Documents folder with hundreds of files and dozens subdirectories?
What does that mean for a recovery strategy? It is not sufficient to remember the master key to recover from a lost gocryptfs.conf. I also need to backup gocryptfs.diriv.
Is that correct?

[rfjakob]

There is one gocryptfs.diriv in every directory, and yes, you should back up the whole gocryptfs folder, including gocryptfs.conf, all the data files, and all the gocryptfs.diriv files.

[rfjakob]

And in addition, you should save or print the masterkey, primarily to recover when you forget your password.

[mabod]

This is very important information which should be shared at a prominent position in the documentation. It makes gocryptfs.diriv a single point of failure. As soon as this is gone my picture gallery with thousands of *.CR2 files and corresponding *.xmp files is destroyed. No chance to recover.

The gocryptfs documentation basically says that all you need to recover a gocryptfs container is the master key. That is surely wrong and needs clarification from my point of view.

[rfjakob]

I'm not sure if there is a misunderstandig here, but all subdirectories will be fine if you lose gocryptfs.diriv in the root directory.

[mabod]

And why is the architecture such that the master key is not sufficient to fully(!) recover a container? Is that a must have or a flaw in the design?

[mabod]

I'm not sure if there is a misunderstandig here, but all subdirectories will be fine if you lose gocryptfs.diriv in the root directory.

But still, with hundreds of pictures and sidecar files in one directory the data is basically lost.

[rfjakob]

The gocryptfs.diriv files exist to prevent an attacker from being able to identify identical file names in different directories. If you don't have a need to protect your file names, you can use -plaintextnames.

[mabod]

Wouldn't it be possible to protect the filenames with the master key? So that the master key is the only info needed to fully recover? The master key is the only thing that I have printed on paper.

[rfjakob]

It's possible, yes, and other solutions do it. But you get weaker security.

gocryptfs exists a few years now, and I have really never seen a report of somebody losing a gocryptfs.diriv file. They are only written once (on directory creation), marked read-only immediately, and cannot be accessed at all through the decrypted view.

[mabod]

I understand your point that it is unlikely to loose the gocryptfs.diriv file. But it can happen. Most likely it will happen by a human mistake, a script going wild, etc.

And if it happens it is probably catching a lot of people by surprise because they might think - like myself - that the master key is all you need to recover your data.

[bexelbie]

I manage my gocryptfs.diriv files just like the other files. I branch them up and periodically test the backup. I suspect having a deletion error that only hit those files would be extremely unusual. The files are safe to backup to even “untrusted” places. What’s the scenario for the loss of these files but of no other files?

[mabod]

What’s the scenario for the loss of these files but of no other files?

That is not the point. This is not about defining unlikely events.

My point is, that the gocryptfs.diriv file is a single point of failure which is unrecoverable! You can recover a lost gocryptfs.conf file with the master key, but you can not recover from a lost gocryptfs.diriv file. Your folder with 1000+ RAW photos including the same amount of sidecar files with the same basename are basically lost. I dont care how unlikely this event is. If the risk is to loose all my photos and settings in darktable, I rather want to know about that risk before I start using gocryptfs.

I am not asking to fix that behaviour. If it is a design/security decision to do it that way, so be it. But it should be communicated at a prominent position that gocryptfs.diriv files are that important.

[newhoa]

Just some clarification, your files/photos and data will not be lost if you lose a gocryptfs.diriv file. The file names will be lost, when you do the steps in this comment, you will get all your files, just with different filenames. Losing file names in some cases is almost as bad (esp if specific file names are important for the reconstruction of your program or situation -- say a source file directory that is responsible for compiling a program). But the data, all your photos, will still be there.

I do think this is a problem and it would be nice to see some sort of solution.

I do think there should be a way for gocryptfs to automate the process in this comment if it finds a corrupt or missing .diriv file when you run an fsck. For a casual person trying to encrypt/backup their files, I think they would have a really difficult time doing that themselves, especially since it is not made aware to them in the program (if fsck detects corrupt or missing .diriv files, there should be some info given to the user on how to fix that problem, maybe a wiki entry and a link in the fsck output). But ideally it could ask the user if they'd like a new .diriv created and warn that the file names would all be changed.

One solution could be to give the option to use the master key file for file names. Even if it's weaker it could still be optional, though I don't know if that would take a lot of effort to implement.

Another solution could be to create some sort of parity file to repair any damaged .diriv files if they're found using fsck. then you could have 2 .conf files, a master key and a .diriv master.

A sort of simple option would be to add a backup option to gocryptfs, which could backup the master key and all the .diriv files. Either the .diriv files themselves in a tar retaining their directory structure, or maybe all the .diriv contents and location info backed up to a file which could be restored later. Of course, the backup would have to be kept up to date by the user for this to be at all useful. But does gocryptfs consider a .diriv corrupt if it's old? If so this would not work if you had old .dirivs once yours were corrupted or lost.

I'm basically doing that last thing right now -- I'm just backing up the master key and all the .diriv files (8MB of dirivs for 1TB of data uncompressed, 500KB as a tar.gz). It's very quick and simple to do. This is in addition to my entire gocryptfs backup of course (so I have 2-3 copies of entire gocryptfs backup, and then backups of just the master key and .dirivs).

[mabod]

In the meantime I am doing backups for the gocryptfs.diriv files myself. There is no way back for me to encfs.

But the ideal solution for me would be if the master key is all you need to restore everything 100 %.

[rfjakob]

Documented in https://github.com/rfjakob/gocryptfs/wiki/Best-Practices

[loungebob]

I’ve got 2 questions:

1. the diriv file from my root dir is about to be deleted which would mean I would lose all directory names and file names in the root dir, right? there is no way to force create a new diriv that would match the one about to be deleted, the only way to retain it would be to create a backup of the root diriv file? (the diriv is not created based on the folder name it resides is but rather from random data?)

2. is there still no automated migration path from diriv to deterministic fs?

[rfjakob]

Hi,

1. Correct

2. Correct. That -deterministic-names exists since gocryptfs v2.2.0 (released in 2021) may interest @mabod .

[mabod]

I am no longer using gocryptfs.