[7.0.3RC2] Local files still encrypted after sync download

[PVince81]

Steps to reproduce:

1. Not sure, but I'm recovering from the situation raised here: All etags changed on server, client redownloads every file #11906 where all etags changed.

Expected result

All files properly downloaded

Actual result

Some files locally encrypted, but fine on server.

One file, "articles.txt" looks like this:

HBEGIN:cipher:AES-256-CFB:HEND-----

Another file "pomodoro.txt" (which is my TODO list) has a conflict:
The file "pomodoro.txt" is still encrypted.
The conflict file "pomodoro_conflict-20141101-091042.txt" is fine.

If I download "articles.txt" and "pomodoro.txt" from the web UI the files are decrypted properly.

VERY STRANGE.

Versions

ownCloud 7.0.3 RC2
Sync client 1.7.0 beta1

CC @schiesbn

@karlitschek @MTRichards @craigpg potential showstopper. Need to find a way to reproduce it locally first (only happened on my own server so far)

[PVince81]

When I change the file in the web UI with the text editor (the file is still decrypted there), the sync client will resync it down but the local result is STILL encrypted.

[PVince81]

Very strange: when I download the file with the Android client it appears in clear text.
Downloading with the Web UI also gives clear text.

Only downloading with the sync client leaves the file encrypted.

[PVince81]

After restarting the sync client the file is downloading properly.

I just remembered that I rebooted my server while the sync client was running...

I have the feeling that this is a session management issue: normally when the sync client logs in, the user's private key will be stored in the session. If I reboot the server, there's a chance that the session will still exist somehow.

Not sure how the session could mess itself up the way it did for me ?

I'll try to reproduce this in a controlled local environment.

@schiesbn @LukasReschke any idea ?

[PVince81]

There was nothing to be seen in the logs.
Even if we can't find the root cause, we should at least add some prevention measures to prevent the server from ever returning encrypted files.

[LukasReschke]

I have the feeling that this is a session management issue: normally when the sync client logs in, the user's private key will be stored in the session. If I reboot the server, there's a chance that the session will still exist somehow.
Not sure how the session could mess itself up the way it did for me ?

Mhm. Yes. But then the key should also still be within the session. - That's really very strange.

Do you happen to have a dump of the request that leads to this result? Also is the session content of this session different from the one when using Web-Based Login?

Does it work via the browser when logging-in via /remote.php/webdav/?

[PVince81]

It's already too late as I restarted the sync client.
Also I didn't find anything in the owncloud log, error log or access log. It seems to happily GET the file and return it encrypted.

As specified before, the Android client did not have any issues and that one is using WebDAV.

I'll have another try tomorrow and see whether rebooting the server while syncing/downloading could cause that issue again...

[schiessle]

Just a guess, maybe the sync client does something similar to the "remember" option for the web login. We disabled it for the web login if encryption is enabled because that will lead to the exact same result, the user will be logged in but without a private key because no password was provided.

cc @danimo @dragotin What do you think, could this be the reason?

[PVince81]

Seems to be difficult to reproduce locally. It could be a race condition.

[PVince81]

I tried setting $this->privateKey to false or empty string in stream_open, but in such case downloading a file through WebDAV will always return an error, never an encrypted file.

[PVince81]

Maybe it happened because I still had a session from before the upgrade, and it was kept after the upgrade, since I did not shut down my sync client. It was an upgrade to 7.0.3 RC2 back then.

@javiergonzper reported that it also happend to iOS clients just after an 7.0.3, so that might be a clue to find a way to reproduce it.

[javiergonzper]

We have the problem here:
owncloud/ios-legacy#246

And also other issue that could be related:
owncloud/ios-legacy#253

@PVince81 maybe I did not explain well on the irc. In our case the sessions are new because we are using a new install of the app.

[MTRichards]

Would be good to get this investigated! This could be real bad...

[PVince81]

Do we have an environment where this happens consistently ?
Because debugging this without that will take a long time, trying to read the code and infer what could have happened with the session.

[PVince81]

I tried upgrading from 7.0.2 to 7.0.3 with a running sync client + cadaver session, with encryption enabled: no issues.

Then I tried setting "session_timeout" to one minute and let sync client, cadaver and web UI timeout. Downloading files still delivers unencrypted files. Works properly.

I only had this once on my personal server but since I restarted the sync client the problem went away, so not possible to debug.

[javiergonzper]

@PVince81 on the daily server at the instance stable7 I can reproduce it using the current iOS app of the market.
But on the version: {"installed":"true","version":"7.0.4.2","versionstring":"7.0.4","edition":""}
I can not reproduce it by now...

I just added two account and change of account several times downloading the same file every time until it fails. Once it fails it does not work never.
We storage the cookies on every change of user to recovery it once we came back with the previous user.

[PVince81]

Just now I also tried stopping Apache2, deleting all session files while the sync client is still running. Then brought back Apache 2. The client still continued working.

So I'm still not sure what exactly is happening with the session.

For the iOS app if it's keeping the same cookie all the time and never re-logins, it means you'll still be on the broken session. On the desktop app, closing it and restarting it will retrigger a login and get a new session/cookie.

From what I remember when it happened to me (#11908 (comment)) I had rebooted the server completely so that killed Apache and maybe also the session (if there was an Apache update as well). But not sure.

[javiergonzper]

Yes, we are reusing the same session al the time. We asume that the server will manage the update of the cookie because on all the requests we send also the user/password. Are we right?

[PVince81]

I believe that if the cookie isn't valid it should reauthenticate automatically.

[PVince81]

One thing that could cause files to be downloaded encrypted is if the file proxy gets disabled (which happens in lots of places) but doesn't get reenabled for some reason, maybe some exception happening in the middle but which also wouldn't be thrown out and continue processing (for example hooks swallow exceptions...)

[PVince81]

The app stores two informations in the session: the private key and the initialization mode.
If the private key is missing, exceptions are thrown.
If the initialization mode is not set, then encryption is disabled and it returns encrypted files (tested locally by commenting out the code that sets the mode into the session).

But looking at the code there is no visible code path that would have the private key present but not the initialization mode. To make this happen the PHP process would have to get killed while a session is being invalidated exactly between these two lines: https://github.com/owncloud/core/blob/master/apps/files_encryption/lib/session.php#L140
And then, in theory, the session would still exist but with only half of the values cleared. (not sure if THAT is even possible)

[PVince81]

I tried to sabotage the sync client's session by setting encryptionInitialized to 0 by editing the session file directly. All files downloaded after that are encrypted.

This doesn't prove that it is actually our problem though...

[PVince81]

If it happens again for anyone we should grab the matching session files from the server for further examination.

[javiergonzper]

@PVince81 is it possible that the encryption are using a wrong session (previous session) to decrypt the files and it is not using the last one?

[PVince81]

Yes. But normally all code paths should lead to either the old session to be destroyed/invalidated or the old session should still contain a valid private key.

For a session to be valid it need to contain:

1. The private key
2. The initialization status must be set to "2"

If for some mysterious reason one of both is missing it could lead to issues.
From my findings, not having the private key doesn't even allow downloads (exception) so maybe the case we saw was rather that the initialization status had the wrong value (0 instead of 2).

But from all code paths it looks like it's impossible to reach a situation where the private key is in the session but initialization status is still 0.

So best would be to find the session file on the server for that broken session so we can look at these two values.

[PVince81]

Something to try: setting session.gc_maxlifetime in php.ini instead of using OC's config.php.
This might trigger premature deletion of the session files when OC isn't expecting it.

[tflidd]

i set this value from ~20 minutes to 10hours. First, it seemed to have changed something but now the problems are back. Maybe it takes longer for them to return.

[sjlawson]

This is happening to me, but I'm not the administrator of the server. Using client ver 1.5.0 on Ubuntu 14.04. The server uname -a output is:
Linux ---.---.org 2.6.32-504.8.1.el6.x86_64 #1 SMP Wed Jan 28 21:11:36 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
OwnCloud server ver 8.0

Is there a way to manually decrypt the files?

[PVince81]

I believe we fixed this a while ago and it was due to cookie handling ? @LukasReschke @javiergonzper @schiesbn

@sjlawson you cannot decrypt the file locally. To recover from this situation, either:
A) Reset your local sync client and let it redownload everything
or
B) Setup a separate sync client and let it download everything (or use selective sync to only focus on specific directories). Make sure that the download files are not encrypted. Then "touch" the files you knew to be broken so that the second sync client reuploads these files. This will make the first sync client think the files changed and it will redownload them. At least that's how I recovered mine.

Advanced stuff: what I did was grep -rl HBEGIN * which gave me the list of "broken" files. Then in the folder of the second client, I triggered "touch" for every file in that list.

[MorrisJobke]

We didn't noticed something similar to this during the past year. I guess this is solved - also encryption 2.0 changed a lot in the architecture.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.