File upload in browser - use PUT if possible

[DeepDiver1975]

Our file upload javascript component is capable of using PUT for file upload
https://github.com/blueimp/jQuery-File-Upload/wiki/Options

Using PUT has some advantages over the existing POST:
The PHP code will be trigger as soon as the first few kbs arrived th server.
In contrary in case of POST the web server will first receive the whole file and trigger the PHP code after the upload has been finished.

Why is this a better approach:

• quota and other checks can be performed right in the beginning
• follow up processing can benefit from stream e.g. push data to external file systems, encrypt on the fly (?)

From my understanding this is why WebDAV uses PUT for file upload as well.

@karlitschek @danimo @dragotin @icewind1991 @butonic @blizzz @schiesbn @bartv2 @bantu @ringmaster

I'm summoning you all because I'd like to know any impact on any component you are working on if you see any issues with this approach.

THX

[danimo]

Next advantage: Once we switch to a PUT-based approach, we can implement file chunking, too. This means that for all practical purposes (with the unfortunate exceptions of the infamous IE 8 and 9), we do not have to deal with stupid PHP upload limitations. This alone will save us quite some headaches in the future.

[icewind1991]

No need for that, there isn't a restriction on the size of PUT uploads in the first place

[danimo]

@icewind1991 good point. still, chunks could be useful for resuming big downloads.

[butonic]

There is an in depth description at http://stackoverflow.com/questions/12005790/how-to-receive-a-file-via-http-put-with-php

Don't know if we have to manually cut the headers from the body. Will take a look along with the client side changes.

[DeepDiver1975]

@butonic webdav/sabredav already uses PUT for file upload - here is the code:
https://github.com/fruux/sabre-dav/blob/master/lib/Sabre/DAV/CorePlugin.php#L421

[blizzz]

Will apps that require the whole file behave properly? Like encryption or versioning for example.

[DeepDiver1975]

Will apps that require the whole file behave properly? Like encryption or versioning for example.

From my understanding all file operations within ownCloud work on streams - if not we need to change this.
We cannot expect to have enough server RAM to process e.g. 10 user requests on a 4 GB file all in RAM.

@icewind1991 @schiesbn

[bartv2]

Also now we have auth via session maybe we can use the webdav interface to do uploading of files?

[LukasReschke]

Adding to technical debt. Would make very much sense. @MorrisJobke Does something similiar in his music app for reading files.

[MorrisJobke]

I just use get because I don't need PUT nowadays, but we should enforce using our own APIs and the WebDAV is the most stable of them ;)

[DeepDiver1975]

this will be implemented once we move over to use webdav in the files apps #12353

[PVince81]

I suggest we keep this ticket as a subticket of #12353.

Implementing PUT is already a challenging task, especially in regards to cross-browser compatibility. So it makes sense to have this implemented in its own PR.

[PVince81]

Moving to Webdav for uploads is scheduled for 9.0, adjusting milestone.

[PVince81]

CC @cmonteroluque

[ghost]

nod

[PVince81]

Hint: this is still open. Maybe we can have a look at this https://github.com/23/resumable.js which seems to allow for chunking

[PVince81]

The toughest part here is making file conflict detection work with our conflict dialog, for BOTH the webdav approach (using precondition + 412 status) AND also for the IE8 workaround that uses ajax/upload.php... I'll see if I can bring their behavior / responses as close as possible.

[PVince81]

"tough" as in: lots of refactoring and rearranging response formats. The file-upload.js is a bit messy 😄

[PVince81]

WIP PR here #21237

[butonic]

I'll keep an eye on the tough part to educate myself on clean js code. Boy this has been a long time ago. And I still remember what the code looked like before ... ;)

[PVince81]

Interesting, the blueimp uploader would POST to this URL with the IE iframe fallback:

192.168.56.102 - - [17/Dec/2015:18:17:06 +0100] "POST /owncloud/remote.php/webdav/modern.IE.1024x768.jpg?_method=PUT HTTP/1.1" 501 247 "http://192.168.56.1/owncloud/index.php/apps/files/" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)"

This means that in theory instead of keeping "ajax/upload.php" we could simply write a Sabre plugin that processes this POST and internally plugs with the PUT logic. 😄

[PVince81]

Still needs a bit more work for the PR #21237, moving to 9.1

[MTRichards]

Estimation: 1 week

[PVince81]

Someone raised a ticket for PUT chunking. Since we didn't have a dedicated ticket, let's use this one #25181

[PVince81]

Hmm, but when overwriting a file we are using POST on collection with "Add-Member" here https://tools.ietf.org/html/rfc5995#page-5 here https://github.com/owncloud/core/blob/master/apps/dav/lib/Connector/Sabre/FilesPlugin.php#L440.
This is used when the conflict dialog appears and the user wants to keep old files, so we use "Add-Member" to suggest a name for the server. The server then append "(2)", "(3)" or whatever depending on existing files.

Drawback is that we still use POST and would suffer from limitations.
Reopening to reconsider this.

Also this approach is not compatible with new chunking, so might still need a client-side only approach that looks like this:

1. PUT file "test.dat"
2. Server answers with 409 conflict
3. User resolves conflict in dialog by ticking both "New files" and "Already existing file"
4. Client checks current file list (which might not be up to date) to add a "(2)" in the name: "test (2).dat"
5. Client uploads "test (2).dat"
6. Server answers with 409 conflict again because someone concurrently created "test (2).dat"
7. Client tries again with next "test (3).dat", etc.

[PVince81]

Fix for autorename is here: #26078
This removes POST completely for uploads.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.