fixes #239: query the database in chunks of 200

[DeepDiver1975]

Currently the sql query will fail as soon as a certain number of items in the IN condition has been reached.
(1000 for Oracle and sqlite - no idea about the others)
The proposed solution will cut the array into tiny little pieces and execute the statement multiple times.

Fixes #239

My setup: apache + sqlite + all on localhost:

current implementation: 0.1331 sec.
proposal within #329: 0.5020 sec
chunked in packs of 999: 0.3075 sec
chunked in packs of 200: 0.2089 sec

chunk size of 200 gave the best results with my setup

A LIKE based query will not work due to the special case of 'Shared' - afaik

[eMerzh]

Seems good for me... 🐑 it then ;)

[dragotin]

I dislike your patch, sorry. Doubtless its more performant but it replaces complex and wrong code with more complex code. Of course we now understand it and can read it, but in three weeks from now everybody going through the code will wonder why there is another loop over the chunks.

The target of ownCloud must be to keep code simple IMO. I also doubt that performance improvements on this level for the cost of more complex code make sense for us. There are many other ways to improve performance on higher levels with the database.

[DeepDiver1975]

array_chunk + one foreach is not what I call complex - sorry!

And if we want to simplify code there are other places which deserves it much more.

And finally imagine a folder with 10000 files - we don't want to fire 10000 database queries.

[danimo]

Whatever impl we choose, can we make sure to make this a blocker for 5.0? This looks to me like the DB requires some serious restructuring here.

[blizzz]

Please add comments which say what is being done and why to solve the readability issue.

[DeepDiver1975]

@blizzz true for sure - thx for the hint
@danimo good luck ;-)

[dragotin]

@DeepDiver1975 : lol, expected this answer :) Please bear with the simple structured guys like me who like less complex code more because it makes life easier ;-)

BTW - can you easily redo your benchmarking of the solutions with a mysql database backend by chance? Just out of curiosity.

[DeepDiver1975]

@dragotin
I can setup the benchmarking with mysql - but not before tomorrow night - too busy with my real life project ;-)

[tanghus]

ps aux|grep reallife | awk '{print $2}' | renice -n 19

[scroogie]

I actually think that this issue shows a weakness in the database scheme, just as danimo said. It would be fine if shared files wouldn't be an exception. But even in this case, I'd probably do it this way:

• Introduce an index on propertypath
• Query the properties with a prefix scan: WHERE propertypath LIKE foo%
• Add the "special" case of shared files

This will be ugly as well, but it will be easier to remove later on, once the special case of shared files is solved. Also it is easier to read in my opinion, as with the current code you don't even realise that shared files have to be handled differently in this situation.

[bartv2]

👍 looks good, simple solution for this bugfix

[LukasReschke]

@DeepDiver1975 Feel free to backport :-)