Run app tests against latest released version

[PVince81]

Currently we only run app tests against stable10 to make sure they will be compatible with the future OC core platform release.

This however doesn't guarantee that the current app's core runs correctly against the last released core version.

Now I'm not sure whether we want to explode the matrix of all apps to also add the last released version.
What we could do is do so only on release branches: so whenever we plan a release (RC), we'd have tests there that don't only run against core stable10 but also against the last released version (presently 10.1.0).

Not sure how to specify said version in Drone, should we hard-code ? Or fetch it from a URL ?

@phil-davis @individual-it

cc @patrickjahns for potential additional ideas

[patrickjahns]

Yes this definitely makes sense to be doing ( was something on my agenda for a long time ).

For this however we need to sort out how our testing harness ( that currently lives in owncloud/core ) can be backward compatible as well.

With running the smoke-test suites on our docker containers ( which are built periodically after a release to keep the underlying linux binaries up-to-date ) we've encountered issues, where tests have changed/improved ( or new test cases have been added due to a new feature ) and suddenly where failing the container builds.

currently we solve this by tagging scenarios with skipOn - and we have to be careful with setting the right tags here. Would envision that we move the skipOn logic into a skipBefore logic - where we compare the ownCloud version and then run according tests.

Something to be later defined is the part about how often these tests should run ( every PR, every night? )

Another thought:

• what about the min-version constraint - I believe if we are able to check the latest version, we would also need to confirm that the min-version still holds true, or raise the min-version accordingly

[dpakach]

Apps to test with latest release version

• activity Run tests against latest stable tarball release activity#745
• admin_audit https://github.com/owncloud/admin_audit/pull/210
• brute_force_protection Run acceptance tests against latest stable release of core brute_force_protection#67
• customgroups customgroups test against latest tarball customgroups#222
• data_exporter data_exporter app test against latest tarball data_exporter#84
• files_antivirus - Test app against latest release tarball files_antivirus#316
• files_classifier wip https://github.com/owncloud/files_classifier/pull/180
• files_texteditor files-editor app test against latest release tarball files_texteditor#294
• firewall - https://github.com/owncloud/firewall/pull/581
• firstrunwizard firstrunwizard test against latest tarball firstrunwizard#111
• guests guest app test against latest release tarball guests#339
• notifications - Run tests against latest stable release tarball notifications#286
• oauth2 - Run acceptance tests in latest stable release of core oauth2#214
• password_policy - Run acceptance tests in latest stable release of core password_policy#232
• ransomware_protection - https://github.com/owncloud/ransomware_protection/pull/184
• search_elastic -Run acceptance tests in latest stable release of core search_elastic#117
• testing Run tests against latest stable release of core testing#100
• twofactor_totp wip twofactor_totp app test against latest tarball twofactor_totp#125
• user_ldap user_ldap app test against latest tarball user_ldap#434
• user_management user management app test against latest tarball owncloud-archive/user_management#192
• windows_network_drive - https://github.com/owncloud/windows_network_drive/pull/244
• workflow https://github.com/owncloud/workflow/pull/186

Ones to think about: (the end of the universe might come before all the tests finish ;) )

• encryption - encryption app test against latest tarball encryption#134
• files_primary_s3 - No acceptance tests (see issue Run core tests from apps against latest core release #625 )

Add missing apps if any, Remove apps if these tests are not relevant on any apps.

[phil-davis]

I updated the list to mention all apps that have acceptance tests.
If an app has acceptance tests, then they should be able to run against the latest core release.
(someone tell me why we would not do this for all these apps?)

[individual-it]

apps that run all core tests, like user_ldap and encryption should only run the app-specific tests with the latest release tarball

[phil-davis]

core 10.2.1 has been released. New PRs for this issue can test against 10.2.1
We may as well quickly go to the apps that have already been done and s/10.2.0/10.2.1/

[paurakhsharma]

@phil-davis Since there are no acceptance tests in files_primary_s3.
Should we run core tests against 10.2.1 in files_primery_s3

Edit: But It will take ages to run.

[phil-davis]

@phil-davis Since there are no acceptance tests in files_primary_s3.
Should we run core tests against 10.2.1 in files_primery_s3

Edit: But It will take ages to run.

The same really applies to user_ldap and encryption. For any app that "plugs in a back end" to ownCloud core, we should run the core acceptance tests with the app "plugged-in" to make sure that the core behavior still works. We want to know that it works for core master, stable10 and the supported release(s).

I will raise a separate issue because for that we need to discuss how and when to run all that - #625

[skshetry]

Closing this as all the apps been adjusted.