Enhanced time resolution handling (subday and overlapping summary periods) #933

SpacemanPaul · 2023-03-09T06:26:55Z

Major refactor of time resolution handling.

There are now only 3 time resolution options:

solar
Replaces the old "raw" option - local-solar-day time model, used for imagery captured in daily swathes by satellites in heliosynchronous orbits.
summary
Replaces the old day, month, and year options. Only looks at the "start" datetime, and so neatly supports products with
overlapping or non-exclusive dataset date-ranges. Expects the time portion of the start date to always be "midnight UTC". Used for summary/calculated products.
subday
New option.
Used for for products with multiple time values per day (e.g. hourly/minutely data). Uses the "start" datetime of the dataset.

Note that the solar and summary options explicitly ignore the time component of the time parameter passed by the user. If you need the time component to be significant, you must use subday. The subday option closes #876

This ended up being a quite extensive PR requiring a significant overhaul of test frameworks which in turn uncovered several long-standing bugs at the intersection of various OWS features - which this PR also fixes.

I am now fairly confident that this PR introduces no major regressions on previously existing functionality.

📚 Documentation preview 📚: https://datacube-ows--933.org.readthedocs.build/en/933/

…pt at handling overlaps.

…ing.

for more information, see https://pre-commit.ci

…pand_time_resolutions # Conflicts: # tests/test_time_res_method.py

codecov · 2023-03-09T06:41:44Z

Codecov Report

Merging #933 (64e294f) into master (3ee0c2a) will decrease coverage by 0.27%.
The diff coverage is 90.60%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #933      +/-   ##
==========================================
- Coverage   94.00%   93.73%   -0.27%     
==========================================
  Files          43       43              
  Lines        6385     6496     +111     
==========================================
+ Hits         6002     6089      +87     
- Misses        383      407      +24

Impacted Files	Coverage Δ
datacube_ows/wcs1_utils.py	`94.91% <75.00%> (-1.21%)`	⬇️
datacube_ows/wms_utils.py	`92.08% <76.47%> (-1.18%)`	⬇️
datacube_ows/data.py	`83.63% <84.00%> (-0.51%)`	⬇️
datacube_ows/product_ranges.py	`83.70% <87.75%> (+1.83%)`	⬆️
datacube_ows/mv_index.py	`96.70% <95.23%> (-0.60%)`	⬇️
datacube_ows/ows_configuration.py	`95.96% <98.24%> (+0.06%)`	⬆️
datacube_ows/update_ranges_impl.py	`87.70% <100.00%> (ø)`
datacube_ows/utils.py	`100.00% <100.00%> (ø)`
datacube_ows/wcs2.py	`92.38% <100.00%> (+0.14%)`	⬆️
datacube_ows/wcs2_utils.py	`91.70% <100.00%> (+0.33%)`	⬆️

... and 3 files with indirect coverage changes

robbibt · 2023-03-09T23:18:44Z

This looks super cool! Could new the "subday" support potentially support @mitchharley's coastal photography use case I mentioned a while back, i.e. potentially multiple georeferenced GeoTIFFs per day across time?

SpacemanPaul · 2023-03-10T03:13:05Z

This looks super cool! Could new the "subday" support potentially support @mitchharley's coastal photography use case I mentioned a while back, i.e. potentially multiple georeferenced GeoTIFFs per day across time?

Yes! #876 is the GH issue I referred to in that Twitter thread.

for more information, see https://pre-commit.ci

…pand_time_resolutions

for more information, see https://pre-commit.ci

…pand_time_resolutions

robbibt · 2023-03-17T05:19:00Z

Not so much a review as a question about how this may work for our upcoming DEA Intertidal product: do you think these changes will support the following scenario?

A multi-timestep product with overlapping 3 year median composites, e.g.:
- timestep 1: 2019, 2020, 2021 data
- timestep 2: 2020, 2021, 2022 data
- timestep 3: 2021, 2022, 2023 data etc
In OWS/Terria, we will probably want data to be labelled with the "middle" year for each composite, e.g.:
- 2020 timestep: 2019, 2020, 2021 data
- 2021 timestep: 2020, 2021, 2022 data
- 2022 timestep: 2021, 2022, 2023 data etc

pindge · 2023-03-17T05:26:25Z

datacube_ows/ows_configuration.py

-            return "solar_day"
-        else:
-            return group_by_statistical()
+        return self.time_resolution.dataset_groupby(is_mosaic=self.mosaic_date_func is not None)


i am surprised flake8 let this long line through

Must be just under the limit.

SpacemanPaul · 2023-03-17T05:58:13Z

* In OWS/Terria, we will probably want data to be labelled with the "middle" year for each composite, e.g.:
  
  * 2020 timestep: 2019, 2020, 2021 data
  * 2021 timestep: 2020, 2021, 2022 data
  * 2022 timestep: 2021, 2022, 2023 data etc

The implementation here is to label the timestep with the start date:

2019 timestep: 2019, 2020, 2021 data
2020 timestep: 2020, 2021, 2022 data
2021 timestep: 2021, 2022, 2023 data etc

This doesn't support what you want - but does make supporting it much easier (in future PRs).

SpacemanPaul and others added 8 commits March 9, 2023 17:18

Refactor of time_resolution. Sub_day time resolutions and first attem…

4f8285d

…pt at handling overlaps.

Refactor to simplify and support non-trivial overlaps. Get tests pass…

36246ed

…ing.

Update documentation.

3e287d2

Bugfixes, suppress SQLAlchemy deprecations.

d0b077d

[pre-commit.ci] auto fixes from pre-commit.com hooks

dd1b6d7

for more information, see https://pre-commit.ci

Add 'subday' to spellcheck word list.

277b447

flake8age.

d2efc79

Merge remote-tracking branch 'origin/expand_time_resolutions' into ex…

73667f2

…pand_time_resolutions # Conflicts: # tests/test_time_res_method.py

Summary time_resolution test data.

a0f3a94

SpacemanPaul and others added 18 commits March 10, 2023 14:44

Bugfix and test coverage.

c8e71b6

Fix grammar mistake in error message.

9432c14

Fix errors in test config.

01b9550

More bug fixes. Some corners of SQLAlchemy are poorly documented.

c032a78

[pre-commit.ci] auto fixes from pre-commit.com hooks

731a97d

for more information, see https://pre-commit.ci

flake8age

3054b96

[pre-commit.ci] auto fixes from pre-commit.com hooks

6c1b81a

for more information, see https://pre-commit.ci

Get subday working. Some cleanup around time matching.

74a640f

Refactor binary time search: recursion to loop.

10db69f

Merge remote-tracking branch 'origin/expand_time_resolutions' into ex…

1c90726

…pand_time_resolutions

[pre-commit.ci] auto fixes from pre-commit.com hooks

4d318ff

for more information, see https://pre-commit.ci

Test for binary date search method.

18bf4f2

Get subday time resolution working with WMS.

30f317c

Get subday time resolution working with WCSv1.

bba4b3f

Get subday time resolution working with WCSv2.

df4de28

[pre-commit.ci] auto fixes from pre-commit.com hooks

aa5ce30

for more information, see https://pre-commit.ci

Removed dead code and flake8age.

fc53a18

Fix user_date_sorter test.

3b44169

SpacemanPaul and others added 11 commits March 16, 2023 14:49

Fix WCS2 bug.

a4f4e09

Improving test coverage.

cb47fdf

More SQLAlchemy2 deprecations and tweak test (hopefully not time out)

dbdfe21

Increase timeout. Finishes promptly locally - why is GH timing out??

f92d828

Cleanup/refactor.

3bd07f2

[pre-commit.ci] auto fixes from pre-commit.com hooks

e335f27

for more information, see https://pre-commit.ci

Move regular time axis to a more appropriate layer in test config.

5f792a1

Merge remote-tracking branch 'origin/expand_time_resolutions' into ex…

2645b7d

…pand_time_resolutions

Remove unused import.

378ed69

Better test cases for coverage.

a48ab40

Fix dimension ordering bug in manual data stacking.

64e294f

SpacemanPaul marked this pull request as ready for review March 17, 2023 04:54

SpacemanPaul requested review from pindge, alexgleith and omad March 17, 2023 05:00

pindge approved these changes Mar 17, 2023

View reviewed changes

SpacemanPaul merged commit c5ecdea into master Mar 17, 2023

SpacemanPaul deleted the expand_time_resolutions branch March 17, 2023 05:47

SpacemanPaul mentioned this pull request Mar 20, 2023

Remove whitespace from WMS GetCaps date lists. #936

Merged

SpacemanPaul mentioned this pull request Jul 14, 2023

New Feature: date display mapping #945

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enhanced time resolution handling (subday and overlapping summary periods) #933

Enhanced time resolution handling (subday and overlapping summary periods) #933

SpacemanPaul commented Mar 9, 2023 •

edited

Loading

codecov bot commented Mar 9, 2023 •

edited

Loading

robbibt commented Mar 9, 2023

SpacemanPaul commented Mar 10, 2023

robbibt commented Mar 17, 2023 •

edited

Loading

pindge Mar 17, 2023

SpacemanPaul Mar 17, 2023

SpacemanPaul commented Mar 17, 2023

Enhanced time resolution handling (subday and overlapping summary periods) #933

Enhanced time resolution handling (subday and overlapping summary periods) #933

Conversation

SpacemanPaul commented Mar 9, 2023 • edited Loading

codecov bot commented Mar 9, 2023 • edited Loading

Codecov Report

robbibt commented Mar 9, 2023

SpacemanPaul commented Mar 10, 2023

robbibt commented Mar 17, 2023 • edited Loading

pindge Mar 17, 2023

Choose a reason for hiding this comment

SpacemanPaul Mar 17, 2023

Choose a reason for hiding this comment

SpacemanPaul commented Mar 17, 2023

SpacemanPaul commented Mar 9, 2023 •

edited

Loading

codecov bot commented Mar 9, 2023 •

edited

Loading

robbibt commented Mar 17, 2023 •

edited

Loading