ATS Regression Test Suite

This is the regression test suite for ATS.

The full set of these runs should always work, and should always be run and, if necessary, updated prior to issuing a pull request.

Quickstart

A basic python environment is needed -- for full details see the ATS python environment installation in the ATS Users Guide (work in progress, link coming!). In the near term, a python3 installation that includes numpy, matplotlib, and h5py is likely sufficient.

To run the complete set of tests:

python regression_tests.py .

See also:

python regression_tests.py --help

Creating new tests

Follow the naming convention: test_category-test_name where test_category is the subdirectory name.

• Add the xml file to the repo.
• Add the xml to a .cfg file in that subdirectory.
• Make sure that checkpoints are being written at sane times, and add "checkpoint=True" to xml files on evaluators whose results you wish to compare.
• Run the test.
  • If the test is sensitive to timestep size history (e.g. it is a variable timestep history problem) then use the strategy: python regression_tests.py -n --save-dt-history path_to.cfg -t my_category-my_name. This runs your test twice -- once to get a sane timestep size history, then parses that, saves it to disk, writes a new input file that uses that exact sequence of step sizes, and runs a second time ensure that the results work with that timestep size history. This is a huge help for reproducibility and removing false negatives -- the solution can be quite sensitive to timestep size history, but rarely do we want to flag when a non-bitwise repeatable change randomly causes a solver to take one more timestep, thereby changing the answer.
  • If the test can be a fixed timestep size run, then just python regression_tests.py -n path_to.cfg -t my_category-my_name is sufficient.
  • Finally, after you've added the test, run once more to ensure that tolerances are good and the tests pass with what should be a bitwise identical run.
• Clean out the resulting gold directory: ./clean_gold.sh XX_my_category/my_category-my_name.regression.gold to make sure that only the checkpoint files and the ats_version.txt files are left.
• Commit and push.

Updating tests:

• Update xml files, etc
• run the update: `python regression_tests.py -u path_to.cfg -t "my_category-my_name"
• clean out the resulting gold directory: ./clean_gold.sh XX_my_category/my_category-my_name.regression.gold to make sure that only the checkpoint files and the ats_version.txt files are left.
• commit and push

Note that any non-bitwise reproducible result should likely get updated. It is extremely useful to be able to rely on bitwise reproducibility (when it is available). So, if changes result in bitwise changes in the solution, prefer to first verify that the tests pass (bitwise deltas should still pass!) and only then update the tests using the above procedure. This way, if someone else makes a change that shouldn't affect bitwise reproducibility, they can assume safely that they should have it.

Obviously any changes that cause a test to fail should be checked extensively, confirmed that the change is acceptible (or better), and the document carefully in a commit message why the solution changed.

Test Index:

Note that missing links mean the test does not yet exist. Please feel free to help!

01_richards_steadystate

These tests solve Richards equation in steadystate form.

• fv A 1D column using finite volumes.
• mfd The same problem using mimetic finite differences.
• mfd-schur The same problem using MFD and a Schur complement preconditioner.

02_richards

Transient Richards equation tests. In particular BCs, including seepage face BCs, cause trouble.

• infiltration_fv A 1D column infiltration problem, using finite volumes.
• infiltration_mfd A 1D column infiltration problem using mimetic finite differences.
• seepage_infiltration_fv Infiltrate with a seepage face.
• seepage_infiltration_mfd Same as above but with MFD.
• seepage_drawdown_fv A 1D column with a flux out the bottom.
• seepage_drawdown_mfd Same as above but with MFD.
• seepage_drawdown_diag_tensor_fv A 1D column with a flux out the bottom with a diagonal, non-isotropic tensor.
• seepage_drawdown_diag_tensor_mfd Same as above with MFD.
• seepage_drawdown_full_tensor_fv A 1D column with a flux out the bottom with a full tensor permeability.
• seepage_drawdown_full_tensor_mfd Same as above with MFD.
• seepage_exfiltration_fv A 1D column exfiltration problem, using finite volumes.
• seepage_exfiltration_mfd A 1D column exfiltration problem using mimetic finite differences.
• infiltration_then_seepage_fv A 1D column that infiltrates, then a seepage face turns on.
• infiltration_then_seepage_mfd Same as above with MFD.

03_surface_water

Solve some form of diffusion wave equation. In particular BCs, including the plethora of outflow options, cause trouble.

• rainfall_fv A simple rainfall problem, with finite volumes.
• rainfall_fv_jac The same problem, with Jacobian terms.
• rainfall_fv_simplified The same problem, but with constant, non-temperature-based densities and viscosities.
• bc_critical_depth Tests the critical depth boundary condition with an injection problem.
• bc_max_head Tests the max head boundary condition with an injection problem.
• bc_zero_gradient Tests the zero gradient boundary condition with an injection problem.
• uphill_head Part of a series of problems dealing with transient BCs and flow causality causing difficulties in upwinding.
• uphill_piecewise_head Part of a series of problems dealing with transient BCs and flow causality causing difficulties in upwinding.
• downhill_head Part of a series of problems dealing with transient BCs and flow causality causing difficulties in upwinding.
• downhill_piecewise_head Part of a series of problems dealing with transient BCs and flow causality causing difficulties in upwinding.
• subgrid_microtopography An example of the model for representing subgrid microtopography in surface flow.

04_integrated_hydro

Coupled flow on the surface and subsurface

• column_infiltration A 1D column, initially unsaturated, rained on. Very similar to column_sat.
• column_exfiltration A 1D column, initially unsaturated, with a flux in from the bottom until water expresses at the surface.
• column_drainage A 1D column, initially saturated with surface water, with a flux out of the bottom until the surface is dry.
• column_inf A 1D column, infiltration limited runoff generation.
• column_sat A 1D column, saturation limited runoff generation.
• column_inf A 1D column, infiltration limited runoff generation.
• hillslope_sat A 2D hillslope, saturation limited runoff generation.
• hillslope_inf A 2D hillslope, infiltration limited runoff generation.
• hillslope_sat-np2 A 2D hillslope, saturation limited runoff generation, on two cores.
• hillslope_inf-np2 A 2D hillslope, infiltration limited runoff generation, on two cores.

05_surface_balance

Surface energy balance equations that somehow calculate evaporation or transpiration or both.

• general A very simple balance ODE, solving exponential decay, to test general surface balance PKs and time integration.
• lingzhang A simple test of the Ling & Zhang based surface energy balance model used in Arctic/Ecohydrology runs.
• priestly_taylor A simple test of the Priestly-Taylor potential ET model.

06_transport

Surface and subsurface transport of nonreactive species.

• surface_tracer A single tracer in a 1D reach
• surface_tracer_logical A single tracer on a logical mesh
• subsurface_tracer A single tracer in a 2D transect.
• column_infiltration Integrated transport on a single column of cells; an infiltration problem where the new water includes the tracer.
• column_exfiltration Integrated transport on a single column of cells; an exfiltration problem where water pushes up, making sure tracer ends up in the surface water.
• column_drainage Integrated transport on a single column of cells; a drainage problem where the water table draws down.
• column_infiltration_hot Same as above, but with a higher order transport discretization.
• column_exfiltration_hot Same as above, but with a higher order transport discretization.
• column_drainage_hot Same as above, but with a higher order transport discretization.

07_reactive_transport

Surface and subsurface reactive transport

• surface_decay_ingrowth A two-species system where the first species decays into the second (daughter) species. A radioactive decay problem.
• surface_decay_ingrowth_logical The same problem on a logical mesh.
• subsurface_decay_ingrowth The same problem on a 2D transect.
• integrated_decay_ingrowth The same problem on surface + subsurface.
• integrated_decay_ingrowth_coupled The same integrated system but with a coupled flow solution.
• surface_denitrification Dual-monod kinetics reaction system for aerobic respiration and DOM denitrification. 6 primary species and reactions.
• surface_denitrification_logical The same problem on a logical mesh.
• subsurface_denitrification The same problem on a 2D transect.
• integrated_denitrification The same problem on surface + subsurface.
• integrated_denitrification_coupled The same integrated system but with a coupled flow solution.
• subsurface_sorption A single species subsurface transport with equilibrium sorption in a generated mesh
• subsurface_sorption_logical Reactive transport (first order transformation taking place only in the aqueous phase) in subsurface where parent and daughter species have different sorption properties
• dilution_test A single tracer injection using "water_source" and geochemical condition (PFLOTRAN) to test units of effective solute injection rate
• column_infiltration_alquimia_tracer Integrated transport on a single column of cells; an infiltration problem where the new water includes the tracer using the alquimia and pflotran.

08_energy

A set of energy equations building toward freeze-thaw problems.

• surface_water A sheet of ice gets thawed on the surface.
• freezeup Freezes a column of water from below.
• permafrost A typical full permafrost problem.
• snow_distribution Some form of 2D snow distribution test.

09_multiscale_models

A hodgepodge of subgrid models, including column-based 2.5D problems and more.

• columnar_permafrost Four-column example of 1D subsurface columns + 2D surface system on an energy+flow system.
• columnar_permafrost_subgrid Four-column example of 1D subsruface columns + 2D subgrid surface system on an energy+flow system with seb subgrid (soil, water, ice) model.
• stream_hyporheic_conservative_tracer_reach Single reach conservative tracer example of the ADELS model using Alquimia.
• stream_hyporheic_sorption Single reach tracer undergoing sorption in the sub-surface represented at subgrid level (ADELS model + Alquimia).
• stream_hyporheic_denitrification_reach Single reach denitrification example (ADELS model + Alquimia) with single-subgrid hyporheic zone.
• stream_hyporheic_denitrification_reach_multisubgrid Single reach denitrification example (ADELS model + Alquimia) with multisubgrid hyporheic zones.
• stream_hyporheic_denitrification_watershed Watershed scale denitrification example (ADELS model + Alquimia) with single-subgrid hyporheic zones.