Skip to content

Latest commit

 

History

History
99 lines (67 loc) · 3.7 KB

README.md

File metadata and controls

99 lines (67 loc) · 3.7 KB

abcunit-cmip5-stats

Processing Framework for calculating temporal statistics from CMIP5 (using ABCUnit).

Overview

The name ABCunit corresponds to the 4 layers the workflow is split in to:

  1. A - all
  2. B - batch
  3. C - chunk
  4. The Unit

These scripts are an example of using the ABCUnit structure on CMIP5 data. This provides a repteable and efficient workflow.

The statistics that can be calculated are the maximum, minimum and mean.

The models, ensembles and variables available can be found in lib/defaults.py

Structure

  1. Run all script = run_all.py
  • Calculates given statistic for chosen models, ensembles and variables
  • These are given as arguments on the command line
  • Statistic must be specified as an argument at the command line
  • Defaults to running for all models, ensembles and variables
  • For each model it runs the batch script
  1. Run batch script = run_batch.py
  • Calculates the statistic over one chosen model and chosen ensembles and variables
  • Defaults to all ensembles and variables
  • Statistic and model must be specified when run at the command line
  • Calls each ensemble as an argument for the chunk script
  • The chunk script is submitted to LOTUS via bsub
  1. Run chunk script = run_chunk.py
  • Typically run on lotus but can be run from the command line
  • Calculates the statstic for the specified model, ensemble and variables
  • Statistic, model and ensemble must all be specified if run from the command line
  • Defaults to all variables
  • For each variable it:
    • Ignores if already calculated
    • Finds the necessary files
    • Checks the date range specified is valid
    • Calculates the statistic and writes to output file
    • Checks output file exists

Example usage

Log in to a JASMIN sci server:

ssh <user-id>@jasmin-sci5.ceda.ac.uk

Edit these files to match your setup:

  • SETTINGS.py
  • setup-env.sh

Clone this repositroy and make sure you are in the top level abcunit-cmip5-stats directory:

  • https://github.com/cedadev/abcunit-cmip5-stats.git
  • cd abcunit-cmip5-stats

First run the setup-env.sh script to setup up your environment.

Options for models, ensembles and variables can be found in lib/defaults.py. Note that the model is defined by its institute/model combination.

Running the top level 'run all' script at the command line:

python run_all.py -s mean

Running the 'run batch' script:

python run_batch.py -s mean -m MOHC/HadGEM2-ES

Running the 'run chunk' script locally, instead of using LOTUS, which is how it is invoked in the batch script:

python run_chunk.py -s mean -m MOHC/HadGEM2-ES -e r1i1p1

In each example, all other arguments are optional and can be included. For example, to calculate the mean of only 2 variables for in one model and ensemble:

python run_chunk.py -s mean -m MOHC/HadGEM2-ES -e r1i1p1 -v rh ra

Outputs

The outputs - success files, failure files and resulting netCDF file if the job is successful are stored in directories with the following structure:

current-directory/ALL_OUTPUTS/output-type/stat/model/ensemble/var_id.nc(.txt)

  • current-directory is the directory you are in when running the python scripts.
  • output-type can be one of:
    • outputs - The netCDF file corresponding to the chosen statistic.
    • success - Empty file produced when the job is successful.
    • bad_data - Empty file produced when no netCDF files could be found for the chosen arguments.
    • bad_num - Empty file produced when the chosen date range is invalid for the chosen files.
    • no_output - Empty file produced when the output file could not be generated.

The lotus outputs follow a similar pattern:

current-directory/ALL_OUTPUTS/output-type/stat/model/ensemble.out or ensemble.err