Rearrange folders according to Python packaging best practices #250

yantosca · 2023-08-09T15:25:42Z

Name and institution

Bob Yantosca
Harvard + GCST

Overview

This PR (currently a draft) is the companion PR to #249. We propose rearranging the GCPy folder structure from the current structure:

gcpy
    |
    gcpy
    |
    benchmark
    |       |
    |      modules
    |
    examples
            |
            several  other subdirectories not listed

to:

gcpy
    |
    gcpy
        |
        benchmark
        |       |
        |       modules
        |
        examples
               |
               several other subdirectories not listed

Related issues

Closes Reorder directories to better conform to Python package standards #249

NOTES

This is currently a draft PR and should not be merged. Any feedback would be appreciated. Tagging @lizziel @msulprizio @yidant @laestrada @SaptSinha

We have now arranged the GCPy folder structure so that: (1) gcpy/gcpy --> gcpy/src/gcpy (2) gcpy/benchmark --> gcpy/src/benchmark (3) gcpy/benchmark/modules --> gcpy/src/gcpy/benchmark/modules (4) gcpy/examples --> gcpy/src/gcpy/examples (5) gcpy/examples/* --> gcpy/src/gcpy/examples/* Also note, the file gcpy/benchmark has now been renamed to gcpy/benchmark_funcs.py, in order to break a namespace collision with the new gcpy/benchmark folder. Nothing else other than the move has been changed. Updated the CHANGELOG.md accordingly. Will test to make sure that benchmark plots are generated properly. TODO: Move benchmark.py and related config files out of gcpy/src/gcpy and into gcpy/src/benchmark TODO: Break up gcpy/src/benchmark_funcs.py into smaller modules, arranged by purpose. For example, group all the functions that print out emissions and mass tables into a single module, etc. Signed-off-by: Bob Yantosca <[email protected]>

We have now moved gcpy/src/gcpy to gcpy/gcpy, which seems to avoid some issues with importing modules. You can now set your PYTHONPATH to the top-level folder: export PYTHONPATH=/path/to/gcpy and then you should be able to import gcpy from gcpy import benchmark Signed-off-by: Bob Yantosca <[email protected]>

We have added __init__.py files in the various subfolders of python/gcpy/gcpy to facilitate importing everything with an "import gcpy" statement. NOTE: To avoid namespace collisions, we have renamed: examples/xarray --> examples/xarray_examples examples/yaml --> examples/yaml_examples Signed-off-by: Bob Yantosca <[email protected]>

gcpy/examples/*.py - Move code to a routine protected by if __name__ == "__main__", which will prevent code from being executed upon import - Now use "from gcpy import ___" where expedient - Now use plt.cm.get_cmap("viridis") instead of plt.cm.viridis - Added code updates suggested by Pylint such as - Make variable names conform to snake_case - Remove extraneous imports - Trim trailing whitespace - Use "except ___ as exc:" and "raise ___ from exc" CHANGELOG.md - Updated accordingly Signed-off-by: Bob Yantosca <[email protected]>

gcpy/benchmark/run_benchmark.py - In the if __name__ == "__main__" block call main(sys.argv), which will pass command-line arguments to main - Replace main() with main(argv), which accepts sys.argv from the command line Signed-off-by: Bob Yantosca <[email protected]>

gcpy/examples/compare_diags.py - In the if __name__ == "__main__" block call main(sys.argv), which will pass command-line arguments to main - Replace main() with main(argv), which accepts sys.argv from the command line CHANGELOG.md - Updated accordingly Signed-off-by: Bob Yantosca <[email protected]>

yantosca · 2023-08-09T21:34:04Z

I have made the following updates:

The PYTHONPATH variable can now be set in your .bashrc as follows:
```
export PYTHONPATH=/path/to/gcpy
```
Moved gcpy/benchmark to gcpy/gcpy/benchmark
Moved gcpy/examples to gcpy/gcpy/examples.
Added __init__.py files in subdirectories of gcpy/gcpy.
Make sure scripts are protected with an if __name__ == "__main__": block to prevent routines from executing upon import.

The gcpy/benchmark/run_benchmark.py script can now be invoked with:

from gcpy.benchmark import run_benchmark
run_benchmark.main("1yr_fullchem_benchmark.yml")

The gcpy/examples/diagnostics/compare_diags.py script can now be invoked with:
```
from gcpy.examples import compare_diags
compare_diags.main("input.yml")
```

The yaml_examples only contained species2wiki.py, which is now obsolete now that we do not use the wiki to display species database info. Removed gcpy/examples/yaml_examples and updated the gcpy/examples/__init__.py folder accordingly. Signed-off-by: Bob Yantosca <[email protected]>

gcpy/examples/plotting/plot_comparisons.py gcpy/examples/plotting/plot_single_panel.py gcpy/examples/timeseries/mda8_o3_timeseries.py gcpy/examples/working_with_files/*.py - Add a main() function (or rename an existing function to main) so that these can be callable with .main() Signed-off-by: Bob Yantosca <[email protected]>

gcpy/benchmark_modules/modules/run_1yr_fullchem_benchmark.py gcpy/benchmark_modules/modules/run_1yr_tt_benchmark.py - Changed permission chmod 755 to chmod 644 (since these are now not usually run directly but through benchmark/run_benchmark.py) Signed-off-by: Bob Yantosca <[email protected]>

CHANGELOG.md - Added blurb about gcpy/benchmark/modules*.py scripts permissions being changed to chmod 644 Signed-off-by: Bob Yantosca <[email protected]>

@yantosca

This merge brings updates from PR #251 (Update gcpy_env environment to install with MambaForge and to use latest package versions; Update scripts accordingly, by @yantosca) into the feature/move-folders branch. This will facilitate merging into dev at a later time. Signed-off-by: Bob Yantosca <[email protected]>

yantosca · 2023-08-17T20:28:55Z

I have merged the updates from PR #251 (now in the dev branch) into the feature/move-folders branch. This will allow us to develop on top of PR #251.

yantosca · 2023-08-17T21:16:51Z

Note. this should be merged after PR #246

yantosca · 2023-08-28T21:29:16Z

PR https://github.com/geoschem/gc-cloud-infrastructure/pull/51 should be merged concurrently with this PR, in order to avoid breaking the cloud benchmarking workflow.

msulprizio · 2023-08-30T12:13:13Z

CHANGELOG.md

@@ -39,6 +41,12 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Routine `print_totals` now prints small and/or large numbers in scientific notation
 - Truncate names in benchmark & emissions tables to improve readability
 - Add TransportTracers species names to `gcpy/emissions_*.yml` files
+- Folder `gcpy/gcpy` is now  `gcpy/src/gcpy`, for adherence to Python packaging standards


Is this still the case or are we not using the src directory?

Nice catch @msulprizio. I'll edit the changelog. If we use gcpy/src then we'd have to add path/to/gcpy/src to PYTHONPATH, which is a little ugly.

Now removed in commit cf6fc12

msulprizio · 2023-08-30T12:15:26Z

gcpy/benchmark/modules/run_1yr_fullchem_benchmark.py

@@ -388,7 +388,7 @@ def run_benchmark(config, bmk_year_ref, bmk_year_dev):

            # Create plots
            print("\nCreating plots for annual mean")
-            bmk.make_benchmark_conc_plots(
+            bmkf.make_benchmark_conc_plots(


I think this is a typo - bmk is still used above

Thanks @msulprizio. I'll fix this.

Now fixed in commit cf6fc12

gcpy/gcpy/benchmark/modules/run_1yr_fullchem_benchmark.py - Fix typo, "bmkf" -> "bmk" in call to make_benchmark_conc_plots CHANGELOG.md - Removed note about gcpy/src/gcpy, this is no longer correct. Signed-off-by: Bob Yantosca <[email protected]>

yantosca · 2023-08-30T18:59:58Z

@msulprizio, I will merge this locally and make sure I can plot benchmarks before pushing.

yantosca added category: Feature Request New feature or request topic: Structural Modifications Related to GCPy structural modifications (as opposed to scientific updates) labels Aug 9, 2023

yantosca self-assigned this Aug 9, 2023

yantosca added 5 commits August 9, 2023 14:54

yantosca added 5 commits August 10, 2023 10:44

Updated CHANGELOG.md w/ info about file permissions

cd95d39

CHANGELOG.md - Added blurb about gcpy/benchmark/modules*.py scripts permissions being changed to chmod 644 Signed-off-by: Bob Yantosca <[email protected]>

yantosca mentioned this pull request Aug 17, 2023

Now allow specification of the number of cores for benchmark plotting jobs #246

Merged

yantosca marked this pull request as ready for review August 17, 2023 21:23

yantosca requested review from lizziel and msulprizio August 17, 2023 21:23

yantosca added this to the 1.4.0 milestone Aug 17, 2023

yantosca linked an issue Aug 28, 2023 that may be closed by this pull request

[FEATURE REQUEST] Move gcpy/benchmark.py to the benchmark/ folder #245

Closed

msulprizio requested changes Aug 30, 2023

View reviewed changes

yantosca requested a review from msulprizio August 30, 2023 18:59

msulprizio approved these changes Aug 30, 2023

View reviewed changes

yantosca merged commit 04bc173 into dev Aug 31, 2023

yantosca deleted the feature/move-folders branch August 31, 2023 14:23

This was referenced Aug 31, 2023

[FEATURE REQUEST] Move gcpy/benchmark.py to the benchmark/ folder #245

Closed

Reorder directories to better conform to Python package standards #249

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rearrange folders according to Python packaging best practices #250

Rearrange folders according to Python packaging best practices #250

yantosca commented Aug 9, 2023 •

edited

Loading

yantosca commented Aug 9, 2023 •

edited

Loading

yantosca commented Aug 17, 2023

yantosca commented Aug 17, 2023

yantosca commented Aug 28, 2023

msulprizio Aug 30, 2023

yantosca Aug 30, 2023

yantosca Aug 30, 2023

msulprizio Aug 30, 2023

yantosca Aug 30, 2023

yantosca Aug 30, 2023

yantosca commented Aug 30, 2023

Rearrange folders according to Python packaging best practices #250

Rearrange folders according to Python packaging best practices #250

Conversation

yantosca commented Aug 9, 2023 • edited Loading

Name and institution

Overview

Related issues

NOTES

yantosca commented Aug 9, 2023 • edited Loading

yantosca commented Aug 17, 2023

yantosca commented Aug 17, 2023

yantosca commented Aug 28, 2023

msulprizio Aug 30, 2023

Choose a reason for hiding this comment

yantosca Aug 30, 2023

Choose a reason for hiding this comment

yantosca Aug 30, 2023

Choose a reason for hiding this comment

msulprizio Aug 30, 2023

Choose a reason for hiding this comment

yantosca Aug 30, 2023

Choose a reason for hiding this comment

yantosca Aug 30, 2023

Choose a reason for hiding this comment

yantosca commented Aug 30, 2023

yantosca commented Aug 9, 2023 •

edited

Loading

yantosca commented Aug 9, 2023 •

edited

Loading