Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Version 0.2.0 #62

Merged
merged 84 commits into from
Mar 1, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
84 commits
Select commit Hold shift + click to select a range
a8dcd00
Configuration refactor.
Flegyas Jan 5, 2022
b83a495
New working example.
Flegyas Jan 5, 2022
9cc98d1
Merge pull request #8 from lucmos/refactor/structure
lucmos Jan 6, 2022
eaf56bb
Refactor project to a python package (#10)
lucmos Jan 7, 2022
af2065b
Add tooling configuration (#9)
lucmos Jan 7, 2022
9bfe399
Refactor codebase to be compliant to the pre-commits (#11)
lucmos Jan 7, 2022
9c21467
Refactor the project root management (#12)
lucmos Jan 7, 2022
d0f023c
Added wandb to .gitignore (#14)
Flegyas Jan 7, 2022
b629434
Refactor logging (#15)
lucmos Jan 8, 2022
9e413c2
Enable pin-memory if not on CPU (#16)
lucmos Jan 8, 2022
36038da
Factor our PyTorch Module from the Lightning Module (#17)
lucmos Jan 8, 2022
0faeeb0
Force the .cache folder to be in the PROJECT_ROOT (#19)
lucmos Jan 8, 2022
0667469
Add the configuration to the Lightning checkpoints (#20)
lucmos Jan 8, 2022
8b13b07
Use extend-ignore instead of ignore in .flake8 (#21)
lucmos Jan 8, 2022
acf9120
Fix formatting (#22)
lucmos Jan 8, 2022
ebe07f8
Log the code used in the current experiment to wandb (#18)
lucmos Jan 8, 2022
7b81f0d
Functionalities decoupling via external library (nn-core).
Flegyas Jan 8, 2022
40996e7
Env variable handling decoupled to nn-core.
Flegyas Jan 8, 2022
b887a3b
Updated .gitignore
Flegyas Jan 8, 2022
399d4e8
Setup storage_dir and NNLogger configured via callback.
Flegyas Jan 8, 2022
b5bb5f2
Updated requirements according to the pypi upload: nn-template-core
Flegyas Jan 8, 2022
fdd544f
Move logger disabling to nn-template-core
lucmos Jan 8, 2022
90bd238
Merge pull request #23 from lucmos/feature/nn-core
lucmos Jan 9, 2022
560314f
Add tests (#24)
lucmos Jan 9, 2022
78b3d6c
Implement resuming behaviour (#25)
lucmos Jan 10, 2022
d2759f5
Refactor NNLogger usages (#27)
lucmos Jan 11, 2022
db88bbb
Add CI on pre-commits and tests (#26)
lucmos Jan 11, 2022
eef4e82
Remove some trigger from the Test Suite workflow
lucmos Jan 11, 2022
39af6db
Merge pull request #28 from lucmos/feature/ci-trigger
lucmos Jan 11, 2022
1871afb
Overwrite Lightning logging configuration
lucmos Jan 12, 2022
926284f
Merge pull request #29 from lucmos/feature/logging
Flegyas Jan 13, 2022
6cc6302
Ensure tags are defined asking interactively for them (#30)
lucmos Jan 16, 2022
127e6e2
Introduce the seed index concept (#31)
lucmos Jan 16, 2022
f4dd42f
Force execution of __init__.py on direct execution (#33)
lucmos Jan 16, 2022
3004067
Move functions from template to core (#34)
lucmos Jan 21, 2022
cd1c990
Add functionality to upload the run files in the storage to wandb (#35)
lucmos Jan 21, 2022
dba7653
Move ui_utils entirely to nn-core (#36)
lucmos Jan 21, 2022
709482e
Reorganize into a cookiecutter template
Flegyas Jan 21, 2022
e3d1cbd
Parametrize with cookiecutter
Flegyas Jan 21, 2022
d19d7b2
Add .gitignore in cookiecutter project
lucmos Jan 21, 2022
0059196
Add LICENSE
lucmos Jan 21, 2022
f957ec6
Add parametrized docs
lucmos Jan 22, 2022
1bd3fc9
Parametrize LICENSE
lucmos Jan 22, 2022
3117eb2
Minor improvements
lucmos Jan 22, 2022
387ae96
Add interactive environment setup
lucmos Jan 22, 2022
af7a01b
Parametrize nn-template badge version
lucmos Jan 22, 2022
8b045ba
Disable use_docs question until implemented
lucmos Jan 22, 2022
b502a19
Draft CI/CD for tests
lucmos Jan 22, 2022
202eb82
Add a reminder about PyCharm's setup after the init
Flegyas Jan 23, 2022
048ca70
Add a comment to .env.template explaining its usefulness
Flegyas Jan 23, 2022
dedf427
Change black version in pre-commit as it is now out of beta
Flegyas Jan 31, 2022
ba777f1
Fix minor typos
Flegyas Jan 31, 2022
4ac9856
Implement zip-based resume behavior
Flegyas Feb 1, 2022
3636977
Change cache version
Flegyas Feb 1, 2022
c0d316e
Merge pull request #37 from lucmos/feature/cookiecutter
Flegyas Feb 1, 2022
cd0c186
Fix setuptools conflict in the CI/CD (#38)
lucmos Feb 2, 2022
7de86c3
Remove requirements.txt not necessary
lucmos Feb 2, 2022
4e4315a
Fix load model in the streamlit UI (#40)
lucmos Feb 2, 2022
1036774
Pre initialize LICENSE correctly
lucmos Feb 2, 2022
2921a28
Merge pull request #43 from grok-ai/feauture/remove-requirements
Flegyas Feb 3, 2022
881da88
Merge pull request #41 from grok-ai/feature/link-license
Flegyas Feb 3, 2022
cd5167b
Add missing docstrings (#39)
lucmos Feb 3, 2022
76714de
Update README (#42)
lucmos Feb 3, 2022
ef16882
Add documentation (#13)
lucmos Feb 3, 2022
a7e398e
Add dynamic parametrized badges for the Test Suite and docs (#45)
lucmos Feb 3, 2022
b95bc45
Fix files hashing in workflow cache keys (#46)
lucmos Feb 3, 2022
de15814
Add seed_index determinism test (#44)
lucmos Feb 3, 2022
9fed921
Refactor references to organization name into grok-ai (#48)
lucmos Feb 3, 2022
cdc4fd7
Push the default version in mike (#49)
lucmos Feb 3, 2022
4d94534
Improve docs status badge to monitor the github-pages environment (#50)
lucmos Feb 3, 2022
20fbd3c
Fix mike rebasing and pushing logic on release (#51)
lucmos Feb 3, 2022
21c34b1
Add a DAG in the post hook interactive setup (#47)
lucmos Feb 6, 2022
36d88e5
Skip test if no dataset is provided (#52)
Flegyas Feb 14, 2022
052aae5
Fix remote parametrization in the README (#53)
lucmos Feb 15, 2022
41ff1e9
Fix type hint in dataset.py (#55)
lucmos Feb 15, 2022
1e83c56
Improve the "add git remote" message in the post hook (#54)
lucmos Feb 15, 2022
a4157f6
Update nn-template-core dependency to 0.0.7 (#56)
lucmos Feb 23, 2022
3a125e4
Update docs (#57)
lucmos Mar 1, 2022
da78b1a
Add custom collate function (#58)
Flegyas Mar 1, 2022
faed0fa
Set metadata as a cached property in DataModule (#59)
Flegyas Mar 1, 2022
0fd94fe
Pass run tags to the WandbLogger (#60)
Flegyas Mar 1, 2022
a72a8bc
Highlight example code blocks
Flegyas Mar 1, 2022
22397b5
Bump nn-core version (#61)
Flegyas Mar 1, 2022
e8cad1e
Bump version
Flegyas Mar 1, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 0 additions & 4 deletions .env.template

This file was deleted.

50 changes: 50 additions & 0 deletions .github/workflows/publish_docs.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
name: Publish docs

on:
release:
types:
- created

jobs:
build:
strategy:
fail-fast: false
matrix:
python-version: ['3.9']
include:
- os: ubuntu-20.04
label: linux-64
prefix: /usr/share/miniconda3/envs/

name: ${{ matrix.label }}-py${{ matrix.python-version }}
runs-on: ${{ matrix.os }}

steps:
- uses: actions/checkout@v2
with:
fetch-depth: 0

- name: Set up Python
uses: actions/setup-python@v1

- name: Install Dependencies
shell: bash -l {0}
run: |
pip install cookiecutter mkdocs mkdocs-material mike

# extract the first two digits from the release note
- name: Set release notes tag
run: |
export RELEASE_TAG_VERSION=${{ github.event.release.tag_name }}
echo "RELEASE_TAG_VERSION=${RELEASE_TAG_VERSION%.*}">> $GITHUB_ENV

- name: Echo release notes tag
run: |
echo "${RELEASE_TAG_VERSION}"

- name: Build docs website
shell: bash -l {0}
run: |
git config user.name ci-bot
git config user.email [email protected]
mike deploy --push --rebase --update-aliases ${RELEASE_TAG_VERSION} latest
195 changes: 195 additions & 0 deletions .github/workflows/test_suite.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,195 @@
name: Test Suite

on:
push:
branches:
- main
- develop

pull_request:
types:
- opened
- reopened
- synchronize

env:
CACHE_NUMBER: 3 # increase to reset cache manually
CONDA_ENV_FILE: 'env.yaml'
CONDA_ENV_NAME: 'project-test'
COOKIECUTTER_PROJECT_NAME: 'project-test'

jobs:
build:

strategy:
fail-fast: false
matrix:
python-version: ['3.8', '3.9']
include:
- os: ubuntu-20.04
label: linux-64
prefix: /usr/share/miniconda3/envs/

# - os: macos-latest
# label: osx-64
# prefix: /Users/runner/miniconda3/envs/$CONDA_ENV_NAME

# - os: windows-latest
# label: win-64
# prefix: C:\Miniconda3\envs\$CONDA_ENV_NAME

name: ${{ matrix.label }}-py${{ matrix.python-version }}
runs-on: ${{ matrix.os }}

steps:
- name: Parametrize conda env name
run: echo "PY_CONDA_ENV_NAME=${{ env.CONDA_ENV_NAME }}-${{ matrix.python-version }}" >> $GITHUB_ENV
- name: echo conda env name
run: echo ${{ env.PY_CONDA_ENV_NAME }}

- name: Parametrize conda prefix
run: echo "PY_PREFIX=${{ matrix.prefix }}${{ env.PY_CONDA_ENV_NAME }}" >> $GITHUB_ENV
- name: echo conda prefix
run: echo ${{ env.PY_PREFIX }}

- name: Define generated project files paths
run: |
echo "PROJECT_SETUPCFG_FILE=${{ env.COOKIECUTTER_PROJECT_NAME }}/setup.cfg" >> $GITHUB_ENV
echo "PROJECT_CONDAENV_FILE=${{ env.COOKIECUTTER_PROJECT_NAME }}/${{ env.CONDA_ENV_FILE }}" >> $GITHUB_ENV
echo "PROJECT_PRECOMMIT_FILE=${{ env.COOKIECUTTER_PROJECT_NAME }}/.pre-commit-config.yaml" >> $GITHUB_ENV

- uses: actions/checkout@v2

# COOKIECUTTER GENERATION
- name: Set up Python
uses: actions/setup-python@v1

- name: Install Dependencies
shell: bash -l {0}
run: |
pip install cookiecutter

- name: Generate Repo
shell: bash -l {0}
run: |
echo -e 'n\nn\nn\n' | cookiecutter . --no-input project_name=${{ env.COOKIECUTTER_PROJECT_NAME }}

- name: Init git into generated repo
shell: bash -l {0}
run: |
git config --global user.name ci-bot
git config --global user.email [email protected]
git init
git add --all
git commit -m "Initial commit"
working-directory: ${{ env.COOKIECUTTER_PROJECT_NAME }}

- name: Define cache key postfix
run: |
echo "CACHE_KEY_POSTFIX=${{ matrix.label }}-${{ matrix.python-version }}-${{ env.CACHE_NUMBER }}-${{ env.PY_CONDA_ENV_NAME }}-${{ hashFiles(env.PROJECT_CONDAENV_FILE) }}-${{ hashFiles(env.PROJECT_SETUPCFG_FILE) }}" >> $GITHUB_ENV

- name: Echo cache keys
run: |
echo ${{ env.PROJECT_SETUPCFG_FILE }}
echo ${{ env.PROJECT_CONDAENV_FILE }}
echo ${{ env.PROJECT_PRECOMMIT_FILE }}
echo ${{ env.CACHE_KEY_POSTFIX }}

# GENERATED PROJECT CI/CD

# Remove the python version pin from the env.yml which could be inconsistent
- name: Remove explicit python version from the environment
shell: bash -l {0}
run: |
sed -Ei '/^\s*-?\s*python\s*([#=].*)?$/d' ${{ env.CONDA_ENV_FILE }}
cat ${{ env.CONDA_ENV_FILE }}
working-directory: ${{ env.COOKIECUTTER_PROJECT_NAME }}

- name: Setup Mambaforge
uses: conda-incubator/setup-miniconda@v2
with:
miniforge-variant: Mambaforge
miniforge-version: latest
activate-environment: ${{ env.PY_CONDA_ENV_NAME }}
python-version: ${{ matrix.python-version }}
use-mamba: true

- uses: actions/cache@v2
name: Conda cache
with:
path: ${{ env.PY_PREFIX }}
key: conda-${{ env.CACHE_KEY_POSTFIX }}
id: conda_cache

- uses: actions/cache@v2
name: Pip cache
with:
path: ~/.cache/pip
key: pip-${{ env.CACHE_KEY_POSTFIX }}

- uses: actions/cache@v2
name: Pre-commit cache
with:
path: ~/.cache/pre-commit
key: pre-commit-${{ hashFiles(env.PROJECT_PRECOMMIT_FILE) }}-conda-${{ env.CACHE_KEY_POSTFIX }}

# Ensure the hack for the python version worked
- name: Ensure we have the right Python
shell: bash -l {0}
run: |
echo "Installed Python: $(python --version)"
echo "Expected: ${{ matrix.python-version }}"
python --version | grep "Python ${{ matrix.python-version }}"
working-directory: ${{ env.COOKIECUTTER_PROJECT_NAME }}

# https://stackoverflow.com/questions/70520120/attributeerror-module-setuptools-distutils-has-no-attribute-version
# https://github.com/pytorch/pytorch/pull/69904
- name: Downgrade setuptools due to a but in PyTorch 1.10.1
shell: bash -l {0}
run: |
pip install setuptools==59.5.0 --upgrade
working-directory: ${{ env.COOKIECUTTER_PROJECT_NAME }}

- name: Update conda environment
run: mamba env update -n ${{ env.PY_CONDA_ENV_NAME }} -f ${{ env.CONDA_ENV_FILE }}
if: steps.conda_cache.outputs.cache-hit != 'true'
working-directory: ${{ env.COOKIECUTTER_PROJECT_NAME }}

# Update pip env whether or not there was a conda cache hit
- name: Update pip environment
shell: bash -l {0}
run: pip install -e ".[dev]"
if: steps.conda_cache.outputs.cache-hit == 'true'
working-directory: ${{ env.COOKIECUTTER_PROJECT_NAME }}

- run: pip3 list
shell: bash -l {0}
working-directory: ${{ env.COOKIECUTTER_PROJECT_NAME }}

- run: mamba info
working-directory: ${{ env.COOKIECUTTER_PROJECT_NAME }}

- run: mamba list
working-directory: ${{ env.COOKIECUTTER_PROJECT_NAME }}

# Ensure the hack for the python version worked
- name: Ensure we have the right Python
shell: bash -l {0}
run: |
echo "Installed Python: $(python --version)"
echo "Expected: ${{ matrix.python-version }}"
python --version | grep "Python ${{ matrix.python-version }}"
working-directory: ${{ env.COOKIECUTTER_PROJECT_NAME }}

- name: Run pre-commits
shell: bash -l {0}
run: |
pre-commit install
pre-commit run -v --all-files --show-diff-on-failure
working-directory: ${{ env.COOKIECUTTER_PROJECT_NAME }}

- name: Test with pytest
shell: bash -l {0}
run: |
pytest -v
working-directory: ${{ env.COOKIECUTTER_PROJECT_NAME }}
6 changes: 6 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,9 @@
wandb
multirun.yaml
storage

# ignore the _version.py file
_version.py

# .gitignore defaults for python and pycharm
.idea
Expand Down
Loading