Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor ECCC functionality and create Preprocess module #165

Draft
wants to merge 46 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
7a9d15c
WIP - major refactoring of ECCC
Zeitsperre Jun 19, 2023
d1a8c67
WIP - more units handling
Zeitsperre Jun 20, 2023
94fe866
WIP - more refactoring - AHCCD incomplete
Zeitsperre Jun 20, 2023
d63523b
broken - refactoring of station writer
Zeitsperre Jun 21, 2023
500ae2a
broken - more refactoring
Zeitsperre Jul 6, 2023
c781287
broken - more refactoring
Zeitsperre Jul 6, 2023
8cf4368
Merge branch 'main' into fix_eccc
Zeitsperre Jul 31, 2023
ce2f6d9
significant refactoring - WIP
Zeitsperre Aug 2, 2023
9baaad5
working version of ahccd conversion
Zeitsperre Aug 3, 2023
53fc8f0
working version of ahccd conversion
Zeitsperre Aug 7, 2023
f454cb6
naming and more dynamic handling of variables
Zeitsperre Aug 7, 2023
9339f30
working version
Zeitsperre Aug 7, 2023
e096824
begin work on obs-summaries
Zeitsperre Aug 7, 2023
268cca1
finishing touches on ahccd
Zeitsperre Aug 9, 2023
8728c66
significant refactoring
Zeitsperre Aug 9, 2023
010d3ea
reduce amount of unit conversions
Zeitsperre Aug 10, 2023
7d8fdf9
refactoring - move treatments to new module, load_json_data_mappings …
Zeitsperre Aug 14, 2023
318957e
more refactoring
Zeitsperre Aug 14, 2023
42b8056
merge main into fix-eccc
Zeitsperre Feb 29, 2024
f5ca682
fix metadata, adjust tests
Zeitsperre Feb 29, 2024
412681c
Merge branch 'main' into fix_eccc
Zeitsperre Mar 4, 2024
c2e442a
error handling
Zeitsperre Mar 6, 2024
f5fce92
better error handling
Zeitsperre Mar 6, 2024
75e8015
import fixes
Zeitsperre Mar 6, 2024
c41e687
Merge branch 'main' into fix_eccc
Zeitsperre Mar 18, 2024
8c4ffcc
fix logic
Zeitsperre Mar 18, 2024
448ba07
fix logic
Zeitsperre Mar 18, 2024
53b9c35
fix logic
Zeitsperre Mar 18, 2024
159fc94
add files to sdist
Zeitsperre Mar 18, 2024
586f4e6
synchronize dependencies
Zeitsperre Mar 18, 2024
6c5586b
debugging
Zeitsperre Mar 18, 2024
e2762d3
chase down bug
Zeitsperre Mar 18, 2024
d20be40
update attr treatments
Zeitsperre Mar 19, 2024
74e7949
fixes
Zeitsperre Mar 19, 2024
42b957d
fix folder name
Zeitsperre Mar 19, 2024
288582f
add support for new `h` freq
Zeitsperre Mar 27, 2024
55ee882
dependencies
Zeitsperre Mar 27, 2024
66a2006
restructure
Zeitsperre Jan 6, 2025
13875cd
update deps
Zeitsperre Jan 6, 2025
6d35a9a
try to negotiate merge
Zeitsperre Jan 6, 2025
fa7be57
remove artifacts
Zeitsperre Jan 6, 2025
8cf5c33
fix style violations
Zeitsperre Jan 6, 2025
e763227
fix bad imports
Zeitsperre Jan 6, 2025
b8756b2
update cookiecutter template
Zeitsperre Jan 6, 2025
0e11f16
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jan 6, 2025
6e8d062
update conventions
Zeitsperre Jan 6, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .cruft.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"template": "https://github.com/Ouranosinc/cookiecutter-pypackage",
"commit": "1d9ee5f08d3e8e4f78a4aabb75e2ce4eff8750bf",
"commit": "63f44fcbfe2e16118a4fa6b09fe847aa44e0715a",
"checkout": null,
"context": {
"cookiecutter": {
Expand Down
11 changes: 4 additions & 7 deletions .github/workflows/bump-version.yml
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ jobs:
github.com:443
pypi.org:443
- name: Checkout Repository (no persist-credentials)
uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
persist-credentials: false
fetch-depth: 0
Expand All @@ -68,23 +68,20 @@ jobs:
run: |
git config --local user.email "bumpversion[bot]@ouranos.ca"
git config --local user.name "bumpversion[bot]"
- name: Current Version
run: |
CURRENT_VERSION="$(grep -E '__version__' src/miranda/__init__.py | cut -d ' ' -f3)"
echo "CURRENT_VERSION=${CURRENT_VERSION}" >> $GITHUB_ENV
- name: Install CI libraries
run: |
python -m pip install --require-hashes -r CI/requirements_ci.txt
- name: Conditional Bump Version
run: |
if [[ ${{ env.CURRENT_VERSION }} =~ -dev(\.\d+)? ]]; then
CURRENT_VERSION=$(bump-my-version show current_version)
if [[ ${CURRENT_VERSION} =~ -dev(\.\d+)? ]]; then
echo "Development version (ends in 'dev(\.\d+)?'), bumping 'build' version"
bump-my-version bump build
else
echo "Version is stable, bumping 'patch' version"
bump-my-version bump patch
fi
bump-my-version show-bump
echo "new_version=$(bump-my-version show current_version)"
- name: Push Changes
uses: ad-m/github-push-action@d91a481090679876dfc4178fef17f286781251df # v0.8.0
with:
Expand Down
4 changes: 3 additions & 1 deletion .github/workflows/cache-cleaner.yml
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,9 @@ jobs:
objects.githubusercontent.com:443

- name: Checkout Repository
uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
persist-credentials: false

- name: Cleanup
run: |
Expand Down
5 changes: 4 additions & 1 deletion .github/workflows/codeql-analysis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,8 @@ name: "CodeQL"

on:
push:
branches: [ "main" ]
branches:
- main
paths-ignore:
- ../../CHANGELOG.rst
- pyproject.toml
Expand Down Expand Up @@ -43,6 +44,8 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@v4
with:
persist-credentials: false

# Initializes the CodeQL tools for scanning.
- name: Initialize CodeQL
Expand Down
4 changes: 3 additions & 1 deletion .github/workflows/dependency-review.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,9 @@ jobs:
github.com:443

- name: Checkout Repository
uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
persist-credentials: false

- name: Dependency Review
uses: actions/dependency-review-action@3b139cfc5fae8b618d3eae3675e383bb1769c019 # v4.5.0
27 changes: 21 additions & 6 deletions .github/workflows/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,9 @@ jobs:
with:
egress-policy: audit
- name: Checkout Repository
uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
persist-credentials: false
- name: Set up Python${{ matrix.python-version }}
uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b # v5.3.0
with:
Expand All @@ -60,14 +62,21 @@ jobs:
strategy:
matrix:
os: [ 'ubuntu-latest' ]
python-version: [ "3.9", "3.10", "3.11", "3.12" ] # "3.13"
python-version:
- "3.9"
- "3.10"
- "3.11"
- "3.12"
# - "3.13"
steps:
- name: Harden Runner
uses: step-security/harden-runner@0080882f6c36860b6ba35c610c98ce87d4e2f26f # v2.10.2
with:
egress-policy: audit
- name: Checkout Repository
uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
persist-credentials: false
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b # v5.3.0
with:
Expand Down Expand Up @@ -102,7 +111,12 @@ jobs:
strategy:
matrix:
os: [ 'ubuntu-latest' ]
python-version: [ "3.9", "3.10", "3.11", "3.12" ]
python-version:
- "3.9"
- "3.10"
- "3.11"
- "3.12"
# - "3.13"
defaults:
run:
shell: bash -l {0}
Expand All @@ -112,15 +126,16 @@ jobs:
with:
egress-policy: audit
- name: Checkout Repository
uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
persist-credentials: false
- name: Setup Conda (Micromamba) with Python${{ matrix.python-version }}
uses: mamba-org/setup-micromamba@068f1ab4b37ed9b3d9f73da7db90a0cda0a48d29 # v2.0.3
with:
cache-downloads: true
environment-file: environment-dev.yml
create-args: >-
python=${{ matrix.python-version }}
micromamba-version: 1.5.10-0 # Pin micromamba version because of following issue: https://github.com/mamba-org/setup-micromamba/issues/225
- name: Install miranda
run: |
python -m pip install --no-deps .
Expand Down
4 changes: 3 additions & 1 deletion .github/workflows/publish-pypi.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,9 @@ jobs:
pypi.org:443
upload.pypi.org:443
- name: Checkout Repository
uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
persist-credentials: false
- name: Set up Python3
uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b # v5.3.0
with:
Expand Down
6 changes: 4 additions & 2 deletions .github/workflows/scorecard.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,9 @@ on:
- main

# Declare default permissions as read only.
permissions: read-all
# Read-all permission is not technically needed for this workflow.
permissions:
contents: read

jobs:
analysis:
Expand Down Expand Up @@ -47,7 +49,7 @@ jobs:
www.bestpractices.dev:443

- name: Checkout Repository
uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
persist-credentials: false

Expand Down
8 changes: 6 additions & 2 deletions .github/workflows/tag-testpypi.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,9 @@ jobs:
with:
egress-policy: audit
- name: Checkout Repository
uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
persist-credentials: false
- name: Create Release
uses: softprops/action-gh-release@7b4da11513bf3f43f9999e90eabced41ab8bb048 # 2.2.0
env:
Expand Down Expand Up @@ -52,7 +54,9 @@ jobs:
pypi.org:443
test.pypi.org:443
- name: Checkout Repository
uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
persist-credentials: false
- name: Set up Python3
uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b # v5.3.0
with:
Expand Down
9 changes: 7 additions & 2 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ repos:
rev: v3.19.0
hooks:
- id: pyupgrade
args: [ '--py38-plus' ]
args: [ '--py39-plus' ]
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v5.0.0
hooks:
Expand Down Expand Up @@ -41,7 +41,7 @@ repos:
hooks:
- id: isort
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.8.1
rev: v0.8.2
hooks:
- id: ruff
args: [ '--fix' ]
Expand Down Expand Up @@ -78,6 +78,11 @@ repos:
hooks:
- id: check-github-workflows
- id: check-readthedocs
- repo: https://github.com/woodruffw/zizmor-pre-commit
rev: v0.8.0
hooks:
- id: zizmor
args: [ '--config=.zizmor.yml' ]
- repo: meta
hooks:
- id: check-hooks-apply
Expand Down
6 changes: 6 additions & 0 deletions .zizmor.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
rules:
dangerous-triggers:
ignore:
- label.yml:9
- first-pull-request.yml:3
- workflow-warning.yml:3
4 changes: 2 additions & 2 deletions CI/requirements_ci.in
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
bump-my-version==0.27.0
bump-my-version==0.28.0
coveralls==4.0.1
pip==24.3.1
flit==3.9.0
tox==4.23.2
tox-gh==1.3.2
tox-gh==1.4.4
22 changes: 10 additions & 12 deletions CI/requirements_ci.txt
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
#
# This file is autogenerated by pip-compile with Python 3.8
# This file is autogenerated by pip-compile with Python 3.9
# by the following command:
#
# pip-compile --generate-hashes --output-file=CI/requirements_ci.txt CI/requirements_ci.in
Expand All @@ -12,9 +12,9 @@ bracex==2.4 \
--hash=sha256:a27eaf1df42cf561fed58b7a8f3fdf129d1ea16a81e1fadd1d17989bc6384beb \
--hash=sha256:efdc71eff95eaff5e0f8cfebe7d01adf2c8637c8c92edaf63ef348c241a82418
# via wcmatch
bump-my-version==0.27.0 \
--hash=sha256:483c517af91559644d45036648e5d99f4f8c85f8d01394097d3d3e42c9e6acad \
--hash=sha256:911bfaf7d847d4348844c8fd16f7a11322233fb8dc90123f638069a369003642
bump-my-version==0.28.0 \
--hash=sha256:cc84ace477022a4cc8c401ef5c035f2f752df45488be90ccb764a47f7de0e395 \
--hash=sha256:ff3cb51bb15509ae8ebb8e8efa3eaa7c02209677f45457c8b007ef2f5bef7179
# via -r CI/requirements_ci.in
cachetools==5.5.0 \
--hash=sha256:02134e8439cdc2ffb62023ce1debca2944c3f289d66bb17ead3ab3dede74b292 \
Expand Down Expand Up @@ -399,27 +399,25 @@ tox==4.23.2 \
# via
# -r CI/requirements_ci.in
# tox-gh
tox-gh==1.3.2 \
--hash=sha256:beb8d277d5d7c1a1f09c107e4ef80bd7dd2f8f5d020edfaf4c1e3ae8fd45bf6f \
--hash=sha256:c2d6e977f66712e7cd5e5d1b655a1bd4c91ebaf3be104befdb53c81587292d7e
tox-gh==1.4.4 \
--hash=sha256:4ea585f66585b90f5826b1677cfc9453747792a0f9ff83d468603bc17556e07b \
--hash=sha256:b962e0f8c4619e98d11c2a135939876691e148b843b7dac4cff7de1dc4f7c215
# via -r CI/requirements_ci.in
typing-extensions==4.12.2 \
--hash=sha256:04e5ca0351e0f3f85c6853954072df659d0d13fac324d0072316b67d7794700d \
--hash=sha256:1a7ead55c7e559dd4dee8856e3a88b41225abfe1ce8df57b7c13915fe121ffb8
# via
# annotated-types
# pydantic
# pydantic-core
# rich
# rich-click
# tox
urllib3==2.2.2 \
--hash=sha256:a448b2f64d686155468037e1ace9f2d2199776e17f0a46610480d311f73e3472 \
--hash=sha256:dd505485549a7a552833da5e6063639d0d177c04f23bc3864e41e5dc5f612168
# via requests
virtualenv==20.26.6 \
--hash=sha256:280aede09a2a5c317e409a00102e7077c6432c5a38f0ef938e643805a7ad2c48 \
--hash=sha256:7345cc5b25405607a624d8418154577459c3e0277f5466dd79c49d5e492995f2
virtualenv==20.27.1 \
--hash=sha256:142c6be10212543b32c6c45d3d3893dff89112cc588b7d0879ae5a1ec03a47ba \
--hash=sha256:f11f1b8a29525562925f745563bfd48b189450f61fb34c4f9cc79dd5aa32a1f4
# via tox
wcmatch==8.5.2 \
--hash=sha256:17d3ad3758f9d0b5b4dedc770b65420d4dac62e680229c287bf24c9db856a478 \
Expand Down
22 changes: 11 additions & 11 deletions environment-dev.yml
Original file line number Diff line number Diff line change
Expand Up @@ -33,20 +33,20 @@ dependencies:
- xesmf
- zarr
# Dev tools and testing
- pip >=24.2.0
- bump-my-version >=0.25.1
- watchdog >=4.0.0
- pip >=24.3.1
- black ==24.10.0
- blackdoc ==0.3.9
- bump-my-version >=0.28.0
- coverage >=7.5.0
- coveralls >=4.0.1
- flake8 >=7.1.1
- flake8-rst-docstrings >=0.3.0
- flit >=3.9.0,<4.0
- tox >=4.17.1
- coverage >=7.5.0
- coveralls >=4.0.1
- pytest >=8.3.2
- pytest-cov >=5.0.0
- black ==24.8.0
- blackdoc ==0.3.9
- isort ==5.13.2
- numpydoc >=1.8.0
- pre-commit >=3.5.0
- ruff >=0.5.7
- pytest >=8.3.2
- pytest-cov >=5.0.0
- ruff >=0.8.2
- tox >=4.23.2
- watchdog >=4.0.0
12 changes: 6 additions & 6 deletions environment-docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,16 +3,16 @@ channels:
- conda-forge
- defaults
dependencies:
- python >=3.12,<3.13
- sphinx >=7.0.0
- pandoc
- furo >=2023.07.26
- ipython
- python >=3.9,<3.13
- ipykernel
- ipython
- nbsphinx
# Docs
- furo >=2023.07.26
- pandoc
- sphinx >=7.1.0
- sphinx-autoapi
- sphinx-codeautolink
- sphinx-copybutton
- sphinx-intl
- sphinx-mdinclude
- sphinxcontrib-napoleon
Loading
Loading