Chore/update linting tools

[mcocdawc]

Description

Updated and unified the pre-commit hooks in all earthkit packages.
Some of them were failing the test suite just because of the pre-commit hooks.
Fixing this it was a good opportunity to unify settings.

Uses now ruff for formatting (former black), linting, and import sort (former isort)

The following PRs do the same changes in all the earthkit packages:

• Chore/update linting tools earthkit-climate#43
• Chore/update linting tools earthkit-data#927
• Chore/update linting tools earthkit-geo#39
• Chore/update linting tools earthkit-hydro#178
• Chore/update linting tools earthkit-meteo#125
• Chore/update linting tools earthkit-plots#176
• Chore/update linting tools earthkit-time#14
• Chore/update linting tools earthkit-transforms#87
• Chore/update linting tools #173

Contributor Declaration

By opening this pull request, I affirm the following:

• All authors agree to the Contributor License Agreement.
• The code follows the project's coding standards.
• I have performed self-review and added comments where needed.
• I have added or updated tests to verify that my changes are effective and functional.
• I have run all existing tests and confirmed they pass.

[Copilot]

Pull request overview

Updates and unifies the repository’s linting/formatting toolchain around Ruff (replacing Black/isort) and applies the resulting formatting/docstring/style updates across the codebase and tests/docs.

Changes:

• Update pre-commit hooks to use ruff-check/ruff-format and add TOML/YAML formatters.
• Add/adjust Ruff + pytest configuration in pyproject.toml.
• Apply widespread import ordering, formatting, and docstring punctuation changes across src/, tests/, docs/, and integration_tests/.

Reviewed changes

Copilot reviewed 139 out of 142 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
tests/earthkit_workflows/test_qubed_expansion.py	Ruff-driven formatting/import ordering updates in qube expansion tests.
tests/earthkit_workflows/test_metadata.py	Docstring punctuation updates in tests.
tests/earthkit_workflows/test_fluent.py	Formatting changes in fluent API tests.
tests/earthkit_workflows/test_decorators.py	Docstring punctuation updates in decorator tests.
tests/earthkit_workflows/helpers.py	Formatting changes in test helper.
tests/earthkit_workflows/graph/test_visit.py	Formatting changes in graph visitor tests.
tests/earthkit_workflows/graph/test_transform.py	Formatting changes in graph transformer tests.
tests/earthkit_workflows/graph/test_split.py	Formatting changes in graph split tests.
tests/earthkit_workflows/graph/test_graph.py	Formatting changes in graph tests.
tests/earthkit_workflows/graph/test_fuse.py	Formatting changes in fuse tests.
tests/earthkit_workflows/backends/test_xarray.py	Formatting changes in xarray backend tests.
tests/earthkit_workflows/backends/test_earthkit.py	Formatting changes in earthkit backend tests.
tests/earthkit_workflows/backends/test_arrayapi.py	Formatting changes in array API backend tests.
tests/earthkit_workflows/backends/generic_tests.py	Formatting changes in shared backend test utilities.
tests/cascade/shm/test_shm.py	Formatting/import cleanup in shm tests.
tests/cascade/scheduler/util.py	Formatting/docstring punctuation updates in scheduler test utilities.
tests/cascade/scheduler/test_checkpoints.py	Import ordering + formatting updates in checkpoint scheduler tests.
tests/cascade/scheduler/test_api.py	Docstring punctuation updates in scheduler API tests.
tests/cascade/low/test_exceptions.py	Formatting + minor assertion style update in exception tests.
tests/cascade/low/test_core.py	Formatting (blank line) update.
tests/cascade/low/test_builders.py	Formatting + comment spacing updates in builder tests.
tests/cascade/gateway/test_run.py	Formatting changes in gateway integration tests.
tests/cascade/executor/util.py	Formatting/docstring wrapping in executor test utilities.
tests/cascade/executor/test_runner.py	Import ordering + formatting in runner tests.
tests/cascade/executor/test_packages.py	Import ordering + formatting + string wrapping in package tests.
tests/cascade/executor/test_executor.py	Import ordering + formatting in executor tests.
tests/cascade/executor/test_checkpoints.py	Import ordering + formatting + byte literal style in checkpoint tests.
tests/cascade/executor/test_callables.py	Formatting/docstring punctuation in callable tests.
tests/cascade/controller/test_run.py	Import ordering + formatting/docstring adjustments in controller tests.
tests/cascade/benchmarks/image_processing.py	Formatting and import ordering in benchmark script.
src/earthkit/workflows/visualise.py	Docstring punctuation update.
src/earthkit/workflows/utility.py	Formatting (list comprehension layout).
src/earthkit/workflows/taskgraph.py	Formatting changes in task graph utilities.
src/earthkit/workflows/plugins/init.py	Docstring punctuation update.
src/earthkit/workflows/nodetree.py	Formatting + error message line wrapping.
src/earthkit/workflows/graph/visit.py	Docstring punctuation updates.
src/earthkit/workflows/graph/transform.py	Docstring punctuation updates.
src/earthkit/workflows/graph/split.py	Docstring punctuation + signature formatting.
src/earthkit/workflows/graph/samplegraphs.py	Docstring punctuation + formatting in sample graphs.
src/earthkit/workflows/graph/rename.py	Docstring punctuation + formatting in rename helpers.
src/earthkit/workflows/graph/pyvis.py	Formatting + docstring punctuation in pyvis export.
src/earthkit/workflows/graph/nodes.py	Docstring punctuation + formatting in node classes.
src/earthkit/workflows/graph/networkx.py	Docstring punctuation + formatting in networkx helpers.
src/earthkit/workflows/graph/graphviz.py	Docstring punctuation + formatting in graphviz export.
src/earthkit/workflows/graph/graph.py	Docstring punctuation + formatting in core graph class.
src/earthkit/workflows/graph/fuse.py	Formatting + docstring punctuation in fuse logic.
src/earthkit/workflows/graph/export.py	Formatting + docstring punctuation in (de)serialisation helpers.
src/earthkit/workflows/graph/expand.py	Formatting + docstring punctuation in graph expansion.
src/earthkit/workflows/graph/deduplicate.py	Docstring punctuation update.
src/earthkit/workflows/graph/copy.py	Docstring punctuation update.
src/earthkit/workflows/compilers.py	Define module logger before optional cascade import; adjust messages.
src/earthkit/workflows/backends/xarray.py	Formatting + docstring punctuation in xarray backend.
src/earthkit/workflows/backends/earthkit.py	Formatting + docstring punctuation in FieldList backend.
src/earthkit/workflows/backends/init.py	Formatting + docstring punctuation in backend registration/helpers.
src/earthkit/workflows/adapters.py	Define module logger before optional cascade import.
src/earthkit/workflows/_qubed.py	Formatting (blank lines) cleanup.
src/cascade/shm/server.py	Docstring punctuation update.
src/cascade/shm/func.py	Comment/docstring punctuation updates.
src/cascade/shm/disk.py	Formatting; split statements for readability.
src/cascade/shm/dataset.py	Docstring bullet punctuation update.
src/cascade/shm/client.py	Docstring punctuation update.
src/cascade/shm/api.py	Line wrapping in serialization method.
src/cascade/shm/algorithms.py	Docstring punctuation update.
src/cascade/shm/init.py	Docstring/comment punctuation update.
src/cascade/scheduler/precompute.py	Formatting + comment wrapping.
src/cascade/scheduler/core.py	Formatting + comment wrapping.
src/cascade/scheduler/checkpoints.py	Docstring formatting + line wrapping.
src/cascade/scheduler/assign.py	Formatting + comment wrapping.
src/cascade/scheduler/api.py	Formatting + docstring punctuation.
src/cascade/main.py	Docstring punctuation + import cleanup/formatting.
src/cascade/low/views.py	Docstring punctuation updates.
src/cascade/low/tracing.py	Docstring/comment wrapping and formatting.
src/cascade/low/into.py	Docstring punctuation update.
src/cascade/low/func.py	Docstring punctuation + formatting.
src/cascade/low/execution_context.py	Docstring punctuation updates.
src/cascade/low/exceptions.py	Formatting and regex wrapping in des().
src/cascade/low/core.py	Formatting/docstring punctuation and minor string style changes.
src/cascade/low/builders.py	Formatting changes in builders.
src/cascade/gateway/spawning.py	Docstring punctuation + signature formatting.
src/cascade/gateway/server.py	Docstring punctuation update.
src/cascade/gateway/router.py	Formatting + docstring punctuation.
src/cascade/gateway/client.py	Docstring punctuation update.
src/cascade/gateway/api.py	Docstring punctuation update.
src/cascade/gateway/main.py	Remove unused import; formatting.
src/cascade/gateway/init.py	Docstring wrapping/formatting.
src/cascade/executor/serde.py	Docstring punctuation updates.
src/cascade/executor/runner/runner.py	Docstring punctuation update.
src/cascade/executor/runner/packages.py	Formatting + docstring improvements + minor exception clause tweak.
src/cascade/executor/runner/memory.py	Formatting; minor change around fake publish code path.
src/cascade/executor/runner/entrypoint.py	Remove unused imports; docstring improvements + formatting.
src/cascade/executor/runner/init.py	Docstring punctuation update.
src/cascade/executor/platform.py	Docstring punctuation + signature formatting.
src/cascade/executor/msg.py	Comment/docstring formatting; minor type alias layout.
src/cascade/executor/executor.py	Import cleanup/formatting; minor conditional style fix.
src/cascade/executor/data_server.py	Formatting + comment/docstring punctuation.
src/cascade/executor/comms.py	Formatting + docstring punctuation; log wrapping.
src/cascade/executor/checkpoints.py	Docstring punctuation + error message tweaks + formatting.
src/cascade/executor/bridge.py	Import cleanup/formatting; type alias layout.
src/cascade/deployment/logging/json_formatter.py	Docstring punctuation; remove unused import.
src/cascade/deployment/logging/defaults.py	Formatting/indent cleanup in dict factories.
src/cascade/deployment/logging/init.py	Formatting + typing spacing; remove unused import.
src/cascade/deployment/init.py	Docstring whitespace cleanup.
src/cascade/controller/report.py	Docstring punctuation update.
src/cascade/controller/notify.py	Docstring punctuation + comment wrapping.
src/cascade/controller/impl.py	Import cleanup/formatting; log wrapping.
src/cascade/controller/core.py	Formatting + docstring punctuation updates.
src/cascade/controller/act.py	Docstring punctuation + formatting/log wrapping.
pyproject.toml	Add Ruff + pytest config; add dependency groups; reorder/format project metadata.
integration_tests/runtime/init.py	Docstring punctuation + blank line after import.
integration_tests/job_runnerRestarts.py	Import ordering + formatting.
integration_tests/harness.py	Import ordering/cleanup.
integration_tests/benchmarks_old/util.py	Reduce/clean imports + formatting.
integration_tests/benchmarks_old/reporting.py	Formatting changes in benchmark reporting.
integration_tests/benchmarks_old/plotting.py	Formatting changes in plotting utilities.
integration_tests/benchmarks_old/matmul.py	Split statement for readability.
integration_tests/benchmarks_old/job1.py	Import ordering + formatting in benchmark job script.
integration_tests/benchmarks_old/generators.py	Docstring punctuation + formatting.
integration_tests/benchmarks_old/dist.py	Formatting changes in benchmark graph generator.
integration_tests/benchmarks_old/dask.py	Formatting changes in dask benchmark adapter.
integration_tests/benchmarks_old/anemoi.py	Formatting changes in anemoi benchmark script.
integration_tests/base.py	Formatting (blank line) update.
docs/tutorials/cascade-job_building/t03_custom.py	Formatting + docstring punctuation/wrapping in tutorial.
docs/tutorials/cascade-job_building/t02_builders.py	Docstring punctuation update.
docs/tutorials/cascade-job_building/t01_low.py	Formatting changes in tutorial code.
docs/tutorials/cascade-job_building/t00_generate.py	Docstring punctuation + formatting.
docs/tutorials/cascade-job_building/t00_execute.py	Docstring punctuation + import spacing.
docs/examples/xarray.ipynb	Formatting/quote normalization in notebook cells.
docs/examples/arrayapi.ipynb	Formatting/whitespace normalization in notebook cells.
docs/benchmarking/run3/tasklanes.py	Formatting in benchmark plotting script.
docs/benchmarking/run3/tasklanes.ipynb	Formatting/quote normalization in notebook cells.
docs/benchmarking/run2/analysis.py	Formatting/line wrapping in analysis script.
docs/benchmarking/run2/analysis.ipynb	Formatting/quote normalization in notebook cells.
docs/benchmarking/run1/analysis.py	Formatting/line wrapping in analysis script.
docs/benchmarking/run1/analysis.ipynb	Formatting/quote normalization in notebook cells.
benchmarks/scheduling/sat_experiments.py	Formatting changes in scheduling benchmark.
benchmarks/scenario-shm_throughput.py	Docstring punctuation + formatting.
.pre-commit-config.yaml	Switch to newer Ruff hooks; add TOML/YAML formatters.
.github/workflows/test-pypi.yml	YAML cleanup for manual trigger syntax.
.github/workflows/macos-test.yml	YAML cleanup (but currently introduces invalid indentation).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[tmi]

I don't think we want to follow a common setting, definitively not in this form, for example because of
1/ particularly the line breaking causes quite a mess
2/ from what you did, it appears to me you disabled the isorting actually? There was the --select I but you removed that
3/ why did you remove the large files filter? There was a reason for that, uv locks can be quite great

I'm not saying that our pre-commit config is perfect, but this does not look like a change for the better

Eartkhit Workflows is perhaps not well-named, it uses things from Earthkit ecosystem, but is not that much part of it in the sense of a common developer experience or a release bundling

[tmi]

incidentally the current precommit is broken 🙈 I just did the minimum fix in #175

[mcocdawc]

incidentally the current precommit is broken

That was also the case in some other projects and since this affected the downstream-CI, I wanted to fix it.
Since I was touching the files anyway I thought a more uniform behaviour would be good.

1. Some projects in earthkit had 120, others 80 as max line length. I thought it would be a good idea to unify this and looked into coding guidelines by ECMWF; the ones I found said 120 e.g. here. I thought I am doing a favour by this maintenance work. If you want to stick to 140, then I revert it of course.

2. isort is fully redundant and contained in ruff. You can verify by scrambling the sort order and calling ruff check --fix (or the pre-commit hook). In addition, I moved the -I option of ruff into the pyproject.toml. This means that the IDE, delegating to ruff, and the pre-commit hook use the same settings, otherwise you might end up in a situation where IDE autoformatting and pre-commit hooks differ.

3. I did not remove the large check file. The line just moved because of the YAML formatting. But a genuine mistake was that I accidentally removed the exclude exception; sorry for that, it is fixed now.

Long story short, the changed line lengths add a lot of changes, which are indeed unnecessary, if you want to keep 140.
I will close this PR and branch off from develop again. Then I only add the isort change and a few pre-commit hooks but don't change the formatting. Does that sound good?

[tmi]

and since this affected the downstream-CI

we are affecting downstream-CI? I thought we excluded earthkit-workflows from there

If you want to stick to 140

yes :)

In addition, I moved the -I option of ruff into the pyproject.toml. This means that the IDE, delegating to ruff, and the pre-commit hook use the same settings, otherwise you might end up in a situation where IDE autoformatting and pre-commit hooks differ.

completely agreed!

I will close this PR and branch off from develop again.

don't worry about this -- I'll adapt the -I -> pyproject.toml change myself on top of the other precommit changes I did in the other branch (which you inspired as well -- thanks for that!). For this project its more important to keep it in sync with https://github.com/ecmwf/forecast-in-a-box than with other earthkit projects, because of developer intersection (and to some extent, with pproc and anemoi too... and you don't want to embark on unification of all that :) ). So I'll then make similar changes in the forecast-in-a-box project (the respective precommits are near identical)

(on a completely side note, I think we should long term replace precommit config with prek config... the only reason I havent done it yet is that not enough people switched to prek completely yet... over time, I think our projects will keep converging and diverging, back and forth. But thats healthy -- complete unification implies stale innovation)

[mcocdawc]

I thought we excluded earthkit-workflows from there

And you are fully right in that.

don't worry about this -- I'll adapt the -I -> pyproject.toml change myself on top of the other precommit changes [...]

Wonderful, then I just close the PR.

complete unification implies stale innovation

Completely agreed. I consciously did not use reusable workflows or other single yaml files as source.

Have to look into prek. Did not know it, but it sounds interesting.

Cheers.