Summary:

This PR refactors the project by adding Ruff Linting and formatting, integrated with Nox and an optional Pre-commit hook. It enhances both code quality and development efficiency.

Key Changes:

1. Ruff Linting:
   • We have integrated Ruff linting, “an extremely fast Python linter designed as a drop-in replacement for Flake8 (plus dozens of plugins), isort, pydocstyle, pyupgrade, autoflake, and more.”
2. Ruff Formatter:
   • Compatible with the popular Black formatter but more performant.
3. Nox Automation:
   • Nox has been introduced to automate the execution of our CI tasks, including Ruff linting and other testing steps. Nox provides a flexible way to manage and run our tasks in isolated virtual environments, improving the reproducibility and ease of running multiple environments and workflows.
4. CI Pipeline Update:
   • Our CI configuration has been updated to include Ruff linting and formatting checks. From now on, Ruff will automatically lint all new commits and pull requests to ensure they adhere to our code quality standards.
5. Optional Pre-commit Hook:
   • A pre-commit hook for Ruff is now available, ensuring developers can run lint checks locally before committing code. This helps catch and resolve linting issues early in the development process.

Next Steps & Open Questions:

• Jupyter Notebooks: Should we include Jupyter notebooks in the Ruff linting process? -> Yes!
• Are all errors raised by the linter fixed?
• Contributing file updated?

Resolves issue #51

Configuration Details:

1. Ruff Linting Configuration:

The current Ruff configuration includes the most commonly used linting rules. (Future PRs will likely incorporate more linting rules, as discussed here).

• Pycodestyle (E): Enforces style consistency based on PEP8.
• Pyflakes (F): Detects basic errors such as undefined variables or missing imports.
• Pyupgrade (UP): Ensures usage of modern Python syntax, making our code more future-proof.
• Flake8-simplify (SIM): Improves code readability by suggesting simplified expressions. We have chosen to ignore rule SIM105 since our team prefers explicit boolean comparisons in some cases.
• Isort (I): Ensures imports are organized and ordered consistently, improving code readability and maintainability.

Note that a couple of linting rules were ignored with #noqa <ruff rule> (ignoring a single line) and #ruff: noqa <ruff rule> (ignoring entire file):

• E501 for too long lines has been ignored several times to ensure for example that links will still work or flow charts remain readable

• I001 sorting imports has been ignored once in src/plenoptic/__init__.py as the import order matters to avoid circular dependencies and tests to fail

• F401 unused imports

  • # ruff: noqa: F401 is used in several __init__.py files to ignore unused imports. When unused imports are removed, tests fail.

• SIM105 : Checks for try-except-pass blocks that can be replaced with the contextlib.suppress context manager.

  • currently ignored globally in pyproject.toml

• F403 flags wildcard imports * , which is bad practice.

  To remove wildcard (*) imports, issue Improve API #246 should be considered. 2 recommended approaches are:

  1. Explicit Imports in the __init__.py File:
     • Manually specify the functions and classes to be imported.
     • Example: from .validate import remove_grad
     • This approach aligns with the flat hierarchy recommendation from the OpenSci review (see issue Improve API #246)
  2. Nested Structure with Subpackages/Modules:
     • Organize the code into subpackages and import modules explicitly.
     • Example: plenoptic.tools.validate
       • Use from . import validate within the subpackage.
     • In files like validate.py, define a list called __all__ = ["remove_grad", ...], which specifies what can be imported from the file.
       • This can live in either the module file itself or the __init__.py, and it controls the public API of that module.

2. Ruff Formatting

Currently, formatting is enforced to emulate Black in pyproject.toml:

indent-width = 4 # same as Black
[tool.ruff.format]
# Like Black, use double quotes for strings.
quote-style = "double"

# Like Black, indent with spaces, rather than tabs.
indent-style = "space"

# Like Black, respect magic trailing commas.
skip-magic-trailing-comma = false

# Like Black, automatically detect the appropriate line ending.
line-ending = "auto"

Nox & Pre-commit Hooks and their Configurations:

TL;DR:

• Both tools are complementary and contribute to better maintainability by promoting consistent coding practices and making it easier to identify and fix issues early in the development process. Both are recommended by the Scientific Python Library Development Guide.
• Nox: A task-runner automating testing across different environments after commits, ensuring code works well in varied setups specified in noxfile.py.
• Pre-commit: Ensures code quality before commits by enforcing checks like formatting and linting. For this to work, pre-commit needs to be installed with pip install pre-commit . Pre-commit checks can be ignored, by adding --no-verify to your commit message.

2. Nox Configuration:

Nox is a Python-based task runner that is used to automate various development workflows, such as testing, building, and deploying applications. It allows you to define sessions, which are Python functions in the noxfile.py decorated with @nox.session with various parameters, such as name, by which these sessions can then be executed via the command line. This easily allows to test code by running pre-defined tasks with a single command across multiple Python versions without manually setting up environments.

Example: Based on the following noxfile.py , the command nox -s tests would run the session named "tests” below. nox without any commands would run all the sessions defined in the noxfile.py .

@nox.session(name="tests", python=["3.10", "3.11", "3.12"])
def tests(session):
    # run tests
    session.install("pytest")
    session.run("pytest")

Here are the current Nox sessions available for execution:

• nox -s lint
• nox -s tests
• nox -s coverage

3. Pre-Commit Configurations:

Hook:
A pre-commit hook can be added to run Ruff linting on code before it is committed. This ensures that developers catch and fix linting issues early in the development process. It only runs when installed and can be bypassed with git commit -m <my commit message> --no-verify.

Considered Configurations: (in .pre-commit.yaml:
(Legend: ✅: included, ❓: recommended, but not included, possibly in future PR)

• check-added-large-files ✅
  • Purpose: Checks for large files added to the repository, typically to prevent accidental inclusion of large binaries or datasets.
• check-case-conflict ✅
  • Purpose: Detects potential filename conflicts due to case-insensitive filesystems (e.g., Windows) where File.txt and file.txt would be considered the same
• check-merge-conflict ✅
  • Purpose: Checks for leftover merge conflict markers (<<<<<<<, =======, >>>>>>>) in files.
• check-symlinks ❓
  • Purpose: Verifies that symbolic links are valid and do not point to non-existent targets.
• check-yaml✅
  • Purpose: Validates YAML files for syntax errors.
• debug-statements ✅
  • Purpose: Detects debug statements (e.g., print, console.log) left in code.
• end-of-file-fixer✅
  • Purpose: Ensures files have a newline at the end.
• trailing-whitespace✅
  • Purpose: Removes trailing whitespace characters from files.

CI Pipeline:

Finally, the CI configuration ci.yml has been updated to include a step each for running Ruff formatting and linting check on all new commits and pull requests. Two separate actions were chosen to signal the developer which combination of the two might be causing an error.

Next steps:

0. update contributing file

1.Ruff Linting

Include the following linting rules (in pyproject.toml ) and resolve linting errors:

• B (flake8-bugbear):
  Checks for common programming errors, style issues, or patterns that are error-prone
• ARG (flake8-unused-arguments):
  Flags unused function arguments.
• C4 (flake8-comprehensions):
  Checks for issues in list comprehensions and generator expressions.
• EM (flake8-errmsg):
  Lints error message formatting.
• ICN (flake8-import-conventions):
  Ensures imports are formatted according to best practices.
• PTH (flake8-use-pathlib):
  Encourages the use of pathlib instead of os.path.
• RET (flake8-return):
  Checks for issues related to return statements.
• SIM (flake8-simplify):
  Suggests simplifications for code.
• T20 (flake8-print):
  Flags print statements.
• UP (pyupgrade):
  Upgrades syntax to modern Python standards.
• YTT (flake8-2020):
  Enforces Python 3.8+ compatibility.

2. Nox:

We might want to consider to include nox sessions for:

• format: Runs code formatting too
• docs: Builds and verifies the project's documentation to ensure it's up to date and properly rendered.
• type check: uses mypy or a similar tool to perform static type checking, to catch type-related errors early.

3. Pre-commit

We might want to add the following to .pre-commit-config.yaml :

• mixed-line-ending
  • Purpose: Checks for files with mixed line endings (e.g., both LF and CRLF).
    • Pros: Prevents compatibility issues across platforms, ensures consistent line endings.
    • Cons: Can require adjustments for projects that intentionally use mixed line endings for specific reasons.
• requirements-txt-fixer
  • Purpose: Standardizes the format of requirements.txt files, ensuring they follow conventions and are easy to manage.
  • Requires adjustments for projects with complex dependency requirements.