Code Linting and formatting

[rsb-23]

@da4089 Are we looking to add linters and formatters to this project?
If so, I'd be happy to raise a PR for the same. (Reference: Linter configs)

1. Do we have a contribution guide?
2. Which python versions are supported? 1bd6c3b

Suggestion:
Sourcery is a good tool to refactor python code and is free for open-source projects. Hence I think we should use it.
It is a good starting point to improve and optimise legacy code. (another PR)

Disclaimer : I have been using these tools for 3+ years in office projects for easier code maintenance.

[return42]

In SearXNG we had good experience with black code formatter and pylint for static code analysis.

[akuker]

I've had good experience with these tools as well. We may want to suppress the "invalid-name" warnings from pylint (for now). Changing the names of methods is going to break the users of the library.

[da4089]

I'd like to add some things of this sort.

I have two (possibly contradictory) views here: (a) I'd like to use something that developers can run on their own machines before submitting a PR, and (b) a nicely formatted report is helpful.

I've previously used a couple of different services for code quality analysis. While I was more-or-less happy with the output, I didn't like that you couldn't run the analysis locally, and then one of the services shut down (it was a startup) which was a hassle.

[return42]

(a) I'd like to use something that developers can run on their own machines before submitting a PR

That is the way we use black and pylint in SearXNG .. both tools can be installed by pip.

I've previously used a couple of different services for code quality analysis.

We have a Makefile and the developer has to run make format.python and make test before commit. But there is no need for a Makefile / you can run the tools directly from the command line black --help and pylint --help.

and (b) a nicely formatted report is helpful.

Pylint has an option --output-format=<format> .. the default output format can be read by major IDEs.

Black has its own output format / not sure you can customize it .. but most often you will run black to reformat the files in place (without option --check the files will be formatted).

Additional in our project, we started to use type hints .. when you run make test in our project additional to black and pylint, pyright a static type checker rates the code ..

All three tools (black, pylint and pyright)

• can be installed by pip / can be used locally on developer's machine
• have integration for major IDEs

[da4089]

Yes, I've used both black and pylint before. I think they're quite good.

Regarding type hints, that's waiting on me making a branch to maintain the 0.9 releases that are compatible with Python 2.7. After that, the master branch can move to Python 3.7, and that brings type hints, f-strings, and (perhaps most useful) the modern bytes / str split that will all take some time to work through the codebase.

So that's probably the right time to add pyright.

For pylint and black, it'd likely be nice to add them before branching, so merging any fixes forward (or cherry-picking them back) is easy.

[rsb-23]

If so, I'd be happy to raise a PR for the same. (Reference: Linter configs)

Main config file

This uses pre-commit hooks and hence able to integrate all major linters, formatters, type checkers, etc.
Developers need to run pre-commit install once locally during project setup, then it automatically runs before every git commit.

Sourcery has plugins for IDEs and hence automatically suggests refactoring on the go.

Additionally, I prefer flake8 over pylint; but its for the community to decide. Both are equally good.

PS: I too follow fail-fast philosophy, hence this setup.

[da4089]

Re: Python versions, see the comments in #1, but in short: the plan is to support whatever CPython versions are actively supported by the Python Software Foundation, with an added 12 months to ensure people have time to migrate.

Right now, that means Python 3.7 and later, with 3.7 to be dropped at the end of June 2024.

[rsb-23]

@da4089 Its time to drop py37.

IMO, I suggest to ignore py38 & focus on py39+ and we will support it till Oct 2026.
Reasoning :

• End of life for py38 is in Oct, hence no new project will build on py38 for just 3 months.
• Also, old projects would have started migrating to py39+ by now.
• Its easier to maintain tool compatibility from py39, hence we can release changes quickly.

By focusing on py39, we can deliver more stable v1 by Oct 2024 which will serve more users.

[da4089]

Re: contributions guide ... no, not yet. I'll work on adding at least a skeleton pretty quickly, I guess.

[da4089]

I've put a trivial placeholder in place. I'll put some more cycles into this soon. Feel free to send PRs, of course 😛

[da4089]

I did a trial run of black over the code base, and while there aren't too many changes it wants to make, it does touch basically every string, a lot of function definitions, and various other bits of code that are differently formatted.

So ... I think the best time to apply this will be before making the Python2.7-compatible branch, so that it's easy to merge stuff between the there and master in future. I hope to get almost all the outstanding PRs merged in that same timeframe, so there will be a minimum of disruption if we blacken the code after that, and right before branching.