-
Notifications
You must be signed in to change notification settings - Fork 606
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Separate tox.ini
into multiple files
#2508
base: main
Are you sure you want to change the base?
Conversation
.github/workflows/lint.yml
Outdated
@@ -68,8 +68,6 @@ jobs: | |||
- "processor-baggage" | |||
- "propagator-aws-xray" | |||
- "propagator-ot-trace" | |||
- "resource-detector-container" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am using these 2 packages as an example.
.github/workflows/lint_1.yml
Outdated
~/.cache/pip | ||
key: v7-build-tox-cache-${{ env.RUN_MATRIX_COMBINATION }}-${{ hashFiles('tox.ini', 'gen-requirements.txt', 'dev-requirements.txt') }} | ||
- name: run tox | ||
run: tox -c ${{ matrix.package }}/tox.ini -e lint |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Notice how we are running tox
using a specific tox.ini
file, we use this pattern to avoid having to cd
into every package directory and back.
coverage.xml | ||
.coverage | ||
.nox | ||
.tox | ||
.cache | ||
htmlcov | ||
benchmark.json |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This file is generated when benchmark tests are run in the local computer of a developer. Added in this file to avoid adding it to git
unintentionally.
@@ -14,6 +14,6 @@ profile=black | |||
; docs: https://github.com/timothycrosley/isort#multi-line-output-modes | |||
multi_line_output=3 | |||
skip=target | |||
skip_glob=**/gen/*,.venv*/*,venv*/*,.tox/* | |||
skip_glob=**/gen/*,.venv*/*,venv*/*,.tox/*,**/.tox/* |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since new tox.ini
files are being added, new .tox
directories will be generated into every package when tox
is executed locally. This tells isort to not check these new .tox
directories.
lint: pylint --rcfile={toxinidir}/../../.pylintrc {toxinidir}/src/opentelemetry | ||
lint: pylint --rcfile={toxinidir}/../../.pylintrc {toxinidir}/tests | ||
|
||
coverage: coverage erase |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Using the coverage
commands directly here, so that it is not necessary to check an additional file (scripts/coverage/sh
) to understand what coverage
tests are doing.
coverage: coverage report --show-missing | ||
coverage: coverage xml | ||
|
||
spellcheck: codespell --config {toxinidir}/../../.codespellrc {toxinidir} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
spellcheck
can now be run only for a certain package instead for the whole repo, making it faster for most changes.
|
||
spellcheck: codespell --config {toxinidir}/../../.codespellrc {toxinidir} | ||
|
||
benchmark: pytest {toxinidir}/benchmarks --benchmark-json=benchmark.json |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Notice how the benchmark tests are now in their own directory, not inside tests
. This makes it possible to run them separately and not run them in CI.
@@ -1169,8 +1126,6 @@ commands = | |||
lint-exporter-prometheus-remote-write: pylint {toxinidir}/exporter/opentelemetry-exporter-prometheus-remote-write/src/opentelemetry | |||
lint-exporter-prometheus-remote-write: pylint {toxinidir}/exporter/opentelemetry-exporter-prometheus-remote-write/tests | |||
|
|||
coverage: {toxinidir}/scripts/coverage.sh |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Right now, we should not run coverage nor benchmark tests in CI. As they are configured right now, the coverage tests will not fail if a decrease in coverage is detected, the benchmark test will not fail if a decrease in performance is detected. For this reason, they cannot fail CI in a useful manner and it makes no sense to spend resources running them.
For these tests to be usefully executed in CI we need to configure them so that they can fail CI if a decrease in performance (for the benchmark tests) or a decrease in coverage (for the coverage tests) is detected.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍 makes sense. We should a file a bug regardless of this PR to set up coverage in CI
tox.ini
Outdated
@@ -1120,20 +1091,6 @@ commands = | |||
|
|||
test-util-http: pytest {toxinidir}/util/opentelemetry-util-http/tests {posargs} | |||
|
|||
test-sdk-extension-aws: pytest {toxinidir}/sdk-extension/opentelemetry-sdk-extension-aws/tests {posargs} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The size of this tox.ini
file will be greatly reduced when the rest of the packages are refactored.
I'm not really super enthusiastic about having so many From the issue:
What "important testing details" are you referring to specifically? Maybe if we call them out, there are smaller, less intrusive changes we can make. |
When I was introducing the changes that split Nevertheless, splitting the |
My main concern with this approach is that the common parts of the config that all tox.ini share may drift over time and become inconsistent or difficult to maintain. For example, adding/removing a supported Python version will require updating every tox file. More subtle things like changing the flags to a command or adding a new tool would be tricky. IIRC tox doesn't really have any mechanism for re-use like include/extend? |
That's right, but we have the very same problem now. We can forget to add a new python version in some package in the
Yes, but again, we have the same problem right now, we can make a mistake and forget a command argument for a particular package even if we only have one |
8e7d465
to
9548cc9
Compare
#2666, we have typos in our |
Another problem we have with a big We need
That command above would incorrectly match We can save ourselves the trouble of handling these collisions by having one |
Fixes #2478
This is still a draft PR because I have not yet refactored all packages so that all of them have their own
tox.ini
file.The advantages of this proposal are:
tox.ini
file will be much smaller. Right now, this file is extremely big.tox
commands will be much simpler. Instead of runningtox -e py38-test-opentelemetry-name-of-the-package
we will be able to justcd
into the package directory and then runtox -e test-py38
.pytest
with the benchmark options active for all tests, even when only one package here has benchmark tests. This introduces a warning in all of our logs. We should only runpytest
with the benchmark options when there are benchmark tests, this PR fixes this issue.tox.ini
file per package allows users to simply runtox
in the package they changed to run all necessary tests.