Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Separate tox.ini into multiple files #2508

Draft
wants to merge 27 commits into
base: main
Choose a base branch
from

Conversation

ocelotl
Copy link
Contributor

@ocelotl ocelotl commented May 8, 2024

Fixes #2478

This is still a draft PR because I have not yet refactored all packages so that all of them have their own tox.ini file.

The advantages of this proposal are:

  1. The main tox.ini file will be much smaller. Right now, this file is extremely big.
  2. The tox commands will be much simpler. Instead of running tox -e py38-test-opentelemetry-name-of-the-package we will be able to just cd into the package directory and then run tox -e test-py38.
  3. This PR will add coverage testing for all packages.
  4. Right now we run pytest with the benchmark options active for all tests, even when only one package here has benchmark tests. This introduces a warning in all of our logs. We should only run pytest with the benchmark options when there are benchmark tests, this PR fixes this issue.
  5. Right now, we should not run coverage nor benchmark tests in CI (please read the comments below for an explanation). This PR allows us to not run these tests in CI.
  6. Having a separate tox.ini file per package allows users to simply run tox in the package they changed to run all necessary tests.

@ocelotl ocelotl added the Skip Changelog PRs that do not require a CHANGELOG.md entry label May 8, 2024
@ocelotl ocelotl self-assigned this May 8, 2024
.github/workflows/instrumentations_0.yml Outdated Show resolved Hide resolved
@@ -68,8 +68,6 @@ jobs:
- "processor-baggage"
- "propagator-aws-xray"
- "propagator-ot-trace"
- "resource-detector-container"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am using these 2 packages as an example.

~/.cache/pip
key: v7-build-tox-cache-${{ env.RUN_MATRIX_COMBINATION }}-${{ hashFiles('tox.ini', 'gen-requirements.txt', 'dev-requirements.txt') }}
- name: run tox
run: tox -c ${{ matrix.package }}/tox.ini -e lint
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Notice how we are running tox using a specific tox.ini file, we use this pattern to avoid having to cd into every package directory and back.

coverage.xml
.coverage
.nox
.tox
.cache
htmlcov
benchmark.json
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file is generated when benchmark tests are run in the local computer of a developer. Added in this file to avoid adding it to git unintentionally.

@@ -14,6 +14,6 @@ profile=black
; docs: https://github.com/timothycrosley/isort#multi-line-output-modes
multi_line_output=3
skip=target
skip_glob=**/gen/*,.venv*/*,venv*/*,.tox/*
skip_glob=**/gen/*,.venv*/*,venv*/*,.tox/*,**/.tox/*
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since new tox.ini files are being added, new .tox directories will be generated into every package when tox is executed locally. This tells isort to not check these new .tox directories.

lint: pylint --rcfile={toxinidir}/../../.pylintrc {toxinidir}/src/opentelemetry
lint: pylint --rcfile={toxinidir}/../../.pylintrc {toxinidir}/tests

coverage: coverage erase
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using the coverage commands directly here, so that it is not necessary to check an additional file (scripts/coverage/sh) to understand what coverage tests are doing.

coverage: coverage report --show-missing
coverage: coverage xml

spellcheck: codespell --config {toxinidir}/../../.codespellrc {toxinidir}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

spellcheck can now be run only for a certain package instead for the whole repo, making it faster for most changes.


spellcheck: codespell --config {toxinidir}/../../.codespellrc {toxinidir}

benchmark: pytest {toxinidir}/benchmarks --benchmark-json=benchmark.json
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Notice how the benchmark tests are now in their own directory, not inside tests. This makes it possible to run them separately and not run them in CI.

@@ -1169,8 +1126,6 @@ commands =
lint-exporter-prometheus-remote-write: pylint {toxinidir}/exporter/opentelemetry-exporter-prometheus-remote-write/src/opentelemetry
lint-exporter-prometheus-remote-write: pylint {toxinidir}/exporter/opentelemetry-exporter-prometheus-remote-write/tests

coverage: {toxinidir}/scripts/coverage.sh
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right now, we should not run coverage nor benchmark tests in CI. As they are configured right now, the coverage tests will not fail if a decrease in coverage is detected, the benchmark test will not fail if a decrease in performance is detected. For this reason, they cannot fail CI in a useful manner and it makes no sense to spend resources running them.

For these tests to be usefully executed in CI we need to configure them so that they can fail CI if a decrease in performance (for the benchmark tests) or a decrease in coverage (for the coverage tests) is detected.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 makes sense. We should a file a bug regardless of this PR to set up coverage in CI

tox.ini Outdated
@@ -1120,20 +1091,6 @@ commands =

test-util-http: pytest {toxinidir}/util/opentelemetry-util-http/tests {posargs}

test-sdk-extension-aws: pytest {toxinidir}/sdk-extension/opentelemetry-sdk-extension-aws/tests {posargs}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The size of this tox.ini file will be greatly reduced when the rest of the packages are refactored.

@ocelotl ocelotl changed the title WIP Separate tox.ini into multiple files May 8, 2024
@ocelotl ocelotl requested review from aabmass and lzchen May 24, 2024 13:41
@lzchen
Copy link
Contributor

lzchen commented May 24, 2024

I'm not really super enthusiastic about having so many tox files. Is the largeness of the singular tox file so much that it warrants separating these out?

From the issue:

Right now, tox.ini is extremely long and it is easy to overlook important testing details.

What "important testing details" are you referring to specifically? Maybe if we call them out, there are smaller, less intrusive changes we can make.

@ocelotl
Copy link
Contributor Author

ocelotl commented May 28, 2024

I'm not really super enthusiastic about having so many tox files. Is the largeness of the singular tox file so much that it warrants separating these out?

From the issue:

Right now, tox.ini is extremely long and it is easy to overlook important testing details.

What "important testing details" are you referring to specifically? Maybe if we call them out, there are smaller, less intrusive changes we can make.

When I was introducing the changes that split lint into multiple tests, I noticed that we were missing several missing testing for a few of our components. It was hard to notice among the hundreds of lines in our tox.ini.

Nevertheless, splitting the tox.ini file is just another enhancement that this PR will introduce, please take a look at the advantages listed in this PR description.

@aabmass
Copy link
Member

aabmass commented Jun 6, 2024

My main concern with this approach is that the common parts of the config that all tox.ini share may drift over time and become inconsistent or difficult to maintain. For example, adding/removing a supported Python version will require updating every tox file. More subtle things like changing the flags to a command or adding a new tool would be tricky.

IIRC tox doesn't really have any mechanism for re-use like include/extend?

@ocelotl
Copy link
Contributor Author

ocelotl commented Jun 11, 2024

My main concern with this approach is that the common parts of the config that all tox.ini share may drift over time and become inconsistent or difficult to maintain. For example, adding/removing a supported Python version will require updating every tox file.

That's right, but we have the very same problem now. We can forget to add a new python version in some package in the tox.ini file as it is right now too. Having one single tox.ini file is no guarantee of consistency either.

More subtle things like changing the flags to a command or adding a new tool would be tricky.

Yes, but again, we have the same problem right now, we can make a mistake and forget a command argument for a particular package even if we only have one tox.ini file.

tox.ini Outdated Show resolved Hide resolved
@ocelotl
Copy link
Contributor Author

ocelotl commented Jul 5, 2024

#2666, we have typos in our tox.ini file.

@ocelotl
Copy link
Contributor Author

ocelotl commented Jul 10, 2024

Another problem we have with a big tox.ini file is that we can't have consistent tox environment names.

We need py3{8,9,10,11,12}-test-opentelemetry-instrumentation and py3{8,9,10,11,12}-test-instrumentation-aiohttp-client instead of py3{8,9,10,11,12}-test-opentelemetry-instrumentation-aiohttp-client because we have commands like this:

test-opentelemetry-instrumentation: pytest {toxinidir}/opentelemetry-instrumentation/tests {posargs}

That command above would incorrectly match py3{8,9,10,11,12}-test-opentelemetry-instrumentation-aiohttp-client too.

We can save ourselves the trouble of handling these collisions by having one tox.ini file per package.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Skip Changelog PRs that do not require a CHANGELOG.md entry
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Separate tox.ini into multiple files
5 participants