chore: Clean up toxgen (#4855)

### Description
Polish toxgen a bit, improving wording, examples, formatting, removing
obsolete parts.

#### Issues
<!--
* resolves: #1234
* resolves: LIN-1234
-->

#### Reminders
- Please add tests to validate your changes, and lint your code using
`tox -e linters`.
- Add GH Issue ID _&_ Linear ID (if applicable)
- PR title should use [conventional
commit](https://develop.sentry.dev/engineering-practices/commit-messages/#type)
style (`feat:`, `fix:`, `ref:`, `meta:`)
- For external contributors:
[CONTRIBUTING.md](https://github.com/getsentry/sentry-python/blob/master/CONTRIBUTING.md),
[Sentry SDK development docs](https://develop.sentry.dev/sdk/), [Discord
community](https://discord.gg/Ww9hbqr)

Author: sentrivana

scripts/populate_tox/README.md

@@ -16,8 +16,8 @@ The `populate_tox.py` script fills out the auto-generated part of that template.
 It does this by querying PyPI for each framework's package and its metadata and
 then determining which versions make sense to test to get good coverage.
 
-The lowest supported and latest version of a framework are always tested, with
-a number of releases in between:
+By default, the lowest supported and latest version of a framework are always
+tested, with a number of releases in between:
 - If the package has majors, we pick the highest version of each major.
 - If the package doesn't have multiple majors, we pick two versions in between
   lowest and highest.
@@ -34,7 +34,8 @@ the main package (framework, library) to test with; any additional test
 dependencies, optionally gated behind specific conditions; and optionally
 the Python versions to test on.
 
-Constraints are defined using the format specified below. The following sections describe each key.
+Constraints are defined using the format specified below. The following sections
+describe each key.
 
 ```
 integration_name: {
@@ -43,7 +44,7 @@ integration_name: {
          rule1: [package1, package2, ...],
          rule2: [package3, package4, ...],
      },
-     "python": python_version_specifier,
+     "python": python_version_specifier | dict[package_version_specifier, python_version_specifier],
      "include": package_version_specifier,
      "integration_name": integration_name,
      "num_versions": int,
@@ -102,15 +103,14 @@ Python versions, you can say:
     ...
 }
 ```
+
 This key is optional.
 
 ### `python`
 
 Sometimes, the whole test suite should only run on specific Python versions.
-This can be achieved via the `python` key.
-
-There are two variants how to define the Python versions to run the test suite
-on.
+This can be achieved via the `python` key. There are two variants how to define
+the Python versions to run the test suite on.
 
 If you want the test suite to only be run on specific Python versions, you can
 set `python` to a version specifier. For example, if you want AIOHTTP tests to
@@ -142,18 +142,17 @@ say:
 The `python` key is optional, and when possible, it should be omitted. The script
 should automatically detect which Python versions the package supports. However,
 if a package has broken metadata or the SDK is explicitly not supporting some
-packages on specific Python versions (because of, for example, broken context
-vars), the `python` key can be used.
+packages on specific Python versions, the `python` key can be used.
 
 ### `include`
 
-Sometimes we only want to consider testing some specific versions of packages.
-For example, the Starlite package has two alpha prereleases of version 2.0.0, but
-we do not want to test these, since Starlite 2.0 was renamed to Litestar.
+Sometimes we only want to test specific versions of packages. For example, the
+Starlite package has two alpha prereleases of version 2.0.0, but we do not want
+to test these, since Starlite 2.0 was renamed to Litestar.
 
 The value of the `include` key expects a version specifier defining which
 versions should be considered for testing. For example, since we only want to test
-versions below 2.x in Starlite, we can use
+versions below 2.x in Starlite, we can use:
 
 ```python
 "starlite": {
@@ -178,13 +177,21 @@ be expressed like so:
 
 Sometimes, the name of the test suite doesn't match the name of the integration.
 For example, we have the `openai_base` and `openai_notiktoken` test suites, both
-of which are actually testing the `openai` integration. If this is the case, you can use the `integration_name` key to define the name of the integration. If not provided, it will default to the name of the test suite.
+of which are actually testing the `openai` integration. If this is the case, you
+can use the `integration_name` key to define the name of the integration. If not
+provided, it will default to the name of the test suite.
 
-Linking an integration to a test suite allows the script to access integration configuration like for example the minimum version defined in `sentry_sdk/integrations/__init__.py`.
+Linking an integration to a test suite allows the script to access integration
+configuration like, for example, the minimum supported version defined in
+`sentry_sdk/integrations/__init__.py`.
 
 ### `num_versions`
 
-With this option you can tweak the default version picking behavior by specifying how many package versions should be tested. It accepts an integer equal to or greater than 2, as the oldest and latest supported versions will always be picked. Additionally, if there is a recent prerelease, it'll also always be picked (this doesn't count towards `num_versions`).
+With this option you can tweak the default version picking behavior by specifying
+how many package versions should be tested. It accepts an integer equal to or
+greater than 2, as the oldest and latest supported versions will always be
+picked. Additionally, if there is a recent prerelease, it'll also always be
+picked (this doesn't count towards `num_versions`).
 
 
 ## How-Tos
@@ -202,26 +209,3 @@ With this option you can tweak the default version picking behavior by specifyin
    `scripts/split_tox_gh_actions/split_tox_gh_actions.py`.
 4. Add the `TESTPATH` for the test suite in `tox.jinja`'s `setenv` section.
 5. Run `scripts/generate-test-files.sh` and commit the changes.
-
-### Migrate a test suite to populate_tox.py
-
-A handful of integration test suites are still hardcoded. The goal is to migrate
-them all to `populate_tox.py` over time.
-
-1. Remove the integration from the `IGNORE` list in `populate_tox.py`.
-2. Remove the hardcoded entries for the integration from the `envlist` and `deps` sections of `tox.jinja`.
-3. Run `scripts/generate-test-files.sh`.
-4. Run the test suite, either locally or by creating a PR.
-5. Address any test failures that happen.
-
-You might have to introduce additional version bounds on the dependencies of the
-package. Try to determine the source of the failure and address it.
-
-Common scenarios:
-- An old version of the tested package installs a dependency without defining
-  an upper version bound on it. A new version of the dependency is installed that
-  is incompatible with the package. In this case you need to determine which
-  versions of the dependency don't contain the breaking change and restrict this
-  in `TEST_SUITE_CONFIG`.
-- Tests are failing on an old Python version. In this case first double-check
-  whether we were even testing them on that version in the original `tox.ini`.

scripts/populate_tox/populate_tox.py

@@ -1,5 +1,7 @@
 """
 This script populates tox.ini automatically using release data from PyPI.
+
+See scripts/populate_tox/README.md for more info.
 """
 
 import functools
@@ -123,6 +125,7 @@ def _save_to_cache(package: str, version: Version, release: Optional[dict]) -> N
         releases_cache.write(json.dumps(_normalize_release(release)) + "\n")
 
     CACHE[_normalize_name(package)][str(version)] = release
+    CACHE[_normalize_name(package)][str(version)]["_accessed"] = True
 
 
 def _prefilter_releases(
@@ -150,7 +153,8 @@ def _prefilter_releases(
         min_supported = Version(".".join(map(str, min_supported)))
     else:
         print(
-            f"  {integration} doesn't have a minimum version defined in sentry_sdk/integrations/__init__.py. Consider defining one"
+            f"  {integration} doesn't have a minimum version defined in "
+            f"sentry_sdk/integrations/__init__.py. Consider defining one"
         )
 
     include_versions = None
@@ -225,7 +229,8 @@ def get_supported_releases(
     Get a list of releases that are currently supported by the SDK.
 
     This takes into account a handful of parameters (Python support, the lowest
-    version we've defined for the framework, the date of the release).
+    supported version we've defined for the framework, optionally the date
+    of the release).
 
     We return the list of supported releases and optionally also the newest
     prerelease, if it should be tested (meaning it's for a version higher than
@@ -272,9 +277,9 @@ def pick_releases_to_test(
 ) -> list[Version]:
     """Pick a handful of releases to test from a sorted list of supported releases."""
     # If the package has majors (or major-like releases, even if they don't do
-    # semver), we want to make sure we're testing them all (unless there's too
-    # many). If not, we just pick the oldest, the newest, and a couple
-    # in between.
+    # semver), we want to make sure we're testing them all. If it doesn't have
+    # multiple majors, we just pick the oldest, the newest, and a couple of
+    # releases in between.
     #
     # If there is a relevant prerelease, also test that in addition to the above.
     num_versions = TEST_SUITE_CONFIG[integration].get("num_versions")
@@ -342,7 +347,7 @@ def supported_python_versions(
     custom_supported_versions: Optional[
         Union[SpecifierSet, dict[SpecifierSet, SpecifierSet]]
     ] = None,
-    version: Optional[Version] = None,
+    release_version: Optional[Version] = None,
 ) -> list[Version]:
     """
     Get the intersection of Python versions supported by the package and the SDK.
@@ -362,6 +367,12 @@ def supported_python_versions(
       on Python 3.7, so we can provide this as `custom_supported_versions`. The
       result of this function will then by the intersection of all three, i.e.,
       [3.7].
+    - The Python SDK supports Python 3.6-3.13. The package supports 3.5-3.8.
+      Additionally, we have a limitation in place to only test this framework on
+      Python 3.5 if the framework version is <2.0. `custom_supported_versions`
+      will contain this restriction, and `release_version` will contain the
+      version of the package we're currently looking at, to determine whether the
+      <2.0 restriction applies in this case.
     """
     supported = []
 
@@ -377,11 +388,11 @@ def supported_python_versions(
                     if curr in custom_supported_versions:
                         supported.append(curr)
 
-                elif version is not None and isinstance(
+                elif release_version is not None and isinstance(
                     custom_supported_versions, dict
                 ):
                     for v, py in custom_supported_versions.items():
-                        if version in v:
+                        if release_version in v:
                             if curr in py:
                                 supported.append(curr)
                             break
@@ -435,8 +446,10 @@ def _parse_python_versions_from_classifiers(classifiers: list[str]) -> list[Vers
 
 def determine_python_versions(pypi_data: dict) -> Union[SpecifierSet, list[Version]]:
     """
-    Given data from PyPI's release endpoint, determine the Python versions supported by the package
-    from the Python version classifiers, when present, or from `requires_python` if there are no classifiers.
+    Determine the Python versions supported by the package from PyPI data.
+
+    We're looking at Python version classifiers, if present, and
+    `requires_python` if there are no classifiers.
     """
     try:
         classifiers = pypi_data["info"]["classifiers"]
@@ -453,7 +466,7 @@ def determine_python_versions(pypi_data: dict) -> Union[SpecifierSet, list[Versi
 
     # We only use `requires_python` if there are no classifiers. This is because
     # `requires_python` doesn't tell us anything about the upper bound, which
-    # depends on when the release first came out
+    # implicitly depends on when the release first came out.
     try:
         requires_python = pypi_data["info"]["requires_python"]
     except (AttributeError, KeyError):
@@ -666,7 +679,8 @@ def main(fail_on_changes: bool = False) -> None:
         # timestamp so that we don't fail CI on a PR just because a new package
         # version was released, leading to unrelated changes in tox.ini.
         print(
-            f"Since we're in fail_on_changes mode, we're only considering releases before the last tox.ini update at {last_updated.isoformat()}."
+            f"Since we're in fail_on_changes mode, we're only considering "
+            f"releases before the last tox.ini update at {last_updated.isoformat()}."
         )
 
     global MIN_PYTHON_VERSION, MAX_PYTHON_VERSION
@@ -789,19 +803,16 @@ def main(fail_on_changes: bool = False) -> None:
 
                 Please don't make manual changes to `tox.ini`. Instead, make the
                 changes to the `tox.jinja` template and/or the `populate_tox.py`
-                script (as applicable) and regenerate the `tox.ini` file with:
-
-                python -m venv toxgen.env
-                . toxgen.env/bin/activate
-                pip install -r scripts/populate_tox/requirements.txt
-                python scripts/populate_tox/populate_tox.py
+                script (as applicable) and regenerate the `tox.ini` file by
+                running scripts/generate-test-files.sh
                 """
                 )
             )
         print("Done checking tox.ini. Looking good!")
     else:
         print(
-            "Done generating tox.ini. Make sure to also update the CI YAML files to reflect the new test targets."
+            "Done generating tox.ini. Make sure to also update the CI YAML "
+            "files to reflect the new test targets."
         )