Test in Docker with Alpine Linux on CI #1826

EliahKagan · 2024-02-15T20:42:07Z

As suggested in #1745 (comment), this adds a test job that runs the tests inside a Docker container.

This can be done with Ubuntu in the container, but instead I used Alpine Linux in the container. This provides a greater variety of platforms GitPython tests on, potentially uncovering some bugs that would otherwise go undetected. In particular, Alpine Linux does not satisfy some of the assumptions that are sometimes made about platforms, as it does not, by default, have bash; its ps and related tools are not procps; it does not have GNU coreutils or GNU libc; and so forth.

No tests had to be marked xfail for this. In particular, as noted in #1745 (comment), we have no tests for #1756, so that ps is not procps in Alpine Linux does not currently result in any failures (and if there were such a test then the existing macOS jobs would still be sufficient for that).

This adds only one test job, testing only one version of Python. Currently that version is Python 3.11, since the Alpine Linux python3 apk package provides 3.11 at this time. The specified Docker image is alpine:latest, and the version will presumably advance to Python 3.12 eventually. I added only one job for two reasons:

This is easiest, since one version of Python 3 is packaged officially in Alpine Linux's downstream repositories, and it makes sense to test with that.
I don't want to create an excessive number of new jobs, since it seems likely that some more jobs covering interesting cases might be added, or at least might be worth considering, in the future. (One possible such case is discussed below.)

I was able to use the container key to do this, so that the workflow is still expressed in separate steps in the same style as the other test workflows, such that the operations that are analogous are represented by steps written in the same (or very similar) way across all workflows. I think the benefit of readability and comparability justifies this at this time, but depending what is done in the future, it might end up being changed to pack most of the work into a smaller number of steps (maybe one) that use uses: docker://..., or run: docker run ... or docker-run-action, or something else.

Specific reasons for existing design and implementation decisions, including for the strange and somewhat hacky sudo usage, are detailed in commit messages.

I've proposed this job as its own pull request, rather than waiting and attempting to introduce it with others, because I think this makes sense to review by itself (and I can incorporate any feedback into future changes), and because if I understand #1745 (comment) correctly it seems to be valued as a feature by itself (which I do agree with). I think that even if there are to be more Docker test jobs, they are better added in a subsequent PR.

However, I'm even more motivated by a goal that this PR does not itself achieve: testing on a bigendian architecture. Because GitPython uses git for most of its work, it shouldn't have a problem with that, but maybe some parts of gitdb/smmap that GitPython uses (for which GitPython's use is under test), or more importantly some seemingly innocuous aspects of encoding and decoding, might inadvertently assume little-endianness.

The good news is that no such bugs appear to be present. I have not yet gotten this to work on CI, but on my local amd64 system I am able to enable emulation and run s390x images. This is with no hardware acceleration, and some of the performance tests are too slow for me even to have waited for them to finish (which does not mean they would have a problem on an actual s390x machine). But when I tell pytest to deselect the performance tests, the remaining tests run in a reasonable time and all pass.

I'd like to do that on CI, which is not hard to do. If possible I'd like to use a workflow of this form, with the container key, and have it be a matrix job with both architectures, with special logic for enabling emulation and for skipping the performance tests for non-amd64 architectures (which would, initially and also probably forever, be just the one other architecture, s390x). I haven't managed to get it working that way, though. Using container:, the container is supposed to be able to start before any steps run. Even if I can manage to get emulation enabled without being able to easily run commands directly on the runner, I suspect I can't get that to happen early enough that using container: is both effective and reasonable.

If Docker test jobs for both amd64 and s390x architectures are to exist, I would want both to be written the same way, even if it makes the amd64 one less nice than in this PR. That way, it would be easier to compare their output to troubleshoot any future failures (as well as to understand how they are set up). If you have thoughts on whether or not this could be valuable, and also how much of a problem (if any) it would be to end up later converting the workflow to look more like that with fewer steps and not using container:, then that can guide me. However, whether or not you have guidance about that, I think this PR itself can probably stand on its own.

(I considered posting this section of the PR description as an issue instead, but I figured I'd put it here in case requested changes to this PR would substantially affect it or make it moot. and also because it provides a possible reason, beyond that discussed in #1745 (comment), to have Docker jobs like this one in the first place.)

With only one version of Python, currently 3.11.

This handles the "dubious ownership" error for the Alpine Linux container using safe.directory as is done in the Cygwin job. Another approach may be to actually use a limited user account in the container, though, and that may be better, since I expect some of the rmtree tests in test_util.py to fail due to the root user being able to perform a delete operation the tests assume cannot be done.

To overcome "This environment is externally managed" blocker.

- Add a note to the fixture in test_util.py that its ability to create files where rmtree will fail is contingent on not running as root (since root doesn't need to own a dir to delete from it). - Create a non-root user in the container. Give it the same UID as owns the repository files that are shared with the container. Also create a group with the GID of the repository files that are shared with the container and add the user to the group, though that is less important. Actually creating the user ensures it has a home directory and may help some commands work. Passing `options: --user 1001` under `container:` will not work because, even if we didn't make the user, the `apk add` commands still need to run as root. - Run all commands as the new non-root user, except for the system administration commands that install needed apk packages and set up the new non-root user account. To continue clearly expressing each step separately and have them automatically run in the container, this uses the hacky approach of having the default shell be a "sudo" command that runs the script step with "sh" (and passes the desired shell arguments). - Preserve environment variables that may have been set by or for the GHA runner, in commands that run as the non-root user. That is, pass those through, while still removing/resetting others. If this is not done, then the variables such as `CI`, which the init-tests-after-clone.sh uses to proceed without interactive confirmation, will not be set, and that step will fail. However, I think it is also a good idea to do this, which is why I've included all the relevant variables and not just `CI`. - Now that a non-root user is using "pip", stop using a venv, at least for now. The other test jobs don't use one, since the runners are isolated, and a container on a runner is even more isolated. But it may be best to bring the venv back, maybe even on the other test jobs, or alternatively to use "python -m pip" instead of "pip", to ensure expected version of pip is used. - Don't add safe.directory inside the container, in the hope that this may not be necessary because the cloned repository files should have the same UID (and even GID) as the user using them. But I expect this may need to be put back; it seems to be needed separately from that, as actions/checkout automatically attempts it for the git command it finds and attempts to use. This is not the only approach that could work. Another approach is to make use of the container explicit in each step, rather than using the `container` key. I think that would make the relationship between the commands here and in other test workflows less apparent and make the workflow a bit less clear, but it could also simplify things. A third approach is to create an image with the needed apk packages and user account, which switches to that user, by writing a Dockerfile and building in image, producing it in a previous job and sharing the image with the job that runs the tests so that `container` can still be used. That might be ideal if it could be done with upload-artifact and download-artifact, but I think `container` only supports getting images from registries.

As anticipated, it is still needed.

The "error: externally-managed-environment" stoppage occurs even when the Alpine Linux python command is run by a non-root user.

We chown the workspace, so this shouldn't be needed. This commit also removes the "Debug ownership" step.

Byron

Thank you so much for this wonderful addition!

GitPython really is tested rigorously by now with the suite improving over time, given that features aren't usually added, and with the platform coverage increasing. So despite being in maintenance mode, it does manage to maintain and increase its quality. Thank you, again and again, for your tremendous work with this!

I appreciate the early PR and believe this setup can change as it has to, and according to your preferences, in the future without problems.

Reading about having a test-matrix for this, I thought maybe multiple containers can run at the same time so they are pre-started? The CI agent should have enough memory to this.

EliahKagan · 2024-02-21T01:18:55Z

Reading about having a test-matrix for this, I thought maybe multiple contains can run at the same time so they are pre-started? The CI agent should have enough memory to this.

A GitHub Actions job can run multiple containers at a time without problems. This happens automatically when a workflow defines a job using the container key (which causes most steps listed for the job to run in the container it specifies), but also has one or more steps with a uses key specifying a container action. The container specified in the container key or its subkeys is started first, before any steps of any kind are run.

But this is unrelated to the matrix strategy feature, since separate jobs always run on separate runners. (GitHub-hosted runners are virtual machines that are created and destroyed on demand per-job. With self-hosted runners, two jobs may run on the same runner, but not, I don't think, concurrently.)

A job with a container key tries to start that container before it runs any steps, but the GitHub-hosted GHA runners don't have emulation enabled. This can be enabled easily, with a command like:

docker run --rm --privileged multiarch/qemu-user-static --reset -p yes

However, that or a similar command has to actually run on the runner, or its effect otherwise achieved on the runner. Giving it as the value of a run key in a script step would ordinarily achieve this, but not when the job has a container key, which causes most steps, including all steps that run commands with run, to run in the container specified in the container key. In addition, even if there is a way to do it, I'm not sure it would be adequately clear. For this reason, I am unsure if there's any reasonable way to use a container key while having an amd64 runner run an s390x container with QEMU emulation.

Byron · 2024-02-21T20:30:00Z

Thanks for the clarification - it's definitely quite a bit more complicated 😅.

Some of the CI tests use WSL. This switches the WSL distribution from Debian to Alpine, which might be slightly faster. For the way it is being used here, the main expected speed improvement would be to how long the image would take to download, as Alpine is smaller. (The reason for this is thus unrelated to the reason for the Alpine docker CI test job added in gitpython-developers#1826. There, the goal was to test on a wider variety of systems and environments, and that runs the whole test suite in Alpine. This just changes the WSL distro, used by a few tests on Windows, from Debian to Alpine.) Two things have changed that, taken together, have unblocked this: - Vampire/setup-wsl#50 was fixed, so the action we are using is able to install Alpine Linux. See: gitpython-developers#1917 (review) - gitpython-developers#1893 was fixed in gitpython-developers#1888. So if switching the WSL distro from Debian to Alpine breaks any tests, including by making them fail in an unexpected way that raises the wrong exception, we are likely to find out.

EliahKagan added 10 commits February 15, 2024 15:45

Test Alpine Linux on CI

44b8562

With only one version of Python, currently 3.11.

Use venv on Alpine Linux

a45d0b0

To overcome "This environment is externally managed" blocker.

Add back safe.directory step

ab37ae7

As anticipated, it is still needed.

Debug ownership

46e4234

Take ownership of cloned repository

20780cb

Bring back venv

b32932f

The "error: externally-managed-environment" stoppage occurs even when the Alpine Linux python command is run by a non-root user.

Re-remove safe.directory step

bad545a

We chown the workspace, so this shouldn't be needed. This commit also removes the "Debug ownership" step.

Factor venv activation into the venv creation step

4b427c9

EliahKagan force-pushed the docker branch from 0b35598 to 44b8562 Compare February 15, 2024 20:46

EliahKagan marked this pull request as ready for review February 15, 2024 20:54

Byron approved these changes Feb 16, 2024

View reviewed changes

Byron merged commit afa5754 into gitpython-developers:main Feb 16, 2024
26 checks passed

EliahKagan deleted the docker branch February 16, 2024 13:41

This was referenced Jul 24, 2024

Enable Python 3.8 and 3.9 on M1 runners #1944

Merged

Use Alpine Linux in WSL on CI #1945

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Test in Docker with Alpine Linux on CI #1826

Test in Docker with Alpine Linux on CI #1826

EliahKagan commented Feb 15, 2024 •

edited

Loading

Byron left a comment •

edited

Loading

EliahKagan commented Feb 21, 2024 •

edited

Loading

Byron commented Feb 21, 2024

Test in Docker with Alpine Linux on CI #1826

Test in Docker with Alpine Linux on CI #1826

Conversation

EliahKagan commented Feb 15, 2024 • edited Loading

Byron left a comment • edited Loading

Choose a reason for hiding this comment

EliahKagan commented Feb 21, 2024 • edited Loading

Byron commented Feb 21, 2024

EliahKagan commented Feb 15, 2024 •

edited

Loading

Byron left a comment •

edited

Loading

EliahKagan commented Feb 21, 2024 •

edited

Loading