Packaging for ducktape test dependencies

[dotnwat]

One category of test that we run in ducktape is to check for regressions with a range of kafka client versions. Currently this is limited to a few select binary releases of the kafka tools which we can unpack directly into the docker container when it's built. This works well for pretty much any client that is packaged like a binary cli tool (e.g. kafka tools, kaf, kafka-cat, rpk, etc...)

We'd like to expand this capability to libraries. For example by testing a range of sarama, librdkafka, and kafka-python clients. However, these clients don't have generic cli tools. So effectively what I'd do is create a simple wrapper exectuable in go, python, etc... and invoke that, and we'd link it against a range of versions for each library.

The question then for this discussion is how we should go about packaging these. I think the simplest approach is to build each tool as a docker container that we can invoke via python. So it might be something like:

tests/clients/kafka-python/
  - Dockerfile
  - wrapper.py

And then I'd build redpanda-kafka-python for several released versions of the kafka-python library. Repeat for different client libraries. The alternative I suppose would be to install different virtual environments for each case, and figure out a similar approach for other languages like Go.

The dockerfile approach seems like it would work alright for ducktape on cloud vm, but the unknown for me is what options we have for running docker containers from within docker itself for the docker-compose provider. My understanding is that this is possible and as simple as mounting the docker device inside the outer docker container, but I don't know what restrictions we have in our environments. Would something podman work better inside the container? Any other opinions or ideas about how to approach this issue?

CC @ivotron

[ivotron]

Earlier today, we briefly discussed about putting all dependencies in this rather big image. This is currently done at the time the integration tests run, but we could optimize this and build/update it as part of CI (same as we currently do for the builder images). For context, currently, dependencies for integration tests are fetched and installed in the "ducky" docker image (tests/docker/Dockerfile).

This could work in the short term. The problem I see arising is when not all tests are intended to be executed, in which case a build would be downloading the entire image, potentially downloading multi-GBs worth of docker image when only a few bits are needed.

The question then for this discussion is how we should go about packaging these. I think the simplest approach is to build each tool as a docker container that we can invoke via python. So it might be something like:

tests/clients/kafka-python/
  - Dockerfile
 - wrapper.py

I like this idea, as this would deal with the issue I mentioned above (having to download all test dependencies always, when in some cases only a few might be needed). On the other hand, this would complicate the workflow/orchestration.

Another alternative can be to have a single parametrized dockerfile, and selectively install dependencies when integration tests run (as we currently do), based on the given args. This would be more close to what it is done currently, so implementing this option is relatively straight forward.

what do you think?

[dotnwat]

Cool, it sounds like there are a few options. Are any of these things we need to decide on now? It kind of seems like we can expand the dockerfile and until the base image becomes too big and then do something like parameterize the dockerfile.

I guess this is a downside to google cloud builder? If we were running builds on a machine that were persistent then we could share the local docker cache and the image size wouldn't matter?