From 05946a1f619dfaf6f8ee5fa20c5085ec4ad416cb Mon Sep 17 00:00:00 2001 From: Nick Pope Date: Thu, 5 Dec 2024 09:42:28 +0000 Subject: [PATCH 01/48] Removed references to `django-box` from README. The `django-box` repository was archived over 5 years ago and many people coming to this now will have no idea what Vagrant or VirtualBox even are. --- README.md | 19 ++++--------------- 1 file changed, 4 insertions(+), 15 deletions(-) diff --git a/README.md b/README.md index 2ca7cae..6ae6a9a 100644 --- a/README.md +++ b/README.md @@ -4,10 +4,6 @@ Run the django test suite across multiple databases. -Inspired by [django-box](https://github.com/django/django-box) this project contains -some compose definitions to quickly run the Django test suite across multiple Python and -database versions, including Oracle. - ## Quickstart Clone this repository somewhere. Also make sure you have docker and docker-compose installed. @@ -30,7 +26,7 @@ Then simply run: `docker-compose run --rm sqlite` -All arguments are passed to `runtests.py`. Before they are run all specific dependencies are +All arguments are passed to `runtests.py`. Before they are run all specific dependencies are installed (and cached across runs). ## Different databases @@ -64,12 +60,12 @@ You can also pull the pre-built image in the same way: ## Oracle As usual Oracle is a bit more complex to set up. You need to download the latest `instantclient` **zip file** -[from this page](https://www.oracle.com/technetwork/topics/linuxx86-64soft-092277.html) and place it inside the +[from this page](https://www.oracle.com/technetwork/topics/linuxx86-64soft-092277.html) and place it inside the `./oracle` directory. Ensure only one `.zip` file is present. -The database image is quite large (several gigabytes) and takes a fairly long time to initialise (5-10 minutes). +The database image is quite large (several gigabytes) and takes a fairly long time to initialise (5-10 minutes). Once it has initialised subsequent starts should be very quick. To begin this process run: - + `docker-compose run oracle-db` And wait for it to initialize. Once it has, use ctrl+c to terminate it and execute: @@ -101,10 +97,3 @@ To enter a bash shell within the container, run: | `POSTGRES_VERSION` | `13` | The version of Postgres to use | | `MYSQL_VERSION` | `8` | The mysql version to use | | `MARIADB_VERSION` | `10.5` | The mariadb version to use | - - -## Why? - -I prefer using docker over Vagrant and virtualbox, which is what django-box uses. I think this -approach is also simpler to quickly change the various database/python versions the test suite -runs across. However both approaches work, so use whatever floats your boat. From d60cafd6415ed6695be1883f62ebcb12058f6f3c Mon Sep 17 00:00:00 2001 From: Nick Pope Date: Thu, 5 Dec 2024 08:09:11 +0000 Subject: [PATCH 02/48] Removed obsolete configuration for Travis CI. Travis CI was used when GitHub Actions was still very new and had limited support. Since then, Travis CI became less tolerant of open source projects and runtimes increased significantly. GitHub Actions provides much more flexibility and can be looked into as a replacement for this matrix of tests. --- .travis.yml | 88 ----------------------------------------------------- README.md | 2 -- 2 files changed, 90 deletions(-) delete mode 100644 .travis.yml diff --git a/.travis.yml b/.travis.yml deleted file mode 100644 index f1d6e3e..0000000 --- a/.travis.yml +++ /dev/null @@ -1,88 +0,0 @@ -sudo: required -dist: xenial -services: -- docker -python: 3.10 -language: python -cache: pip - -env: -# Linting and docs -- COMPOSE_APP=flake8 -- COMPOSE_APP=docs - -# Quick Runs. Lowest supported database and Python version. -- COMPOSE_APP=sqlite PYTHON_VERSION=3.10 -- COMPOSE_APP=sqlite-gis PYTHON_VERSION=3.10 -- COMPOSE_APP=postgres PYTHON_VERSION=3.10 POSTGRES_VERSION=13 -- COMPOSE_APP=postgres-gis PYTHON_VERSION=3.10 POSTGRES_VERSION=13 POSTGIS_VERSION=3.0 -- COMPOSE_APP=mysql PYTHON_VERSION=3.10 MYSQL_VERSION=8 -- COMPOSE_APP=mysql-gis PYTHON_VERSION=3.10 MYSQL_VERSION=8 -- COMPOSE_APP=mariadb PYTHON_VERSION=3.10 MARIADB_VERSION=10.5 - -# Sqlite -- COMPOSE_APP=sqlite PYTHON_VERSION=3.10 - -# Sqlite gis: -- COMPOSE_APP=sqlite-gis PYTHON_VERSION=3.10 - -# Postgres: -- COMPOSE_APP=postgres PYTHON_VERSION=3.10 POSTGRES_VERSION=13 -- COMPOSE_APP=postgres PYTHON_VERSION=3.10 POSTGRES_VERSION=14 -- COMPOSE_APP=postgres PYTHON_VERSION=3.10 POSTGRES_VERSION=15 -- COMPOSE_APP=postgres PYTHON_VERSION=3.10 POSTGRES_VERSION=16 - -# Postgres gis: -- COMPOSE_APP=postgres-gis PYTHON_VERSION=3.10 POSTGRES_VERSION=13 POSTGIS_VERSION=3.0 -- COMPOSE_APP=postgres-gis PYTHON_VERSION=3.10 POSTGRES_VERSION=14 POSTGIS_VERSION=3.1 -- COMPOSE_APP=postgres-gis PYTHON_VERSION=3.10 POSTGRES_VERSION=15 POSTGIS_VERSION=3.3 -- COMPOSE_APP=postgres-gis PYTHON_VERSION=3.10 POSTGRES_VERSION=16 POSTGIS_VERSION=3.4 - -# MySQL: -- COMPOSE_APP=mysql PYTHON_VERSION=3.10 MYSQL_VERSION=8.0 - -# MySQL gis: -- COMPOSE_APP=mysql-gis PYTHON_VERSION=3.10 MYSQL_VERSION=8.0 - -# MariaDB: -- COMPOSE_APP=mariadb PYTHON_VERSION=3.10 MARIADB_VERSION=10.5 -- COMPOSE_APP=mariadb PYTHON_VERSION=3.10 MARIADB_VERSION=10.6 -- COMPOSE_APP=mariadb PYTHON_VERSION=3.10 MARIADB_VERSION=10.11 -- COMPOSE_APP=mariadb PYTHON_VERSION=3.10 MARIADB_VERSION=11.0 -- COMPOSE_APP=mariadb PYTHON_VERSION=3.10 MARIADB_VERSION=11.1 - -# Browsers: -- COMPOSE_APP=chrome PYTHON_VERSION=3.10 -- COMPOSE_APP=firefox PYTHON_VERSION=3.10 - -matrix: - allow_failures: - - env: COMPOSE_APP=firefox PYTHON_VERSION=3.10 - -before_install: - - echo 'DOCKER_OPTS="${DOCKER_OPTS} --registry-mirror=https://mirror.gcr.io"' | sudo tee -a /etc/default/docker - - sudo service docker restart - -install: -- pip install docker-compose --upgrade - -script: -- git clone https://github.com/django/django.git --depth=1 /tmp/django -- export DJANGO_PATH=/tmp/django -- docker-compose pull --include-deps $COMPOSE_APP || true -- docker-compose build --pull $COMPOSE_APP -- | - if [[ $COMPOSE_APP == *-gis ]]; then - docker-compose run $COMPOSE_APP gis_tests - else - docker-compose run $COMPOSE_APP - fi - -after_failure: - - docker-compose logs - -deploy: - provider: script - script: bash docker_push.sh $COMPOSE_APP - on: - branch: main diff --git a/README.md b/README.md index 6ae6a9a..42147ac 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,5 @@ # django-docker-box -[![Build Status](https://travis-ci.org/django/django-docker-box.svg?branch=main)](https://travis-ci.org/django/django-docker-box) - Run the django test suite across multiple databases. ## Quickstart From fe49569328ec4cb7fd8d833e82c5a5b9e9af4698 Mon Sep 17 00:00:00 2001 From: Nick Pope Date: Thu, 5 Dec 2024 08:09:13 +0000 Subject: [PATCH 03/48] Removed obsolete configuration for Azure Pipelines. This looks incomplete and was likely considered as an alternative to Travis CI at some point. We should look into GitHub Actions as a replacement for running a matrix of tests. --- .azure.yml | 8 -------- 1 file changed, 8 deletions(-) delete mode 100644 .azure.yml diff --git a/.azure.yml b/.azure.yml deleted file mode 100644 index b82fd74..0000000 --- a/.azure.yml +++ /dev/null @@ -1,8 +0,0 @@ -pool: - vmImage: 'vs2017-win2016' - -steps: -- script: | - docker pull microsoft/windowsservercore - docker-compose --version - displayName: Test From cc4dfaf2ed3128ad83c41580ab2c4e4d61081633 Mon Sep 17 00:00:00 2001 From: Nick Pope Date: Thu, 5 Dec 2024 08:09:15 +0000 Subject: [PATCH 04/48] Removed obsolete script for pushing images to Docker Hub. Ownership of the repository for images on Docker Hub is likely in the hands of the original author rather than the Django project itself. In addition, we should probably look at making use of the GitHub Container Repository instead which will be easier to manage and control. --- docker_push.sh | 4 ---- 1 file changed, 4 deletions(-) delete mode 100644 docker_push.sh diff --git a/docker_push.sh b/docker_push.sh deleted file mode 100644 index 915046e..0000000 --- a/docker_push.sh +++ /dev/null @@ -1,4 +0,0 @@ -#!/usr/bin/env bash - -echo "${DOCKER_PASSWORD}" | docker login -u "${DOCKER_USERNAME}" --password-stdin -docker-compose push $@ From ef7421b8b3f12242be4efd6718e31ba7da2d7d9a Mon Sep 17 00:00:00 2001 From: Nick Pope Date: Thu, 5 Dec 2024 08:12:55 +0000 Subject: [PATCH 05/48] Removed broken support for Oracle Database. We will be able to support Oracle again now that they have started publishing container images themselves via their own repository. It's easier to remove this broken support and start again, however, as - aside from using a very old version of Oracle Database - this approach required many hacks. --- .gitignore | 1 - README.md | 17 -------------- docker-compose.yml | 26 --------------------- oracle/.gitkeep | 0 oracle_entrypoint.sh | 12 ---------- settings/test_oracle.py | 52 ----------------------------------------- 6 files changed, 108 deletions(-) delete mode 100644 oracle/.gitkeep delete mode 100755 oracle_entrypoint.sh delete mode 100644 settings/test_oracle.py diff --git a/.gitignore b/.gitignore index c3eed66..9f11b75 100644 --- a/.gitignore +++ b/.gitignore @@ -1,2 +1 @@ .idea/ -oracle/*.zip \ No newline at end of file diff --git a/README.md b/README.md index 42147ac..7d79675 100644 --- a/README.md +++ b/README.md @@ -55,23 +55,6 @@ You can also pull the pre-built image in the same way: `PYTHON_VERSION=3.10 docker-compose pull sqlite` -## Oracle - -As usual Oracle is a bit more complex to set up. You need to download the latest `instantclient` **zip file** -[from this page](https://www.oracle.com/technetwork/topics/linuxx86-64soft-092277.html) and place it inside the -`./oracle` directory. Ensure only one `.zip` file is present. - -The database image is quite large (several gigabytes) and takes a fairly long time to initialise (5-10 minutes). -Once it has initialised subsequent starts should be very quick. To begin this process run: - - `docker-compose run oracle-db` - -And wait for it to initialize. Once it has, use ctrl+c to terminate it and execute: - -`docker-compose run --rm oracle` - -To run the test suite against it. All other databases support different versions, however Oracle does not. - ## Utilities To run the docs spellchecker: diff --git a/docker-compose.yml b/docker-compose.yml index 638f883..c8b0d16 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -12,9 +12,6 @@ x-base: &base volumes: - ${DJANGO_PATH}:/tests/django/ - ./settings:/tests/django/tests/settings - - ./oracle_entrypoint.sh:/oracle_entrypoint.sh - # Using yaml merging here overwrites the arrays. Simpler to just mount this in every service. - - ./oracle:/oracle entrypoint: bash -c "wait-for-it.sh -s -t 20 $${WAIT_FOR} -- python runtests.py $$@" -- x-postgres-base: &postgres-base @@ -118,24 +115,6 @@ services: - MYSQL_DATABASE=django - MYSQL_ALLOW_EMPTY_PASSWORD=1 - oracle: - <<: *base - image: djangobox/django-docker-box:oracle-${PYTHON_VERSION} - environment: - - DJANGO_SETTINGS_MODULE=settings.test_oracle - - WAIT_FOR=oracle-db:1521 - entrypoint: /oracle_entrypoint.sh python runtests.py - depends_on: - - oracle-db - - memcached - - memcached2 - oracle-db: - image: sath89/oracle-12c - volumes: - - oracle:/u01/app/oracle/ - environment: - - WEB_CONSOLE=false - memcached: &memcached image: memcached:alpine @@ -177,8 +156,3 @@ services: image: selenium/standalone-firefox volumes: - /dev/shm:/dev/shm - - -volumes: - oracle: - diff --git a/oracle/.gitkeep b/oracle/.gitkeep deleted file mode 100644 index e69de29..0000000 diff --git a/oracle_entrypoint.sh b/oracle_entrypoint.sh deleted file mode 100755 index 18a333b..0000000 --- a/oracle_entrypoint.sh +++ /dev/null @@ -1,12 +0,0 @@ -#!/usr/bin/env bash - -set -e -IFS=$'\n\t' - -if compgen -G "/oracle/*.zip" > /dev/null; then - sudo mkdir -p /opt/oracle - sudo unzip /oracle/*.zip -d /opt/oracle - sudo bash -c "echo /opt/oracle/instantclient* > /etc/ld.so.conf.d/oracle-instantclient.conf" - sudo ldconfig -fi -exec $@ diff --git a/settings/test_oracle.py b/settings/test_oracle.py deleted file mode 100644 index 99e4775..0000000 --- a/settings/test_oracle.py +++ /dev/null @@ -1,52 +0,0 @@ -DATABASES = { - 'default': { - 'ENGINE': 'django.db.backends.oracle', - 'NAME': 'oracle-db:1521/xe', - 'USER': 'system', - 'PASSWORD': 'oracle', - 'TEST': { - 'USER': 'default_test', - 'TBLSPACE': 'default_test_tbls', - 'TBLSPACE_TMP': 'default_test_tbls_tmp', - }, - }, - 'other': { - 'ENGINE': 'django.db.backends.oracle', - 'NAME': 'oracle-db:1521/xe', - 'USER': 'system', - 'PASSWORD': 'oracle', - 'TEST': { - 'USER': 'other_test', - 'TBLSPACE': 'other_test_tbls', - 'TBLSPACE_TMP': 'other_test_tbls_tmp', - }, - }, -} - -SECRET_KEY = "django_tests_secret_key" - -# Use a fast hasher to speed up tests. -PASSWORD_HASHERS = [ - 'django.contrib.auth.hashers.MD5PasswordHasher', -] - -CACHES = { - 'default': { - 'BACKEND': 'django.core.cache.backends.locmem.LocMemCache', - }, - 'pymemcache': { - 'BACKEND': 'django.core.cache.backends.memcached.PyMemcacheCache', - 'LOCATION': 'memcached:11211', - 'KEY_PREFIX': 'pymemcache:', - }, - 'pylibmc': { - 'BACKEND': 'django.core.cache.backends.memcached.PyLibMCCache', - 'LOCATION': 'memcached2:11211', - 'KEY_PREFIX': 'pylibmc:', - }, -} - -TEST_RUNNER = 'xmlrunner.extra.djangotestrunner.XMLTestRunner' -TEST_OUTPUT_DIR = '/tests/results/' -DEFAULT_AUTO_FIELD = 'django.db.models.AutoField' -USE_TZ = False From 7c9db3b2414d899a9c0f52b880e813d8dad9c4b7 Mon Sep 17 00:00:00 2001 From: Nick Pope Date: Thu, 5 Dec 2024 08:17:16 +0000 Subject: [PATCH 06/48] Removed obsolete version attribute from compose file. This raises a warning when running `docker compose`, so remove it. --- docker-compose.yml | 2 -- 1 file changed, 2 deletions(-) diff --git a/docker-compose.yml b/docker-compose.yml index c8b0d16..e7e2145 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -1,5 +1,3 @@ -version: "2.3" - x-base: &base image: djangobox/django-docker-box:${PYTHON_VERSION} build: From 547e86793554329e3facae527e7d1f84259983f9 Mon Sep 17 00:00:00 2001 From: Nick Pope Date: Thu, 5 Dec 2024 08:34:42 +0000 Subject: [PATCH 07/48] Removed obsolete `GEOIP_PATH` setting. Django now bundles test Maxmind GeoIP and GeoLite databases which means that this setting - and a manual download of these databases - is no longer required. --- settings/test_mysql_gis.py | 2 -- settings/test_postgres_gis.py | 2 -- settings/test_sqlite_gis.py | 2 -- 3 files changed, 6 deletions(-) diff --git a/settings/test_mysql_gis.py b/settings/test_mysql_gis.py index 6185cd1..92220bf 100644 --- a/settings/test_mysql_gis.py +++ b/settings/test_mysql_gis.py @@ -28,8 +28,6 @@ 'django.contrib.auth.hashers.MD5PasswordHasher', ] -GEOIP_PATH = '/geolite2/' - TEST_RUNNER = 'xmlrunner.extra.djangotestrunner.XMLTestRunner' TEST_OUTPUT_DIR = '/tests/results/' DEFAULT_AUTO_FIELD = 'django.db.models.AutoField' diff --git a/settings/test_postgres_gis.py b/settings/test_postgres_gis.py index 1e70561..57bcf7a 100644 --- a/settings/test_postgres_gis.py +++ b/settings/test_postgres_gis.py @@ -36,8 +36,6 @@ 'django.contrib.auth.hashers.MD5PasswordHasher', ] -GEOIP_PATH = '/geolite2/' - TEST_RUNNER = 'xmlrunner.extra.djangotestrunner.XMLTestRunner' TEST_OUTPUT_DIR = '/tests/results' DEFAULT_AUTO_FIELD = 'django.db.models.AutoField' diff --git a/settings/test_sqlite_gis.py b/settings/test_sqlite_gis.py index 55173c1..4587b4b 100644 --- a/settings/test_sqlite_gis.py +++ b/settings/test_sqlite_gis.py @@ -28,8 +28,6 @@ 'django.contrib.auth.hashers.MD5PasswordHasher', ] -GEOIP_PATH = '/geolite2/' - TEST_RUNNER = 'xmlrunner.extra.djangotestrunner.XMLTestRunner' TEST_OUTPUT_DIR = '/tests/results/' DEFAULT_AUTO_FIELD = 'django.db.models.AutoField' From 0f61241de2786d40ec198c464d3b809ea92ec854 Mon Sep 17 00:00:00 2001 From: Nick Pope Date: Thu, 5 Dec 2024 09:38:01 +0000 Subject: [PATCH 08/48] Removed unused `PIP_CACHE_VOLUME` environment variable. Unused since 9c8e95f66a96eb84906387f5162690df503518d5. --- .env | 1 - 1 file changed, 1 deletion(-) diff --git a/.env b/.env index 9467ae3..a33865a 100644 --- a/.env +++ b/.env @@ -3,4 +3,3 @@ POSTGRES_VERSION=13 POSTGIS_VERSION=3.0 MYSQL_VERSION=8 MARIADB_VERSION=10.5 -PIP_CACHE_VOLUME=pip-cache From 701563171ee6562f6ed715abe2f530215fe2edc6 Mon Sep 17 00:00:00 2001 From: Nick Pope Date: Thu, 5 Dec 2024 08:21:21 +0000 Subject: [PATCH 09/48] Renamed docker and compose files. Instead of `docker-compose.yml`, rename to `compose.yml` as supported by the Compose Specification. Also rename to `Containerfile` instead of using `Dockerfile` to be more tool-agnostic. It would be good to also support Podman if and when it supports all of the features we require. --- Dockerfile => Containerfile | 0 docker-compose.yml => compose.yml | 2 +- 2 files changed, 1 insertion(+), 1 deletion(-) rename Dockerfile => Containerfile (100%) rename docker-compose.yml => compose.yml (98%) diff --git a/Dockerfile b/Containerfile similarity index 100% rename from Dockerfile rename to Containerfile diff --git a/docker-compose.yml b/compose.yml similarity index 98% rename from docker-compose.yml rename to compose.yml index e7e2145..dded122 100644 --- a/docker-compose.yml +++ b/compose.yml @@ -2,7 +2,7 @@ x-base: &base image: djangobox/django-docker-box:${PYTHON_VERSION} build: context: ${DJANGO_PATH} - dockerfile: ${PWD}/Dockerfile + dockerfile: ${PWD}/Containerfile args: - PYTHON_VERSION=${PYTHON_VERSION} cache_from: From 8204a3c761f629a5d93377cedbe6db7147d05596 Mon Sep 17 00:00:00 2001 From: Nick Pope Date: Thu, 5 Dec 2024 09:04:46 +0000 Subject: [PATCH 10/48] Consolidated settings into a single file. Most settings are exactly the same, so by dynamically generating the `DATABASES` setting we can ensure greater consistency and reduce the maintenance required. The `TEST_RUNNER` and `TEST_OUTPUT_DIR` settings have been hidden behind an `XUNIT` environment variable. These are typically only useful in CI, specifically on Jenkins, suppress some output, and do not support various features such as debugging SQL output and test durations. --- Containerfile | 1 + compose.yml | 23 +++++++++----- settings.py | 58 +++++++++++++++++++++++++++++++++++ settings/__init__.py | 0 settings/test_mariadb.py | 50 ------------------------------ settings/test_mysql.py | 50 ------------------------------ settings/test_mysql_gis.py | 34 -------------------- settings/test_postgres.py | 58 ----------------------------------- settings/test_postgres_gis.py | 42 ------------------------- settings/test_sqlite.py | 51 ------------------------------ settings/test_sqlite_gis.py | 34 -------------------- 11 files changed, 74 insertions(+), 327 deletions(-) create mode 100644 settings.py delete mode 100644 settings/__init__.py delete mode 100644 settings/test_mariadb.py delete mode 100644 settings/test_mysql.py delete mode 100644 settings/test_mysql_gis.py delete mode 100644 settings/test_postgres.py delete mode 100644 settings/test_postgres_gis.py delete mode 100644 settings/test_sqlite.py delete mode 100644 settings/test_sqlite_gis.py diff --git a/Containerfile b/Containerfile index 3dc2c7a..d5ed98a 100644 --- a/Containerfile +++ b/Containerfile @@ -32,6 +32,7 @@ RUN for f in /requirements/*.txt; do pip install -r $f; done && \ RUN mkdir /tests && chown -R test:test /tests RUN mkdir /tests/results && chown -R test:test /tests/results/ USER test:test +ENV DJANGO_SETTINGS_MODULE=settings ENV PYTHONPATH "${PYTHONPATH}:/tests/django/" VOLUME /tests/django WORKDIR /tests/django/tests diff --git a/compose.yml b/compose.yml index dded122..3ce7f23 100644 --- a/compose.yml +++ b/compose.yml @@ -9,7 +9,7 @@ x-base: &base - djangobox/django-docker-box:${PYTHON_VERSION} volumes: - ${DJANGO_PATH}:/tests/django/ - - ./settings:/tests/django/tests/settings + - ./settings.py:/tests/django/tests/settings.py entrypoint: bash -c "wait-for-it.sh -s -t 20 $${WAIT_FOR} -- python runtests.py $$@" -- x-postgres-base: &postgres-base @@ -26,7 +26,7 @@ services: sqlite: <<: *base environment: - - DJANGO_SETTINGS_MODULE=settings.test_sqlite + - DATABASE_ENGINE=django.db.backends.sqlite3 entrypoint: python runtests.py depends_on: - memcached @@ -36,12 +36,13 @@ services: <<: *base entrypoint: python runtests.py environment: - - DJANGO_SETTINGS_MODULE=settings.test_sqlite_gis + - DATABASE_ENGINE=django.contrib.gis.db.backends.spatialite postgres: <<: *base environment: - - DJANGO_SETTINGS_MODULE=settings.test_postgres + - DATABASE_ENGINE=django.db.backends.postgresql + - DATABASE_HOST=postgres-db - WAIT_FOR=postgres-db:5432 depends_on: - postgres-db @@ -54,7 +55,8 @@ services: postgres-gis: <<: *base environment: - - DJANGO_SETTINGS_MODULE=settings.test_postgres_gis + - DATABASE_ENGINE=django.contrib.gis.db.backends.postgis + - DATABASE_HOST=postgres-gis-db - WAIT_FOR=postgres-gis-db:5432 depends_on: - postgres-gis-db @@ -65,7 +67,8 @@ services: mysql: <<: *base environment: - - DJANGO_SETTINGS_MODULE=settings.test_mysql + - DATABASE_ENGINE=django.db.backends.mysql + - DATABASE_HOST=mysql-db - WAIT_FOR=mysql-db:3306 depends_on: - mysql-db @@ -85,7 +88,8 @@ services: mysql-gis: <<: *base environment: - - DJANGO_SETTINGS_MODULE=settings.test_mysql_gis + - DATABASE_ENGINE=django.contrib.gis.db.backends.mysql + - DATABASE_HOST=mysql-gis-db - WAIT_FOR=mysql-gis-db:3306 depends_on: - mysql-gis-db @@ -97,7 +101,8 @@ services: mariadb: <<: *base environment: - - DJANGO_SETTINGS_MODULE=settings.test_mariadb + - DATABASE_ENGINE=django.db.backends.mysql + - DATABASE_HOST=mariadb-db - WAIT_FOR=mariadb-db:3306 depends_on: - mariadb-db @@ -134,6 +139,7 @@ services: <<: *base command: --selenium=chrome --selenium-hub=http://chrome-browser:4444/wd/hub environment: + - DATABASE_ENGINE=django.db.backends.sqlite3 - WAIT_FOR=chrome-browser:4444 depends_on: - chrome-browser @@ -147,6 +153,7 @@ services: <<: *base command: --selenium=firefox --selenium-hub=http://firefox-browser:4444/wd/hub environment: + - DATABASE_ENGINE=django.db.backends.sqlite3 - WAIT_FOR=firefox-browser:4444 depends_on: - firefox-browser diff --git a/settings.py b/settings.py new file mode 100644 index 0000000..1336ac7 --- /dev/null +++ b/settings.py @@ -0,0 +1,58 @@ +import os + + +def _build_databases_setting(): + engine = os.environ["DATABASE_ENGINE"] + host = os.environ.get("DATABASE_HOST", "") + settings = {} + + for n, alias in enumerate(("default", "other"), start=1): + settings[alias] = entry = {"ENGINE": engine} + + if not engine.endswith((".sqlite", ".spatialite")): + entry |= { + "HOST": host, + "NAME": "django" if n < 2 else f"django{n}", + "USER": "django", + "PASSWORD": "django", + } + + if engine.endswith(".mysql"): + entry |= { + "USER": "root", + "PASSWORD": "", + "TEST": {"CHARSET": "utf8"}, + } + + return settings + + +CACHES = { + "default": { + "BACKEND": "django.core.cache.backends.locmem.LocMemCache", + }, + "pymemcache": { + "BACKEND": "django.core.cache.backends.memcached.PyMemcacheCache", + "LOCATION": "memcached:11211", + "KEY_PREFIX": "pymemcache:", + }, + "pylibmc": { + "BACKEND": "django.core.cache.backends.memcached.PyLibMCCache", + "LOCATION": "memcached2:11211", + "KEY_PREFIX": "pylibmc:", + }, +} + +DATABASES = _build_databases_setting() + +DEFAULT_AUTO_FIELD = "django.db.models.AutoField" + +PASSWORD_HASHERS = ["django.contrib.auth.hashers.MD5PasswordHasher"] + +SECRET_KEY = "django_tests_secret_key" + +USE_TZ = False + +if os.environ.get("XUNIT", "0").lower() in {"1", "on", "true", "yes"}: + TEST_RUNNER = "xmlrunner.extra.djangotestrunner.XMLTestRunner" + TEST_OUTPUT_DIR = "/tests/results" diff --git a/settings/__init__.py b/settings/__init__.py deleted file mode 100644 index e69de29..0000000 diff --git a/settings/test_mariadb.py b/settings/test_mariadb.py deleted file mode 100644 index 82069ac..0000000 --- a/settings/test_mariadb.py +++ /dev/null @@ -1,50 +0,0 @@ -DATABASES = { - 'default': { - 'ENGINE': 'django.db.backends.mysql', - 'USER': 'root', - 'PASSWORD': '', - 'NAME': 'django', - 'HOST': 'mariadb-db', - 'TEST': { - 'CHARSET': 'utf8', - }, - }, - 'other': { - 'ENGINE': 'django.db.backends.mysql', - 'USER': 'root', - 'PASSWORD': '', - 'NAME': 'django2', - 'HOST': 'mariadb-db', - 'TEST': { - 'CHARSET': 'utf8', - }, - }, -} - -SECRET_KEY = "django_tests_secret_key" - -# Use a fast hasher to speed up tests. -PASSWORD_HASHERS = [ - 'django.contrib.auth.hashers.MD5PasswordHasher', -] - -CACHES = { - 'default': { - 'BACKEND': 'django.core.cache.backends.locmem.LocMemCache', - }, - 'pymemcache': { - 'BACKEND': 'django.core.cache.backends.memcached.PyMemcacheCache', - 'LOCATION': 'memcached:11211', - 'KEY_PREFIX': 'pymemcache:', - }, - 'pylibmc': { - 'BACKEND': 'django.core.cache.backends.memcached.PyLibMCCache', - 'LOCATION': 'memcached2:11211', - 'KEY_PREFIX': 'pylibmc:', - }, -} - -TEST_RUNNER = 'xmlrunner.extra.djangotestrunner.XMLTestRunner' -TEST_OUTPUT_DIR = '/tests/results/' -DEFAULT_AUTO_FIELD = 'django.db.models.AutoField' -USE_TZ = False diff --git a/settings/test_mysql.py b/settings/test_mysql.py deleted file mode 100644 index b05ace0..0000000 --- a/settings/test_mysql.py +++ /dev/null @@ -1,50 +0,0 @@ -DATABASES = { - 'default': { - 'ENGINE': 'django.db.backends.mysql', - 'USER': 'root', - 'PASSWORD': '', - 'NAME': 'django', - 'HOST': 'mysql-db', - 'TEST': { - 'CHARSET': 'utf8', - }, - }, - 'other': { - 'ENGINE': 'django.db.backends.mysql', - 'USER': 'root', - 'PASSWORD': '', - 'NAME': 'django2', - 'HOST': 'mysql-db', - 'TEST': { - 'CHARSET': 'utf8', - }, - }, -} - -SECRET_KEY = "django_tests_secret_key" - -# Use a fast hasher to speed up tests. -PASSWORD_HASHERS = [ - 'django.contrib.auth.hashers.MD5PasswordHasher', -] - -CACHES = { - 'default': { - 'BACKEND': 'django.core.cache.backends.locmem.LocMemCache', - }, - 'pymemcache': { - 'BACKEND': 'django.core.cache.backends.memcached.PyMemcacheCache', - 'LOCATION': 'memcached:11211', - 'KEY_PREFIX': 'pymemcache:', - }, - 'pylibmc': { - 'BACKEND': 'django.core.cache.backends.memcached.PyLibMCCache', - 'LOCATION': 'memcached2:11211', - 'KEY_PREFIX': 'pylibmc:', - }, -} - -TEST_RUNNER = 'xmlrunner.extra.djangotestrunner.XMLTestRunner' -TEST_OUTPUT_DIR = '/tests/results/' -DEFAULT_AUTO_FIELD = 'django.db.models.AutoField' -USE_TZ = False diff --git a/settings/test_mysql_gis.py b/settings/test_mysql_gis.py deleted file mode 100644 index 92220bf..0000000 --- a/settings/test_mysql_gis.py +++ /dev/null @@ -1,34 +0,0 @@ -DATABASES = { - 'default': { - 'ENGINE': 'django.contrib.gis.db.backends.mysql', - 'USER': 'root', - 'PASSWORD': '', - 'NAME': 'django', - 'HOST': 'mysql-gis-db', - 'TEST': { - 'CHARSET': 'utf8', - }, - }, - 'other': { - 'ENGINE': 'django.contrib.gis.db.backends.mysql', - 'USER': 'root', - 'PASSWORD': '', - 'NAME': 'django2', - 'HOST': 'mysql-gis-db', - 'TEST': { - 'CHARSET': 'utf8', - }, - }, -} - -SECRET_KEY = "django_tests_secret_key" - -# Use a fast hasher to speed up tests. -PASSWORD_HASHERS = [ - 'django.contrib.auth.hashers.MD5PasswordHasher', -] - -TEST_RUNNER = 'xmlrunner.extra.djangotestrunner.XMLTestRunner' -TEST_OUTPUT_DIR = '/tests/results/' -DEFAULT_AUTO_FIELD = 'django.db.models.AutoField' -USE_TZ = False diff --git a/settings/test_postgres.py b/settings/test_postgres.py deleted file mode 100644 index 85148ba..0000000 --- a/settings/test_postgres.py +++ /dev/null @@ -1,58 +0,0 @@ -# This is an example test settings file for use with the Django test suite. -# -# The 'sqlite3' backend requires only the ENGINE setting (an in- -# memory database will be used). All other backends will require a -# NAME and potentially authentication information. See the -# following section in the docs for more information: -# -# https://docs.djangoproject.com/en/dev/internals/contributing/writing-code/unit-tests/ -# -# The different databases that Django supports behave differently in certain -# situations, so it is recommended to run the test suite against as many -# database backends as possible. You may want to create a separate settings -# file for each of the backends you test against. - -DATABASES = { - 'default': { - 'ENGINE': 'django.db.backends.postgresql', - 'USER': 'django', - 'PASSWORD': 'django', - 'NAME': 'django', - 'HOST': 'postgres-db' - }, - 'other': { - 'ENGINE': 'django.db.backends.postgresql', - 'USER': 'django', - 'NAME': 'django2', - 'PASSWORD': 'django', - 'HOST': 'postgres-db' - } -} - -SECRET_KEY = "django_tests_secret_key" - -# Use a fast hasher to speed up tests. -PASSWORD_HASHERS = [ - 'django.contrib.auth.hashers.MD5PasswordHasher', -] - -CACHES = { - 'default': { - 'BACKEND': 'django.core.cache.backends.locmem.LocMemCache', - }, - 'pymemcache': { - 'BACKEND': 'django.core.cache.backends.memcached.PyMemcacheCache', - 'LOCATION': 'memcached:11211', - 'KEY_PREFIX': 'pymemcache:', - }, - 'pylibmc': { - 'BACKEND': 'django.core.cache.backends.memcached.PyLibMCCache', - 'LOCATION': 'memcached2:11211', - 'KEY_PREFIX': 'pylibmc:', - }, -} - -TEST_RUNNER = 'xmlrunner.extra.djangotestrunner.XMLTestRunner' -TEST_OUTPUT_DIR = '/tests/results/' -DEFAULT_AUTO_FIELD = 'django.db.models.AutoField' -USE_TZ = False diff --git a/settings/test_postgres_gis.py b/settings/test_postgres_gis.py deleted file mode 100644 index 57bcf7a..0000000 --- a/settings/test_postgres_gis.py +++ /dev/null @@ -1,42 +0,0 @@ -# This is an example test settings file for use with the Django test suite. -# -# The 'sqlite3' backend requires only the ENGINE setting (an in- -# memory database will be used). All other backends will require a -# NAME and potentially authentication information. See the -# following section in the docs for more information: -# -# https://docs.djangoproject.com/en/dev/internals/contributing/writing-code/unit-tests/ -# -# The different databases that Django supports behave differently in certain -# situations, so it is recommended to run the test suite against as many -# database backends as possible. You may want to create a separate settings -# file for each of the backends you test against. - -DATABASES = { - 'default': { - 'ENGINE': 'django.contrib.gis.db.backends.postgis', - 'USER': 'django', - 'PASSWORD': 'django', - 'NAME': 'django', - 'HOST': 'postgres-gis-db' - }, - 'other': { - 'ENGINE': 'django.contrib.gis.db.backends.postgis', - 'USER': 'django', - 'NAME': 'django2', - 'PASSWORD': 'django', - 'HOST': 'postgres-gis-db' - } -} - -SECRET_KEY = "django_tests_secret_key" - -# Use a fast hasher to speed up tests. -PASSWORD_HASHERS = [ - 'django.contrib.auth.hashers.MD5PasswordHasher', -] - -TEST_RUNNER = 'xmlrunner.extra.djangotestrunner.XMLTestRunner' -TEST_OUTPUT_DIR = '/tests/results' -DEFAULT_AUTO_FIELD = 'django.db.models.AutoField' -USE_TZ = False diff --git a/settings/test_sqlite.py b/settings/test_sqlite.py deleted file mode 100644 index 1974bec..0000000 --- a/settings/test_sqlite.py +++ /dev/null @@ -1,51 +0,0 @@ -# This is an example test settings file for use with the Django test suite. -# -# The 'sqlite3' backend requires only the ENGINE setting (an in- -# memory database will be used). All other backends will require a -# NAME and potentially authentication information. See the -# following section in the docs for more information: -# -# https://docs.djangoproject.com/en/dev/internals/contributing/writing-code/unit-tests/ -# -# The different databases that Django supports behave differently in certain -# situations, so it is recommended to run the test suite against as many -# database backends as possible. You may want to create a separate settings -# file for each of the backends you test against. - -DATABASES = { - 'default': { - 'ENGINE': 'django.db.backends.sqlite3', - }, - 'other': { - 'ENGINE': 'django.db.backends.sqlite3', - } -} - -SECRET_KEY = "django_tests_secret_key" - -# Use a fast hasher to speed up tests. -PASSWORD_HASHERS = [ - 'django.contrib.auth.hashers.MD5PasswordHasher', -] - - -CACHES = { - 'default': { - 'BACKEND': 'django.core.cache.backends.locmem.LocMemCache', - }, - 'pymemcache': { - 'BACKEND': 'django.core.cache.backends.memcached.PyMemcacheCache', - 'LOCATION': 'memcached:11211', - 'KEY_PREFIX': 'pymemcache:', - }, - 'pylibmc': { - 'BACKEND': 'django.core.cache.backends.memcached.PyLibMCCache', - 'LOCATION': 'memcached2:11211', - 'KEY_PREFIX': 'pylibmc:', - }, -} - -TEST_RUNNER = 'xmlrunner.extra.djangotestrunner.XMLTestRunner' -TEST_OUTPUT_DIR = '/tests/results' -DEFAULT_AUTO_FIELD = 'django.db.models.AutoField' -USE_TZ = False diff --git a/settings/test_sqlite_gis.py b/settings/test_sqlite_gis.py deleted file mode 100644 index 4587b4b..0000000 --- a/settings/test_sqlite_gis.py +++ /dev/null @@ -1,34 +0,0 @@ -# This is an example test settings file for use with the Django test suite. -# -# The 'sqlite3' backend requires only the ENGINE setting (an in- -# memory database will be used). All other backends will require a -# NAME and potentially authentication information. See the -# following section in the docs for more information: -# -# https://docs.djangoproject.com/en/dev/internals/contributing/writing-code/unit-tests/ -# -# The different databases that Django supports behave differently in certain -# situations, so it is recommended to run the test suite against as many -# database backends as possible. You may want to create a separate settings -# file for each of the backends you test against. - -DATABASES = { - 'default': { - 'ENGINE': 'django.contrib.gis.db.backends.spatialite', - }, - 'other': { - 'ENGINE': 'django.contrib.gis.db.backends.spatialite', - } -} - -SECRET_KEY = "django_tests_secret_key" - -# Use a fast hasher to speed up tests. -PASSWORD_HASHERS = [ - 'django.contrib.auth.hashers.MD5PasswordHasher', -] - -TEST_RUNNER = 'xmlrunner.extra.djangotestrunner.XMLTestRunner' -TEST_OUTPUT_DIR = '/tests/results/' -DEFAULT_AUTO_FIELD = 'django.db.models.AutoField' -USE_TZ = False From 407c2100e380295e57a06b68b56ade3c87d81110 Mon Sep 17 00:00:00 2001 From: Nick Pope Date: Thu, 5 Dec 2024 23:32:20 +0000 Subject: [PATCH 11/48] Preferred "PostgreSQL" over "Postgres". All other database use the full name, so do the same for PostgreSQL. --- .env | 2 +- README.md | 2 +- compose.yml | 30 +++++++++++++++--------------- 3 files changed, 17 insertions(+), 17 deletions(-) diff --git a/.env b/.env index a33865a..62d48a4 100644 --- a/.env +++ b/.env @@ -1,5 +1,5 @@ PYTHON_VERSION=3.10 -POSTGRES_VERSION=13 +POSTGRESQL_VERSION=13 POSTGIS_VERSION=3.0 MYSQL_VERSION=8 MARIADB_VERSION=10.5 diff --git a/README.md b/README.md index 7d79675..5244747 100644 --- a/README.md +++ b/README.md @@ -75,6 +75,6 @@ To enter a bash shell within the container, run: | --- | --- | --- | | `DJANGO_PATH` | None | The path to the Django codebase on your local machine | | `PYTHON_VERSION` | `3.10` | The python version to run tests against | -| `POSTGRES_VERSION` | `13` | The version of Postgres to use | +| `POSTGRESQL_VERSION` | `13` | The version of Postgres to use | | `MYSQL_VERSION` | `8` | The mysql version to use | | `MARIADB_VERSION` | `10.5` | The mariadb version to use | diff --git a/compose.yml b/compose.yml index 3ce7f23..e0bb328 100644 --- a/compose.yml +++ b/compose.yml @@ -12,7 +12,7 @@ x-base: &base - ./settings.py:/tests/django/tests/settings.py entrypoint: bash -c "wait-for-it.sh -s -t 20 $${WAIT_FOR} -- python runtests.py $$@" -- -x-postgres-base: &postgres-base +x-postgresql-base: &postgresql-base environment: - POSTGRES_USER=django - POSTGRES_PASSWORD=django @@ -38,31 +38,31 @@ services: environment: - DATABASE_ENGINE=django.contrib.gis.db.backends.spatialite - postgres: + postgresql: <<: *base environment: - DATABASE_ENGINE=django.db.backends.postgresql - - DATABASE_HOST=postgres-db - - WAIT_FOR=postgres-db:5432 + - DATABASE_HOST=postgresql-db + - WAIT_FOR=postgresql-db:5432 depends_on: - - postgres-db + - postgresql-db - memcached - memcached2 - postgres-db: - <<: *postgres-base - image: postgres:${POSTGRES_VERSION}-alpine + postgresql-db: + <<: *postgresql-base + image: postgres:${POSTGRESQL_VERSION}-alpine - postgres-gis: + postgresql-gis: <<: *base environment: - DATABASE_ENGINE=django.contrib.gis.db.backends.postgis - - DATABASE_HOST=postgres-gis-db - - WAIT_FOR=postgres-gis-db:5432 + - DATABASE_HOST=postgresql-gis-db + - WAIT_FOR=postgresql-gis-db:5432 depends_on: - - postgres-gis-db - postgres-gis-db: - <<: *postgres-base - image: postgis/postgis:${POSTGRES_VERSION}-${POSTGIS_VERSION} + - postgresql-gis-db + postgresql-gis-db: + <<: *postgresql-base + image: postgis/postgis:${POSTGRESQL_VERSION}-${POSTGIS_VERSION} mysql: <<: *base From deeeaf6a0f4158b1e3151d612f2797d276df7fad Mon Sep 17 00:00:00 2001 From: Nick Pope Date: Thu, 5 Dec 2024 23:42:46 +0000 Subject: [PATCH 12/48] Grouped and reordered services in compose file. Grouped services and commands by type and ordered alphabetically within each section. --- compose.yml | 178 +++++++++++++++++++++++++++++----------------------- 1 file changed, 99 insertions(+), 79 deletions(-) diff --git a/compose.yml b/compose.yml index e0bb328..a1f68ae 100644 --- a/compose.yml +++ b/compose.yml @@ -23,20 +23,86 @@ x-postgresql-base: &postgresql-base - /var/lib/postgresql services: - sqlite: + + # Services: Databases + + mariadb-db: + image: mariadb:${MARIADB_VERSION} + tmpfs: + - /var/lib/mysql + environment: + - MYSQL_USER=django + - MYSQL_PASSWORD=django + - MYSQL_DATABASE=django + - MYSQL_ALLOW_EMPTY_PASSWORD=1 + + mysql-db: &mysql_base + image: mysql:${MYSQL_VERSION} + entrypoint: ['/entrypoint.sh', '--default-authentication-plugin=mysql_native_password'] + tmpfs: + - /var/lib/mysql + environment: + - MYSQL_USER=django + - MYSQL_PASSWORD=django + - MYSQL_DATABASE=django + - MYSQL_ALLOW_EMPTY_PASSWORD=1 + + mysql-gis-db: + <<: *mysql_base + volumes: + - /var/lib/mysql + + postgresql-db: + <<: *postgresql-base + image: postgres:${POSTGRESQL_VERSION}-alpine + + postgresql-gis-db: + <<: *postgresql-base + image: postgis/postgis:${POSTGRESQL_VERSION}-${POSTGIS_VERSION}-alpine + + # Services: Caches + + memcached: &memcached + image: memcached:alpine + + memcached2: + <<: *memcached + + # Services: Selenium + + chrome-browser: + image: selenium/standalone-chrome + volumes: + - /dev/shm:/dev/shm + + firefox-browser: + image: selenium/standalone-firefox + volumes: + - /dev/shm:/dev/shm + + # Commands: Tests + + mariadb: <<: *base environment: - - DATABASE_ENGINE=django.db.backends.sqlite3 - entrypoint: python runtests.py + - DATABASE_ENGINE=django.db.backends.mysql + - DATABASE_HOST=mariadb-db + - WAIT_FOR=mariadb-db:3306 depends_on: + - mariadb-db - memcached - memcached2 - sqlite-gis: + mysql: <<: *base - entrypoint: python runtests.py environment: - - DATABASE_ENGINE=django.contrib.gis.db.backends.spatialite + - DATABASE_ENGINE=django.db.backends.mysql + - DATABASE_HOST=mysql-db + - WAIT_FOR=mysql-db:3306 + depends_on: + - mysql-db + - memcached + - memcached2 postgresql: <<: *base @@ -48,42 +114,17 @@ services: - postgresql-db - memcached - memcached2 - postgresql-db: - <<: *postgresql-base - image: postgres:${POSTGRESQL_VERSION}-alpine - postgresql-gis: - <<: *base - environment: - - DATABASE_ENGINE=django.contrib.gis.db.backends.postgis - - DATABASE_HOST=postgresql-gis-db - - WAIT_FOR=postgresql-gis-db:5432 - depends_on: - - postgresql-gis-db - postgresql-gis-db: - <<: *postgresql-base - image: postgis/postgis:${POSTGRESQL_VERSION}-${POSTGIS_VERSION} - - mysql: + sqlite: <<: *base environment: - - DATABASE_ENGINE=django.db.backends.mysql - - DATABASE_HOST=mysql-db - - WAIT_FOR=mysql-db:3306 + - DATABASE_ENGINE=django.db.backends.sqlite3 + entrypoint: python runtests.py depends_on: - - mysql-db - memcached - memcached2 - mysql-db: &mysql_base - image: mysql:${MYSQL_VERSION} - entrypoint: ['/entrypoint.sh', '--default-authentication-plugin=mysql_native_password'] - tmpfs: - - /var/lib/mysql - environment: - - MYSQL_USER=django - - MYSQL_PASSWORD=django - - MYSQL_DATABASE=django - - MYSQL_ALLOW_EMPTY_PASSWORD=1 + + # Commands: Tests: GIS mysql-gis: <<: *base @@ -93,48 +134,24 @@ services: - WAIT_FOR=mysql-gis-db:3306 depends_on: - mysql-gis-db - mysql-gis-db: - <<: *mysql_base - volumes: - - /var/lib/mysql - mariadb: + postgresql-gis: <<: *base environment: - - DATABASE_ENGINE=django.db.backends.mysql - - DATABASE_HOST=mariadb-db - - WAIT_FOR=mariadb-db:3306 + - DATABASE_ENGINE=django.contrib.gis.db.backends.postgis + - DATABASE_HOST=postgresql-gis-db + - WAIT_FOR=postgresql-gis-db:5432 depends_on: - - mariadb-db - - memcached - - memcached2 - mariadb-db: - image: mariadb:${MARIADB_VERSION} - tmpfs: - - /var/lib/mysql - environment: - - MYSQL_USER=django - - MYSQL_PASSWORD=django - - MYSQL_DATABASE=django - - MYSQL_ALLOW_EMPTY_PASSWORD=1 - - memcached: &memcached - image: memcached:alpine - - memcached2: - <<: *memcached + - postgresql-gis-db - docs: + sqlite-gis: <<: *base - entrypoint: ["make", "spelling"] - working_dir: /tests/django/docs - user: root + entrypoint: python runtests.py + environment: + - DATABASE_ENGINE=django.contrib.gis.db.backends.spatialite - flake8: - <<: *base - entrypoint: flake8 + # Commands: Tests: Selenium - # Browser tests chrome: <<: *base command: --selenium=chrome --selenium-hub=http://chrome-browser:4444/wd/hub @@ -143,12 +160,7 @@ services: - WAIT_FOR=chrome-browser:4444 depends_on: - chrome-browser - chrome-browser: - image: selenium/standalone-chrome - volumes: - - /dev/shm:/dev/shm - # Firefox tests fail on main firefox: <<: *base command: --selenium=firefox --selenium-hub=http://firefox-browser:4444/wd/hub @@ -157,7 +169,15 @@ services: - WAIT_FOR=firefox-browser:4444 depends_on: - firefox-browser - firefox-browser: - image: selenium/standalone-firefox - volumes: - - /dev/shm:/dev/shm + + # Commands: Other + + docs: + <<: *base + entrypoint: ["make", "spelling"] + working_dir: /tests/django/docs + user: root + + flake8: + <<: *base + entrypoint: flake8 From fc84c10d70f230b49ed6f2b8059450e7d471e0db Mon Sep 17 00:00:00 2001 From: Nick Pope Date: Thu, 5 Dec 2024 23:56:33 +0000 Subject: [PATCH 13/48] Removed obsolete entrypoint for MySQL. The MySQL native password authentication plugin is obsolete. --- compose.yml | 1 - 1 file changed, 1 deletion(-) diff --git a/compose.yml b/compose.yml index a1f68ae..6879af4 100644 --- a/compose.yml +++ b/compose.yml @@ -38,7 +38,6 @@ services: mysql-db: &mysql_base image: mysql:${MYSQL_VERSION} - entrypoint: ['/entrypoint.sh', '--default-authentication-plugin=mysql_native_password'] tmpfs: - /var/lib/mysql environment: From d97c54b3ef8229f41e33601cac82753c1d747d26 Mon Sep 17 00:00:00 2001 From: Nick Pope Date: Thu, 5 Dec 2024 23:55:38 +0000 Subject: [PATCH 14/48] Extracted common compose configuration for services. This allow us to apply configuration consistently and reduce duplication. --- compose.yml | 55 +++++++++++++++++++++++++++-------------------------- 1 file changed, 28 insertions(+), 27 deletions(-) diff --git a/compose.yml b/compose.yml index 6879af4..232cb3f 100644 --- a/compose.yml +++ b/compose.yml @@ -12,44 +12,45 @@ x-base: &base - ./settings.py:/tests/django/tests/settings.py entrypoint: bash -c "wait-for-it.sh -s -t 20 $${WAIT_FOR} -- python runtests.py $$@" -- +x-mysql-base: &mysql-base + environment: + - MYSQL_ALLOW_EMPTY_PASSWORD=1 + - MYSQL_DATABASE=django + - MYSQL_PASSWORD=django + - MYSQL_USER=django + tmpfs: + - /var/lib/mysql + x-postgresql-base: &postgresql-base environment: - - POSTGRES_USER=django - - POSTGRES_PASSWORD=django - POSTGRES_DB=django - # Set a few options to speed up postgresql tests. - command: -c fsync=off -c synchronous_commit=off -c full_page_writes=off + - POSTGRES_PASSWORD=django + - POSTGRES_USER=django tmpfs: - /var/lib/postgresql + command: >- + -c fsync=off + -c full_page_writes=off + -c synchronous_commit=off + +x-memcached: &memcached-base + image: memcached:alpine services: # Services: Databases mariadb-db: + <<: *mysql-base image: mariadb:${MARIADB_VERSION} - tmpfs: - - /var/lib/mysql - environment: - - MYSQL_USER=django - - MYSQL_PASSWORD=django - - MYSQL_DATABASE=django - - MYSQL_ALLOW_EMPTY_PASSWORD=1 - - mysql-db: &mysql_base - image: mysql:${MYSQL_VERSION} - tmpfs: - - /var/lib/mysql - environment: - - MYSQL_USER=django - - MYSQL_PASSWORD=django - - MYSQL_DATABASE=django - - MYSQL_ALLOW_EMPTY_PASSWORD=1 + + mysql-db: + <<: *mysql-base + image: &mysql-image mysql:${MYSQL_VERSION} mysql-gis-db: - <<: *mysql_base - volumes: - - /var/lib/mysql + <<: *mysql-base + image: *mysql-image postgresql-db: <<: *postgresql-base @@ -61,11 +62,11 @@ services: # Services: Caches - memcached: &memcached - image: memcached:alpine + memcached: + <<: *memcached-base memcached2: - <<: *memcached + <<: *memcached-base # Services: Selenium From 66f9500618e400cc8902988739710db9cfd70dff Mon Sep 17 00:00:00 2001 From: Nick Pope Date: Fri, 6 Dec 2024 00:01:06 +0000 Subject: [PATCH 15/48] Enforced single instance for backend services. And also ensure that they restart unless explicitly stopped. --- compose.yml | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/compose.yml b/compose.yml index 232cb3f..9a5b75a 100644 --- a/compose.yml +++ b/compose.yml @@ -20,6 +20,9 @@ x-mysql-base: &mysql-base - MYSQL_USER=django tmpfs: - /var/lib/mysql + deploy: + mode: global + restart: unless-stopped x-postgresql-base: &postgresql-base environment: @@ -28,6 +31,9 @@ x-postgresql-base: &postgresql-base - POSTGRES_USER=django tmpfs: - /var/lib/postgresql + deploy: + mode: global + restart: unless-stopped command: >- -c fsync=off -c full_page_writes=off @@ -35,6 +41,9 @@ x-postgresql-base: &postgresql-base x-memcached: &memcached-base image: memcached:alpine + deploy: + mode: global + restart: unless-stopped services: @@ -72,11 +81,17 @@ services: chrome-browser: image: selenium/standalone-chrome + deploy: + mode: global + restart: unless-stopped volumes: - /dev/shm:/dev/shm firefox-browser: image: selenium/standalone-firefox + deploy: + mode: global + restart: unless-stopped volumes: - /dev/shm:/dev/shm From 3cc6c645f2aa8343f0952a79e95147c78e3b286d Mon Sep 17 00:00:00 2001 From: Nick Pope Date: Fri, 6 Dec 2024 08:33:22 +0000 Subject: [PATCH 16/48] Replaced `wait-for-it.sh` script with compose healthchecks. This integrates better with compose commands with feedback on service health. Note that this doesn't yet introduce healthchecks for selenium nodes as we'll be switching over to Selenium Grid in a future commit which works slightly differently and the current setup doesn't seem to work correctly. --- Containerfile | 4 --- compose.yml | 83 +++++++++++++++++++++++++++++++++------------------ 2 files changed, 54 insertions(+), 33 deletions(-) diff --git a/Containerfile b/Containerfile index d5ed98a..993ba78 100644 --- a/Containerfile +++ b/Containerfile @@ -11,16 +11,12 @@ RUN apt-get update \ unzip libaio1 \ libenchant-2-2 \ gettext \ - wget \ git \ pkg-config \ && apt-get clean RUN groupadd -r test && useradd --no-log-init -r -g test test -RUN wget -q https://raw.githubusercontent.com/vishnubob/wait-for-it/master/wait-for-it.sh -O /bin/wait-for-it.sh \ - && chmod a+x /bin/wait-for-it.sh - ENV PIP_NO_CACHE_DIR=off ENV PYTHONDONTWRITEBYTECODE=1 RUN pip install --upgrade pip diff --git a/compose.yml b/compose.yml index 9a5b75a..f3de49d 100644 --- a/compose.yml +++ b/compose.yml @@ -10,7 +10,7 @@ x-base: &base volumes: - ${DJANGO_PATH}:/tests/django/ - ./settings.py:/tests/django/tests/settings.py - entrypoint: bash -c "wait-for-it.sh -s -t 20 $${WAIT_FOR} -- python runtests.py $$@" -- + entrypoint: python runtests.py x-mysql-base: &mysql-base environment: @@ -23,6 +23,12 @@ x-mysql-base: &mysql-base deploy: mode: global restart: unless-stopped + healthcheck: &mysql-base-healthcheck + interval: 5s + timeout: 5s + retries: 3 + start_period: 10s + start_interval: 1s x-postgresql-base: &postgresql-base environment: @@ -34,6 +40,13 @@ x-postgresql-base: &postgresql-base deploy: mode: global restart: unless-stopped + healthcheck: + test: pg_isready --quiet --username=django + interval: 5s + timeout: 5s + retries: 3 + start_period: 10s + start_interval: 1s command: >- -c fsync=off -c full_page_writes=off @@ -44,6 +57,19 @@ x-memcached: &memcached-base deploy: mode: global restart: unless-stopped + healthcheck: + test: echo stats | nc 127.0.0.1 11211 + interval: 5s + timeout: 5s + retries: 3 + start_period: 10s + start_interval: 1s + +x-cache-depends: &depends-on-caches + memcached: + condition: service_healthy + memcached2: + condition: service_healthy services: @@ -52,14 +78,21 @@ services: mariadb-db: <<: *mysql-base image: mariadb:${MARIADB_VERSION} + healthcheck: + <<: *mysql-base-healthcheck + test: healthcheck.sh --connect mysql-db: <<: *mysql-base image: &mysql-image mysql:${MYSQL_VERSION} + healthcheck: &mysql-healthcheck + <<: *mysql-base-healthcheck + test: mysqladmin ping --silent mysql-gis-db: <<: *mysql-base image: *mysql-image + healthcheck: *mysql-healthcheck postgresql-db: <<: *postgresql-base @@ -99,69 +132,63 @@ services: mariadb: <<: *base + depends_on: + <<: *depends-on-caches + mariadb-db: + condition: service_healthy environment: - DATABASE_ENGINE=django.db.backends.mysql - DATABASE_HOST=mariadb-db - - WAIT_FOR=mariadb-db:3306 - depends_on: - - mariadb-db - - memcached - - memcached2 mysql: <<: *base + depends_on: + <<: *depends-on-caches + mysql-db: + condition: service_healthy environment: - DATABASE_ENGINE=django.db.backends.mysql - DATABASE_HOST=mysql-db - - WAIT_FOR=mysql-db:3306 - depends_on: - - mysql-db - - memcached - - memcached2 postgresql: <<: *base + depends_on: + <<: *depends-on-caches + postgresql-db: + condition: service_healthy environment: - DATABASE_ENGINE=django.db.backends.postgresql - DATABASE_HOST=postgresql-db - - WAIT_FOR=postgresql-db:5432 - depends_on: - - postgresql-db - - memcached - - memcached2 sqlite: <<: *base + depends_on: + <<: *depends-on-caches environment: - DATABASE_ENGINE=django.db.backends.sqlite3 - entrypoint: python runtests.py - depends_on: - - memcached - - memcached2 # Commands: Tests: GIS mysql-gis: <<: *base + depends_on: + mysql-gis-db: + condition: service_healthy environment: - DATABASE_ENGINE=django.contrib.gis.db.backends.mysql - DATABASE_HOST=mysql-gis-db - - WAIT_FOR=mysql-gis-db:3306 - depends_on: - - mysql-gis-db postgresql-gis: <<: *base + depends_on: + postgresql-gis-db: + condition: service_healthy environment: - DATABASE_ENGINE=django.contrib.gis.db.backends.postgis - DATABASE_HOST=postgresql-gis-db - - WAIT_FOR=postgresql-gis-db:5432 - depends_on: - - postgresql-gis-db sqlite-gis: <<: *base - entrypoint: python runtests.py environment: - DATABASE_ENGINE=django.contrib.gis.db.backends.spatialite @@ -172,7 +199,6 @@ services: command: --selenium=chrome --selenium-hub=http://chrome-browser:4444/wd/hub environment: - DATABASE_ENGINE=django.db.backends.sqlite3 - - WAIT_FOR=chrome-browser:4444 depends_on: - chrome-browser @@ -181,7 +207,6 @@ services: command: --selenium=firefox --selenium-hub=http://firefox-browser:4444/wd/hub environment: - DATABASE_ENGINE=django.db.backends.sqlite3 - - WAIT_FOR=firefox-browser:4444 depends_on: - firefox-browser From 45e57324697a50a43d6787ed6034b6ba2698141d Mon Sep 17 00:00:00 2001 From: Nick Pope Date: Fri, 6 Dec 2024 08:39:28 +0000 Subject: [PATCH 17/48] Fixed indentation of compose file. --- compose.yml | 62 +++++++++++++++++++++++++++-------------------------- 1 file changed, 32 insertions(+), 30 deletions(-) diff --git a/compose.yml b/compose.yml index f3de49d..59c3c6c 100644 --- a/compose.yml +++ b/compose.yml @@ -1,25 +1,27 @@ +--- + x-base: &base image: djangobox/django-docker-box:${PYTHON_VERSION} build: context: ${DJANGO_PATH} dockerfile: ${PWD}/Containerfile args: - - PYTHON_VERSION=${PYTHON_VERSION} + - PYTHON_VERSION=${PYTHON_VERSION} cache_from: - - djangobox/django-docker-box:${PYTHON_VERSION} + - djangobox/django-docker-box:${PYTHON_VERSION} volumes: - - ${DJANGO_PATH}:/tests/django/ - - ./settings.py:/tests/django/tests/settings.py + - ${DJANGO_PATH}:/tests/django/ + - ./settings.py:/tests/django/tests/settings.py entrypoint: python runtests.py x-mysql-base: &mysql-base environment: - - MYSQL_ALLOW_EMPTY_PASSWORD=1 - - MYSQL_DATABASE=django - - MYSQL_PASSWORD=django - - MYSQL_USER=django + - MYSQL_ALLOW_EMPTY_PASSWORD=1 + - MYSQL_DATABASE=django + - MYSQL_PASSWORD=django + - MYSQL_USER=django tmpfs: - - /var/lib/mysql + - /var/lib/mysql deploy: mode: global restart: unless-stopped @@ -32,11 +34,11 @@ x-mysql-base: &mysql-base x-postgresql-base: &postgresql-base environment: - - POSTGRES_DB=django - - POSTGRES_PASSWORD=django - - POSTGRES_USER=django + - POSTGRES_DB=django + - POSTGRES_PASSWORD=django + - POSTGRES_USER=django tmpfs: - - /var/lib/postgresql + - /var/lib/postgresql deploy: mode: global restart: unless-stopped @@ -118,7 +120,7 @@ services: mode: global restart: unless-stopped volumes: - - /dev/shm:/dev/shm + - /dev/shm:/dev/shm firefox-browser: image: selenium/standalone-firefox @@ -126,7 +128,7 @@ services: mode: global restart: unless-stopped volumes: - - /dev/shm:/dev/shm + - /dev/shm:/dev/shm # Commands: Tests @@ -137,8 +139,8 @@ services: mariadb-db: condition: service_healthy environment: - - DATABASE_ENGINE=django.db.backends.mysql - - DATABASE_HOST=mariadb-db + - DATABASE_ENGINE=django.db.backends.mysql + - DATABASE_HOST=mariadb-db mysql: <<: *base @@ -147,8 +149,8 @@ services: mysql-db: condition: service_healthy environment: - - DATABASE_ENGINE=django.db.backends.mysql - - DATABASE_HOST=mysql-db + - DATABASE_ENGINE=django.db.backends.mysql + - DATABASE_HOST=mysql-db postgresql: <<: *base @@ -157,15 +159,15 @@ services: postgresql-db: condition: service_healthy environment: - - DATABASE_ENGINE=django.db.backends.postgresql - - DATABASE_HOST=postgresql-db + - DATABASE_ENGINE=django.db.backends.postgresql + - DATABASE_HOST=postgresql-db sqlite: <<: *base depends_on: <<: *depends-on-caches environment: - - DATABASE_ENGINE=django.db.backends.sqlite3 + - DATABASE_ENGINE=django.db.backends.sqlite3 # Commands: Tests: GIS @@ -175,8 +177,8 @@ services: mysql-gis-db: condition: service_healthy environment: - - DATABASE_ENGINE=django.contrib.gis.db.backends.mysql - - DATABASE_HOST=mysql-gis-db + - DATABASE_ENGINE=django.contrib.gis.db.backends.mysql + - DATABASE_HOST=mysql-gis-db postgresql-gis: <<: *base @@ -184,13 +186,13 @@ services: postgresql-gis-db: condition: service_healthy environment: - - DATABASE_ENGINE=django.contrib.gis.db.backends.postgis - - DATABASE_HOST=postgresql-gis-db + - DATABASE_ENGINE=django.contrib.gis.db.backends.postgis + - DATABASE_HOST=postgresql-gis-db sqlite-gis: <<: *base environment: - - DATABASE_ENGINE=django.contrib.gis.db.backends.spatialite + - DATABASE_ENGINE=django.contrib.gis.db.backends.spatialite # Commands: Tests: Selenium @@ -198,7 +200,7 @@ services: <<: *base command: --selenium=chrome --selenium-hub=http://chrome-browser:4444/wd/hub environment: - - DATABASE_ENGINE=django.db.backends.sqlite3 + - DATABASE_ENGINE=django.db.backends.sqlite3 depends_on: - chrome-browser @@ -206,9 +208,9 @@ services: <<: *base command: --selenium=firefox --selenium-hub=http://firefox-browser:4444/wd/hub environment: - - DATABASE_ENGINE=django.db.backends.sqlite3 + - DATABASE_ENGINE=django.db.backends.sqlite3 depends_on: - - firefox-browser + - firefox-browser # Commands: Other From c59c78f876bb1d64d1cff91064acab2664923257 Mon Sep 17 00:00:00 2001 From: Nick Pope Date: Fri, 6 Dec 2024 08:42:03 +0000 Subject: [PATCH 18/48] Improved name of memcached services. --- compose.yml | 8 ++++---- settings.py | 4 ++-- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/compose.yml b/compose.yml index 59c3c6c..744b172 100644 --- a/compose.yml +++ b/compose.yml @@ -68,9 +68,9 @@ x-memcached: &memcached-base start_interval: 1s x-cache-depends: &depends-on-caches - memcached: + memcached-1: condition: service_healthy - memcached2: + memcached-2: condition: service_healthy services: @@ -106,10 +106,10 @@ services: # Services: Caches - memcached: + memcached-1: <<: *memcached-base - memcached2: + memcached-2: <<: *memcached-base # Services: Selenium diff --git a/settings.py b/settings.py index 1336ac7..b949233 100644 --- a/settings.py +++ b/settings.py @@ -33,12 +33,12 @@ def _build_databases_setting(): }, "pymemcache": { "BACKEND": "django.core.cache.backends.memcached.PyMemcacheCache", - "LOCATION": "memcached:11211", + "LOCATION": "memcached-1:11211", "KEY_PREFIX": "pymemcache:", }, "pylibmc": { "BACKEND": "django.core.cache.backends.memcached.PyLibMCCache", - "LOCATION": "memcached2:11211", + "LOCATION": "memcached-2:11211", "KEY_PREFIX": "pylibmc:", }, } From 756ca67e5af8775a633b083ab82d87489869ed0e Mon Sep 17 00:00:00 2001 From: Nick Pope Date: Fri, 6 Dec 2024 08:48:29 +0000 Subject: [PATCH 19/48] Fixed MySQL to use the same credentials as other databases. Prior to this MySQL and MariaDB were creating a "django" user, but were using "root" because the user didn't have the appropriate permissions. Fixing that brings the credentials used in line with PostgreSQL. --- compose.yml | 2 ++ settings.py | 6 +----- startup-mysql.sql | 1 + 3 files changed, 4 insertions(+), 5 deletions(-) create mode 100644 startup-mysql.sql diff --git a/compose.yml b/compose.yml index 744b172..2b19a21 100644 --- a/compose.yml +++ b/compose.yml @@ -31,6 +31,8 @@ x-mysql-base: &mysql-base retries: 3 start_period: 10s start_interval: 1s + volumes: + - ./startup-mysql.sql:/docker-entrypoint-initdb.d/django.sql:ro x-postgresql-base: &postgresql-base environment: diff --git a/settings.py b/settings.py index b949233..5ad2821 100644 --- a/settings.py +++ b/settings.py @@ -18,11 +18,7 @@ def _build_databases_setting(): } if engine.endswith(".mysql"): - entry |= { - "USER": "root", - "PASSWORD": "", - "TEST": {"CHARSET": "utf8"}, - } + entry["TEST"] = {"CHARSET": "utf8"} return settings diff --git a/startup-mysql.sql b/startup-mysql.sql new file mode 100644 index 0000000..4f947cf --- /dev/null +++ b/startup-mysql.sql @@ -0,0 +1 @@ +GRANT ALL PRIVILEGES ON *.* TO 'django'@'%' WITH GRANT OPTION; From a3900b24c8de873fc50c9f697771cd3864381e3d Mon Sep 17 00:00:00 2001 From: Nick Pope Date: Fri, 6 Dec 2024 08:57:12 +0000 Subject: [PATCH 20/48] Fixed support for Selenium using Selenium Grid. Also reintroduces healthchecks. --- compose.yml | 65 +++++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 48 insertions(+), 17 deletions(-) diff --git a/compose.yml b/compose.yml index 2b19a21..55ccfec 100644 --- a/compose.yml +++ b/compose.yml @@ -75,6 +75,20 @@ x-cache-depends: &depends-on-caches memcached-2: condition: service_healthy +x-selenium-base: &selenium-base + shm_size: 2gb + environment: + - SE_EVENT_BUS_HOST=selenium-hub + - SE_EVENT_BUS_PUBLISH_PORT=4442 + - SE_EVENT_BUS_SUBSCRIBE_PORT=4443 + - SE_NODE_GRID_URL=http://selenium-hub:4444 + depends_on: + selenium-hub: + condition: service_started + deploy: + mode: global + restart: unless-stopped + services: # Services: Databases @@ -116,21 +130,26 @@ services: # Services: Selenium - chrome-browser: - image: selenium/standalone-chrome - deploy: - mode: global - restart: unless-stopped - volumes: - - /dev/shm:/dev/shm + selenium-chrome: + <<: *selenium-base + image: selenium/node-chrome - firefox-browser: - image: selenium/standalone-firefox + selenium-firefox: + <<: *selenium-base + image: selenium/node-firefox + + selenium-hub: + image: selenium/hub + ports: + - "4442-4444:4442-4444" deploy: mode: global restart: unless-stopped - volumes: - - /dev/shm:/dev/shm + healthcheck: + test: /opt/bin/check-grid.sh + interval: 15s + timeout: 30s + retries: 5 # Commands: Tests @@ -200,19 +219,31 @@ services: chrome: <<: *base - command: --selenium=chrome --selenium-hub=http://chrome-browser:4444/wd/hub + entrypoint: >- + /django/entrypoint.bash + --selenium=chrome + --selenium-hub=http://selenium-hub:4444/wd/hub + depends_on: + selenium-hub: + condition: service_healthy + selenium-chrome: + condition: service_started environment: - DATABASE_ENGINE=django.db.backends.sqlite3 - depends_on: - - chrome-browser firefox: <<: *base - command: --selenium=firefox --selenium-hub=http://firefox-browser:4444/wd/hub + entrypoint: >- + /django/entrypoint.bash + --selenium=firefox + --selenium-hub=http://selenium-hub:4444/wd/hub + depends_on: + selenium-hub: + condition: service_healthy + selenium-firefox: + condition: service_started environment: - DATABASE_ENGINE=django.db.backends.sqlite3 - depends_on: - - firefox-browser # Commands: Other From a4111ef48634fd5bdcf306551d8ef527aa99beb0 Mon Sep 17 00:00:00 2001 From: Nick Pope Date: Fri, 6 Dec 2024 09:44:48 +0000 Subject: [PATCH 21/48] Optimized services for testing. This disables various features in databases and attempts to reduce disk writes such that testing is as fast as possible. --- compose.yml | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/compose.yml b/compose.yml index 55ccfec..7b240e3 100644 --- a/compose.yml +++ b/compose.yml @@ -52,9 +52,17 @@ x-postgresql-base: &postgresql-base start_period: 10s start_interval: 1s command: >- + -c archive_mode=off + -c checkpoint_completion_target=0.9 + -c checkpoint_timeout=900 -c fsync=off -c full_page_writes=off + -c max_replication_slots=0 + -c max_wal_senders=0 + -c max_wal_size=4096 -c synchronous_commit=off + -c wal_level=minimal + # 13+: -c wal_keep_size=0 x-memcached: &memcached-base image: memcached:alpine @@ -68,6 +76,10 @@ x-memcached: &memcached-base retries: 3 start_period: 10s start_interval: 1s + command: >- + --conn-limit=1024 + --memory-limit=64 + --threads=4 x-cache-depends: &depends-on-caches memcached-1: @@ -96,6 +108,16 @@ services: mariadb-db: <<: *mysql-base image: mariadb:${MARIADB_VERSION} + command: >- + --innodb-fast-shutdown=3 + --innodb-flush-log-at-trx-commit=0 + --innodb-flush-method=nosync + --innodb-random-read-ahead + --skip-innodb-doublewrite + --skip-innodb-file-per-table + --skip-innodb-flush-sync + --skip-innodb-use-atomic-writes + --skip-name-resolve healthcheck: <<: *mysql-base-healthcheck test: healthcheck.sh --connect @@ -103,6 +125,15 @@ services: mysql-db: <<: *mysql-base image: &mysql-image mysql:${MYSQL_VERSION} + command: &mysql-command >- + --innodb-flush-log-at-trx-commit=0 + --innodb-flush-method=nosync + --innodb-random-read-ahead + --skip-innodb-doublewrite + --skip-innodb-extend-and-initialize + --skip-innodb-file-per-table + --skip-innodb-flush-sync + --skip-name-resolve healthcheck: &mysql-healthcheck <<: *mysql-base-healthcheck test: mysqladmin ping --silent @@ -110,6 +141,7 @@ services: mysql-gis-db: <<: *mysql-base image: *mysql-image + command: *mysql-command healthcheck: *mysql-healthcheck postgresql-db: From 1c64296707317da3d6be4074184bf09b6faf0b1e Mon Sep 17 00:00:00 2001 From: Nick Pope Date: Fri, 6 Dec 2024 13:00:19 +0000 Subject: [PATCH 22/48] Reinstated support for Oracle database. This is now using the images provided by Oracle in their own container registry. --- .env | 1 + README.md | 1 + compose.yml | 44 ++++++++++++++++++++++++++++++++++++++++++++ settings.py | 11 +++++++++++ startup-oracle.sql | 3 +++ 5 files changed, 60 insertions(+) create mode 100644 startup-oracle.sql diff --git a/.env b/.env index 62d48a4..2386a15 100644 --- a/.env +++ b/.env @@ -3,3 +3,4 @@ POSTGRESQL_VERSION=13 POSTGIS_VERSION=3.0 MYSQL_VERSION=8 MARIADB_VERSION=10.5 +ORACLE_VERSION=23.5.0.0 diff --git a/README.md b/README.md index 5244747..6bab46d 100644 --- a/README.md +++ b/README.md @@ -78,3 +78,4 @@ To enter a bash shell within the container, run: | `POSTGRESQL_VERSION` | `13` | The version of Postgres to use | | `MYSQL_VERSION` | `8` | The mysql version to use | | `MARIADB_VERSION` | `10.5` | The mariadb version to use | +| `ORACLE_VERSION` | `23.5.0.0` | The Oracle version to use | diff --git a/compose.yml b/compose.yml index 7b240e3..4bf0226 100644 --- a/compose.yml +++ b/compose.yml @@ -34,6 +34,22 @@ x-mysql-base: &mysql-base volumes: - ./startup-mysql.sql:/docker-entrypoint-initdb.d/django.sql:ro +x-oracle-base: &oracle-base + environment: + - ORACLE_PWD=oracle + deploy: + mode: global + restart: unless-stopped + healthcheck: + test: /opt/oracle/checkDBStatus.sh + interval: 5s + timeout: 5s + retries: 3 + start_period: 10s + start_interval: 1s + volumes: + - ./startup-oracle.sql:/opt/oracle/scripts/startup/django.sql:ro + x-postgresql-base: &postgresql-base environment: - POSTGRES_DB=django @@ -144,6 +160,15 @@ services: command: *mysql-command healthcheck: *mysql-healthcheck + oracle-db: + <<: *oracle-base + # yamllint disable-line rule:line-length + image: &oracle-image container-registry.oracle.com/database/free:${ORACLE_VERSION}-lite + + oracle-gis-db: + <<: *oracle-base + image: *oracle-image + postgresql-db: <<: *postgresql-base image: postgres:${POSTGRESQL_VERSION}-alpine @@ -205,6 +230,16 @@ services: - DATABASE_ENGINE=django.db.backends.mysql - DATABASE_HOST=mysql-db + oracle: + <<: *base + depends_on: + <<: *depends-on-caches + oracle-db: + condition: service_healthy + environment: + - DATABASE_ENGINE=django.db.backends.oracle + - DATABASE_NAME=oracle-db:1521/freepdb1 + postgresql: <<: *base depends_on: @@ -233,6 +268,15 @@ services: - DATABASE_ENGINE=django.contrib.gis.db.backends.mysql - DATABASE_HOST=mysql-gis-db + oracle-gis: + <<: *base + depends_on: + oracle-gis-db: + condition: service_healthy + environment: + - DATABASE_ENGINE=django.contrib.gis.db.backends.oracle + - DATABASE_NAME=oracle-gis-db:1521/freepdb1 + postgresql-gis: <<: *base depends_on: diff --git a/settings.py b/settings.py index 5ad2821..8ed2188 100644 --- a/settings.py +++ b/settings.py @@ -4,6 +4,7 @@ def _build_databases_setting(): engine = os.environ["DATABASE_ENGINE"] host = os.environ.get("DATABASE_HOST", "") + name = os.environ.get("DATABASE_NAME", "") settings = {} for n, alias in enumerate(("default", "other"), start=1): @@ -20,6 +21,16 @@ def _build_databases_setting(): if engine.endswith(".mysql"): entry["TEST"] = {"CHARSET": "utf8"} + if engine.endswith(".oracle"): + entry |= { + "NAME": name, + "TEST": { + "USER": f"{alias}_test", + "TBLSPACE": f"{alias}_test_tbls", + "TBLSPACE_TMP": f"{alias}_test_tbls_tmp", + }, + } + return settings diff --git a/startup-oracle.sql b/startup-oracle.sql new file mode 100644 index 0000000..92ae6b1 --- /dev/null +++ b/startup-oracle.sql @@ -0,0 +1,3 @@ +ALTER SESSION SET CONTAINER=FREEPDB1; +CREATE USER django IDENTIFIED BY django; +GRANT DBA TO django; From 6dc210ee600e1389ce76728828d7b9b372de84a4 Mon Sep 17 00:00:00 2001 From: Nick Pope Date: Fri, 6 Dec 2024 09:51:59 +0000 Subject: [PATCH 23/48] Added GIS support for MariaDB. --- compose.yml | 21 ++++++++++++++++++--- 1 file changed, 18 insertions(+), 3 deletions(-) diff --git a/compose.yml b/compose.yml index 4bf0226..8d3f7c6 100644 --- a/compose.yml +++ b/compose.yml @@ -123,8 +123,8 @@ services: mariadb-db: <<: *mysql-base - image: mariadb:${MARIADB_VERSION} - command: >- + image: &mariadb-image mariadb:${MARIADB_VERSION} + command: &mariadb-command >- --innodb-fast-shutdown=3 --innodb-flush-log-at-trx-commit=0 --innodb-flush-method=nosync @@ -134,10 +134,16 @@ services: --skip-innodb-flush-sync --skip-innodb-use-atomic-writes --skip-name-resolve - healthcheck: + healthcheck: &mariadb-healthcheck <<: *mysql-base-healthcheck test: healthcheck.sh --connect + mariadb-gis-db: + <<: *mysql-base + image: *mariadb-image + command: *mariadb-command + healthcheck: *mariadb-healthcheck + mysql-db: <<: *mysql-base image: &mysql-image mysql:${MYSQL_VERSION} @@ -259,6 +265,15 @@ services: # Commands: Tests: GIS + mariadb-gis: + <<: *base + depends_on: + mariadb-gis-db: + condition: service_healthy + environment: + - DATABASE_ENGINE=django.contrib.gis.db.backends.mysql + - DATABASE_HOST=mariadb-gis-db + mysql-gis: <<: *base depends_on: From 42b9383bdedf81175a07afbff28b56d5f617cec9 Mon Sep 17 00:00:00 2001 From: Nick Pope Date: Fri, 6 Dec 2024 09:55:46 +0000 Subject: [PATCH 24/48] Added support for Redis cache backend. Ensures that tests can be run for the cache backend for Redis. --- compose.yml | 18 ++++++++++++++++++ settings.py | 5 +++++ 2 files changed, 23 insertions(+) diff --git a/compose.yml b/compose.yml index 8d3f7c6..3075342 100644 --- a/compose.yml +++ b/compose.yml @@ -97,11 +97,26 @@ x-memcached: &memcached-base --memory-limit=64 --threads=4 +x-redis: &redis-base + image: redis:alpine + deploy: + mode: global + restart: unless-stopped + healthcheck: + test: redis-cli ping | grep -i pong + interval: 5s + timeout: 5s + retries: 3 + start_period: 10s + start_interval: 1s + x-cache-depends: &depends-on-caches memcached-1: condition: service_healthy memcached-2: condition: service_healthy + redis: + condition: service_healthy x-selenium-base: &selenium-base shm_size: 2gb @@ -191,6 +206,9 @@ services: memcached-2: <<: *memcached-base + redis: + <<: *redis-base + # Services: Selenium selenium-chrome: diff --git a/settings.py b/settings.py index 8ed2188..3c6d3c9 100644 --- a/settings.py +++ b/settings.py @@ -48,6 +48,11 @@ def _build_databases_setting(): "LOCATION": "memcached-2:11211", "KEY_PREFIX": "pylibmc:", }, + "redis-py": { + "BACKEND": "django.core.cache.backends.redis.RedisCache", + "LOCATION": "redis://redis:6379", + "KEY_PREFIX": "redis:", + }, } DATABASES = _build_databases_setting() From e3987c593df4f3cd011b92d526e73973dc419c8e Mon Sep 17 00:00:00 2001 From: Nick Pope Date: Fri, 6 Dec 2024 12:51:04 +0000 Subject: [PATCH 25/48] Added support for running Selenium with Edge. --- compose.yml | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/compose.yml b/compose.yml index 3075342..660796e 100644 --- a/compose.yml +++ b/compose.yml @@ -215,6 +215,10 @@ services: <<: *selenium-base image: selenium/node-chrome + selenium-edge: + <<: *selenium-base + image: selenium/node-edge + selenium-firefox: <<: *selenium-base image: selenium/node-firefox @@ -340,6 +344,20 @@ services: environment: - DATABASE_ENGINE=django.db.backends.sqlite3 + edge: + <<: *base + entrypoint: >- + /django/entrypoint.bash + --selenium=edge + --selenium-hub=http://selenium-hub:4444/wd/hub + depends_on: + selenium-hub: + condition: service_healthy + selenium-edge: + condition: service_started + environment: + - DATABASE_ENGINE=django.db.backends.sqlite3 + firefox: <<: *base entrypoint: >- From cad6848cff5aa8c97f24de0301f1c836b1d72972 Mon Sep 17 00:00:00 2001 From: Nick Pope Date: Fri, 13 Dec 2024 17:05:03 +0000 Subject: [PATCH 26/48] Enabled cache backends for GIS tests. There's not much reason why these need to be disabled. --- compose.yml | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/compose.yml b/compose.yml index 660796e..c5721d3 100644 --- a/compose.yml +++ b/compose.yml @@ -290,6 +290,7 @@ services: mariadb-gis: <<: *base depends_on: + <<: *depends-on-caches mariadb-gis-db: condition: service_healthy environment: @@ -299,6 +300,7 @@ services: mysql-gis: <<: *base depends_on: + <<: *depends-on-caches mysql-gis-db: condition: service_healthy environment: @@ -308,6 +310,7 @@ services: oracle-gis: <<: *base depends_on: + <<: *depends-on-caches oracle-gis-db: condition: service_healthy environment: @@ -317,6 +320,7 @@ services: postgresql-gis: <<: *base depends_on: + <<: *depends-on-caches postgresql-gis-db: condition: service_healthy environment: @@ -325,6 +329,8 @@ services: sqlite-gis: <<: *base + depends_on: + <<: *depends-on-caches environment: - DATABASE_ENGINE=django.contrib.gis.db.backends.spatialite From 0cc9a8f16cb40d7bd3ab8780c6d4a975e3716809 Mon Sep 17 00:00:00 2001 From: Nick Pope Date: Fri, 6 Dec 2024 13:20:36 +0000 Subject: [PATCH 27/48] Created improved folder structure under `/django`. MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Use the top-level `/django` folder as a place to copy files and mount volumes. The Django source code will now be mounted to `/django/source` and any generated output, e.g. coverage, will be output to a mounted volume at `/django/output`. It is also no longer necessary to specify the `DJANGO_PATH` environment variable if you have your Django respository checked out alongside this one, e.g. ``` ~/Sources ├── django └── django-docker-box ``` If you keep your source elsewhere, then the default of `../django` can still be overridden using `DJANGO_PATH`. Currently the Django sources are mounted read-write. This is due to various tools that are currently configured to write output into the source tree, e.g. coverage. In addition, some tests may write files to the source tree instead of a temporary location. Hopefully we can gradually address these issues and then move to mounting the source tree read-only instead. This commit also sets up the Django source tree as an additional context instead of the default context. This will allow including files from this repository during the image build but also sets up sourcing requirements files from the source tree which will be crucial to improving the container file. --- .dockerignore | 4 ++-- Containerfile | 12 ++++++------ README.md | 2 +- compose.yml | 9 ++++++--- output/.gitignore | 2 ++ settings.py | 2 +- 6 files changed, 18 insertions(+), 13 deletions(-) create mode 100644 output/.gitignore diff --git a/.dockerignore b/.dockerignore index 9464c95..adc4c50 100644 --- a/.dockerignore +++ b/.dockerignore @@ -1,3 +1,3 @@ .git/ -.dockerignore/ -django \ No newline at end of file +.dockerignore +output/ diff --git a/Containerfile b/Containerfile index 993ba78..66aa56d 100644 --- a/Containerfile +++ b/Containerfile @@ -21,14 +21,14 @@ ENV PIP_NO_CACHE_DIR=off ENV PYTHONDONTWRITEBYTECODE=1 RUN pip install --upgrade pip -COPY --chown=test:test tests/requirements/ /requirements/ +COPY --chown=test:test --from=src tests/requirements/ /requirements/ RUN for f in /requirements/*.txt; do pip install -r $f; done && \ pip install flake8 flake8-isort sphinx pyenchant sphinxcontrib-spelling selenium unittest-xml-reporting -RUN mkdir /tests && chown -R test:test /tests -RUN mkdir /tests/results && chown -R test:test /tests/results/ +RUN mkdir --parents /django/{output,source} && chown --recursive test:test /django USER test:test ENV DJANGO_SETTINGS_MODULE=settings -ENV PYTHONPATH "${PYTHONPATH}:/tests/django/" -VOLUME /tests/django -WORKDIR /tests/django/tests +ENV PYTHONPATH="${PYTHONPATH}:/django/source/" +VOLUME /django/output +VOLUME /django/source +WORKDIR /django/source/tests diff --git a/README.md b/README.md index 6bab46d..bd58b75 100644 --- a/README.md +++ b/README.md @@ -73,7 +73,7 @@ To enter a bash shell within the container, run: | Environment Variable | Default | Description | | --- | --- | --- | -| `DJANGO_PATH` | None | The path to the Django codebase on your local machine | +| `DJANGO_PATH` | `../django` | The path to the Django codebase on your local machine | | `PYTHON_VERSION` | `3.10` | The python version to run tests against | | `POSTGRESQL_VERSION` | `13` | The version of Postgres to use | | `MYSQL_VERSION` | `8` | The mysql version to use | diff --git a/compose.yml b/compose.yml index c5721d3..19574ce 100644 --- a/compose.yml +++ b/compose.yml @@ -3,15 +3,18 @@ x-base: &base image: djangobox/django-docker-box:${PYTHON_VERSION} build: - context: ${DJANGO_PATH} + context: . dockerfile: ${PWD}/Containerfile args: - PYTHON_VERSION=${PYTHON_VERSION} cache_from: - djangobox/django-docker-box:${PYTHON_VERSION} + additional_contexts: + src: ${DJANGO_PATH:-../django} volumes: - - ${DJANGO_PATH}:/tests/django/ - - ./settings.py:/tests/django/tests/settings.py + - ${DJANGO_PATH:-../django}:/django/source:rw + - ./output:/django/output:rw + - ./settings.py:/django/source/tests/settings.py:ro entrypoint: python runtests.py x-mysql-base: &mysql-base diff --git a/output/.gitignore b/output/.gitignore new file mode 100644 index 0000000..d6b7ef3 --- /dev/null +++ b/output/.gitignore @@ -0,0 +1,2 @@ +* +!.gitignore diff --git a/settings.py b/settings.py index 3c6d3c9..c5f3aad 100644 --- a/settings.py +++ b/settings.py @@ -67,4 +67,4 @@ def _build_databases_setting(): if os.environ.get("XUNIT", "0").lower() in {"1", "on", "true", "yes"}: TEST_RUNNER = "xmlrunner.extra.djangotestrunner.XMLTestRunner" - TEST_OUTPUT_DIR = "/tests/results" + TEST_OUTPUT_DIR = "/django/output/xunit" From 6466755dd2857a334590c8d418d55fb28db7f4b3 Mon Sep 17 00:00:00 2001 From: Nick Pope Date: Fri, 6 Dec 2024 14:20:58 +0000 Subject: [PATCH 28/48] Extracted packages into separate files. It's much easier to maintain a list of packages to install in a separate file. We can also get the packages required to build the documentation from the requirements file in the Django source tree. --- Containerfile | 20 ++++++-------------- packages.txt | 17 +++++++++++++++++ requirements.txt | 3 +++ 3 files changed, 26 insertions(+), 14 deletions(-) create mode 100644 packages.txt create mode 100644 requirements.txt diff --git a/Containerfile b/Containerfile index 66aa56d..315125d 100644 --- a/Containerfile +++ b/Containerfile @@ -1,18 +1,9 @@ ARG PYTHON_VERSION FROM python:${PYTHON_VERSION}-slim-bullseye +COPY --chown=test:test packages.txt /django/ RUN apt-get update \ - && apt-get install --no-install-recommends -y \ - libmemcached-dev \ - build-essential \ - libsqlite3-mod-spatialite binutils libproj-dev gdal-bin libgdal28 libgeoip1 \ - default-libmysqlclient-dev default-mysql-client \ - libpq-dev \ - unzip libaio1 \ - libenchant-2-2 \ - gettext \ - git \ - pkg-config \ + && xargs --arg-file=/django/packages.txt apt-get install --no-install-recommends -y \ && apt-get clean RUN groupadd -r test && useradd --no-log-init -r -g test test @@ -21,9 +12,10 @@ ENV PIP_NO_CACHE_DIR=off ENV PYTHONDONTWRITEBYTECODE=1 RUN pip install --upgrade pip -COPY --chown=test:test --from=src tests/requirements/ /requirements/ -RUN for f in /requirements/*.txt; do pip install -r $f; done && \ - pip install flake8 flake8-isort sphinx pyenchant sphinxcontrib-spelling selenium unittest-xml-reporting +COPY --chown=test:test requirements.txt /django/requirements/extra.txt +COPY --chown=test:test --from=src tests/requirements/ /django/requirements/ +COPY --chown=test:test --from=src docs/requirements.txt /django/requirements/docs.txt +RUN for f in /django/requirements/*.txt; do pip install -r $f; done RUN mkdir --parents /django/{output,source} && chown --recursive test:test /django USER test:test diff --git a/packages.txt b/packages.txt new file mode 100644 index 0000000..a0004a2 --- /dev/null +++ b/packages.txt @@ -0,0 +1,17 @@ +binutils +build-essential +default-libmysqlclient-dev +default-mysql-client +gdal-bin +gettext +git +libaio1 +libenchant-2-2 +libgdal28 +libgeoip1 +libmemcached-dev +libpq-dev +libproj-dev +libsqlite3-mod-spatialite +pkg-config +unzip diff --git a/requirements.txt b/requirements.txt new file mode 100644 index 0000000..8e512b1 --- /dev/null +++ b/requirements.txt @@ -0,0 +1,3 @@ +flake8 +flake8-isort +unittest-xml-reporting From d376c4f14e291acaaa20207cd645e272b7a1de38 Mon Sep 17 00:00:00 2001 From: Nick Pope Date: Fri, 6 Dec 2024 14:41:28 +0000 Subject: [PATCH 29/48] Changed user name in container from `test` to `django`. --- Containerfile | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/Containerfile b/Containerfile index 315125d..3c87b75 100644 --- a/Containerfile +++ b/Containerfile @@ -1,24 +1,24 @@ ARG PYTHON_VERSION FROM python:${PYTHON_VERSION}-slim-bullseye -COPY --chown=test:test packages.txt /django/ +COPY --chown=django:django packages.txt /django/ RUN apt-get update \ && xargs --arg-file=/django/packages.txt apt-get install --no-install-recommends -y \ && apt-get clean -RUN groupadd -r test && useradd --no-log-init -r -g test test +RUN groupadd -r django && useradd --no-log-init -r -g django django ENV PIP_NO_CACHE_DIR=off ENV PYTHONDONTWRITEBYTECODE=1 RUN pip install --upgrade pip -COPY --chown=test:test requirements.txt /django/requirements/extra.txt -COPY --chown=test:test --from=src tests/requirements/ /django/requirements/ -COPY --chown=test:test --from=src docs/requirements.txt /django/requirements/docs.txt +COPY --chown=django:django requirements.txt /django/requirements/extra.txt +COPY --chown=django:django --from=src tests/requirements/ /django/requirements/ +COPY --chown=django:django --from=src docs/requirements.txt /django/requirements/docs.txt RUN for f in /django/requirements/*.txt; do pip install -r $f; done -RUN mkdir --parents /django/{output,source} && chown --recursive test:test /django -USER test:test +RUN mkdir --parents /django/{output,source} && chown --recursive django:django /django +USER django:django ENV DJANGO_SETTINGS_MODULE=settings ENV PYTHONPATH="${PYTHONPATH}:/django/source/" VOLUME /django/output From a83b162bb800a93f279deaedfaeda1b22336149e Mon Sep 17 00:00:00 2001 From: Nick Pope Date: Fri, 6 Dec 2024 14:52:26 +0000 Subject: [PATCH 30/48] Modernized and optimized container file. Use modern syntax including heredocs, cache mounts, etc. Also merge the requirements files to reduce to a single install command. --- Containerfile | 46 ++++++++++++++++++++++++++++++++++------------ compose.yml | 1 - 2 files changed, 34 insertions(+), 13 deletions(-) diff --git a/Containerfile b/Containerfile index 3c87b75..b5c11e7 100644 --- a/Containerfile +++ b/Containerfile @@ -1,26 +1,48 @@ -ARG PYTHON_VERSION +# syntax=docker/dockerfile:1.12 + +ARG PYTHON_VERSION=3.10 FROM python:${PYTHON_VERSION}-slim-bullseye -COPY --chown=django:django packages.txt /django/ -RUN apt-get update \ - && xargs --arg-file=/django/packages.txt apt-get install --no-install-recommends -y \ - && apt-get clean +SHELL ["/bin/bash", "-o", "errexit", "-o", "nounset", "-o", "pipefail", "-o", "xtrace", "-c"] -RUN groupadd -r django && useradd --no-log-init -r -g django django +ENV DEBIAN_FRONTEND=noninteractive +ENV PYTHONUNBUFFERED=1 -ENV PIP_NO_CACHE_DIR=off -ENV PYTHONDONTWRITEBYTECODE=1 -RUN pip install --upgrade pip +# Create user and prepare directories. +RUN < /etc/apt/apt.conf.d/keep-cache + apt-get update --quiet --yes + xargs --arg-file=/django/packages.txt apt-get install --no-install-recommends --yes +EOF +# Install all Python requirements in a single command. COPY --chown=django:django requirements.txt /django/requirements/extra.txt COPY --chown=django:django --from=src tests/requirements/ /django/requirements/ COPY --chown=django:django --from=src docs/requirements.txt /django/requirements/docs.txt -RUN for f in /django/requirements/*.txt; do pip install -r $f; done +RUN --mount=type=cache,target=/root/.cache/pip < Date: Fri, 6 Dec 2024 15:00:32 +0000 Subject: [PATCH 31/48] Switched to `uv` for installing packages. --- Containerfile | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/Containerfile b/Containerfile index b5c11e7..ca037cb 100644 --- a/Containerfile +++ b/Containerfile @@ -29,12 +29,13 @@ EOF COPY --chown=django:django requirements.txt /django/requirements/extra.txt COPY --chown=django:django --from=src tests/requirements/ /django/requirements/ COPY --chown=django:django --from=src docs/requirements.txt /django/requirements/docs.txt -RUN --mount=type=cache,target=/root/.cache/pip < Date: Fri, 6 Dec 2024 15:02:56 +0000 Subject: [PATCH 32/48] Added a basic entrypoint script. This will be expanded in future for enabling coverage, etc. --- Containerfile | 4 +++- entrypoint.bash | 3 +++ 2 files changed, 6 insertions(+), 1 deletion(-) create mode 100755 entrypoint.bash diff --git a/Containerfile b/Containerfile index ca037cb..a1961f1 100644 --- a/Containerfile +++ b/Containerfile @@ -38,6 +38,8 @@ RUN --mount=type=cache,target=/root/.cache/uv < Date: Fri, 6 Dec 2024 15:37:32 +0000 Subject: [PATCH 33/48] Added a `LICENSE` file. --- LICENSE | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) create mode 100644 LICENSE diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..5f4f225 --- /dev/null +++ b/LICENSE @@ -0,0 +1,27 @@ +Copyright (c) Django Software Foundation and individual contributors. +All rights reserved. + +Redistribution and use in source and binary forms, with or without modification, +are permitted provided that the following conditions are met: + + 1. Redistributions of source code must retain the above copyright notice, + this list of conditions and the following disclaimer. + + 2. Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in the + documentation and/or other materials provided with the distribution. + + 3. Neither the name of Django nor the names of its contributors may be used + to endorse or promote products derived from this software without + specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND +ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR +ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES +(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; +LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON +ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. From 88ac278f6da1e394f242042b68f2618a9423edd6 Mon Sep 17 00:00:00 2001 From: Nick Pope Date: Fri, 6 Dec 2024 15:38:59 +0000 Subject: [PATCH 34/48] Added some labels to the container image. --- Containerfile | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/Containerfile b/Containerfile index a1961f1..2612910 100644 --- a/Containerfile +++ b/Containerfile @@ -3,6 +3,15 @@ ARG PYTHON_VERSION=3.10 FROM python:${PYTHON_VERSION}-slim-bullseye +LABEL org.opencontainers.image.authors="Django Software Foundation" +LABEL org.opencontainers.image.url="https://github.com/django/django-docker-box" +LABEL org.opencontainers.image.documentation="https://github.com/django/django-docker-box" +LABEL org.opencontainers.image.source="https://github.com/django/django-docker-box" +LABEL org.opencontainers.image.vendor="Django Software Foundation" +LABEL org.opencontainers.image.licenses="BSD-3-Clause" +LABEL org.opencontainers.image.title="Django Docker Box" +LABEL org.opencontainers.image.description="Container image for developing and testing Django." + SHELL ["/bin/bash", "-o", "errexit", "-o", "nounset", "-o", "pipefail", "-o", "xtrace", "-c"] ENV DEBIAN_FRONTEND=noninteractive From 6e377d4870f580fd2ed8bafbc5e02f56412e3261 Mon Sep 17 00:00:00 2001 From: Nick Pope Date: Fri, 6 Dec 2024 16:48:41 +0000 Subject: [PATCH 35/48] Updated to use Debian Bookworm. Also update some packages to install the dev packages. These will install the correct runtime libraries and keeps this maintainable - the version numbers in the package names can change with each distro update, so using the dev package avoids manual updates. --- Containerfile | 2 +- packages.txt | 8 ++++---- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/Containerfile b/Containerfile index 2612910..37d0f53 100644 --- a/Containerfile +++ b/Containerfile @@ -1,7 +1,7 @@ # syntax=docker/dockerfile:1.12 ARG PYTHON_VERSION=3.10 -FROM python:${PYTHON_VERSION}-slim-bullseye +FROM python:${PYTHON_VERSION}-slim-bookworm LABEL org.opencontainers.image.authors="Django Software Foundation" LABEL org.opencontainers.image.url="https://github.com/django/django-docker-box" diff --git a/packages.txt b/packages.txt index a0004a2..14a4662 100644 --- a/packages.txt +++ b/packages.txt @@ -5,10 +5,10 @@ default-mysql-client gdal-bin gettext git -libaio1 -libenchant-2-2 -libgdal28 -libgeoip1 +libaio-dev +libenchant-2-dev +libgdal-dev +libgeoip-dev libmemcached-dev libpq-dev libproj-dev From 2a36fa734a15a8dcf6fef5c2849250632f8d7aca Mon Sep 17 00:00:00 2001 From: Nick Pope Date: Fri, 6 Dec 2024 17:03:46 +0000 Subject: [PATCH 36/48] Added support for running with PyPy. --- .env | 1 + Containerfile | 3 ++- README.md | 1 + compose.yml | 5 +++-- 4 files changed, 7 insertions(+), 3 deletions(-) diff --git a/.env b/.env index 2386a15..d8445b5 100644 --- a/.env +++ b/.env @@ -1,3 +1,4 @@ +PYTHON_IMPLEMENTATION=python PYTHON_VERSION=3.10 POSTGRESQL_VERSION=13 POSTGIS_VERSION=3.0 diff --git a/Containerfile b/Containerfile index 37d0f53..8338989 100644 --- a/Containerfile +++ b/Containerfile @@ -1,7 +1,8 @@ # syntax=docker/dockerfile:1.12 +ARG PYTHON_IMPLEMENTATION=python ARG PYTHON_VERSION=3.10 -FROM python:${PYTHON_VERSION}-slim-bookworm +FROM ${PYTHON_IMPLEMENTATION}:${PYTHON_VERSION}-slim-bookworm LABEL org.opencontainers.image.authors="Django Software Foundation" LABEL org.opencontainers.image.url="https://github.com/django/django-docker-box" diff --git a/README.md b/README.md index bd58b75..dc002aa 100644 --- a/README.md +++ b/README.md @@ -74,6 +74,7 @@ To enter a bash shell within the container, run: | Environment Variable | Default | Description | | --- | --- | --- | | `DJANGO_PATH` | `../django` | The path to the Django codebase on your local machine | +| `PYTHON_IMPLEMENTATION` | `python` | One of `python` or `pypy` | | `PYTHON_VERSION` | `3.10` | The python version to run tests against | | `POSTGRESQL_VERSION` | `13` | The version of Postgres to use | | `MYSQL_VERSION` | `8` | The mysql version to use | diff --git a/compose.yml b/compose.yml index c4d24e2..454cba3 100644 --- a/compose.yml +++ b/compose.yml @@ -1,14 +1,15 @@ --- x-base: &base - image: djangobox/django-docker-box:${PYTHON_VERSION} + image: djangobox/django-docker-box:${PYTHON_IMPLEMENTATION}-${PYTHON_VERSION} build: context: . dockerfile: ${PWD}/Containerfile args: + - PYTHON_IMPLEMENTATION=${PYTHON_IMPLEMENTATION} - PYTHON_VERSION=${PYTHON_VERSION} cache_from: - - djangobox/django-docker-box:${PYTHON_VERSION} + - djangobox/django-docker-box:${PYTHON_IMPLEMENTATION}-${PYTHON_VERSION} additional_contexts: src: ${DJANGO_PATH:-../django} volumes: From 34fedb732e2427b7518d95cad8a48bac895622f1 Mon Sep 17 00:00:00 2001 From: Nick Pope Date: Fri, 6 Dec 2024 17:04:58 +0000 Subject: [PATCH 37/48] Enforced color output when running in CI. Ensure that we'll get colored output when running in GitHub Actions. --- Containerfile | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/Containerfile b/Containerfile index 8338989..29bcb11 100644 --- a/Containerfile +++ b/Containerfile @@ -18,6 +18,11 @@ SHELL ["/bin/bash", "-o", "errexit", "-o", "nounset", "-o", "pipefail", "-o", "x ENV DEBIAN_FRONTEND=noninteractive ENV PYTHONUNBUFFERED=1 +# Force colored output for various tooling in CI. +ENV COLUMNS=120 +ENV FORCE_COLOR=1 +ENV TERM="xterm-256color" + # Create user and prepare directories. RUN < Date: Fri, 6 Dec 2024 17:08:18 +0000 Subject: [PATCH 38/48] Updated `.editorconfig`. --- .editorconfig | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/.editorconfig b/.editorconfig index 37b4d92..eb54abf 100644 --- a/.editorconfig +++ b/.editorconfig @@ -1,12 +1,12 @@ root = true [*] -indent_style = space +charset = utf-8 +end_of_line = lf indent_size = 4 +indent_style = space insert_final_newline = true trim_trailing_whitespace = true -end_of_line = lf -charset = utf-8 -[*.yml] +[*.{yaml,yml}] indent_size = 2 From a7316016dc0aa02b9cad1640d2adde5fbfce6f8c Mon Sep 17 00:00:00 2001 From: Nick Pope Date: Fri, 6 Dec 2024 17:15:02 +0000 Subject: [PATCH 39/48] Replaced `flake8` with `pre-commit`. Using `pre-commit` makes it easier to run all linters with the versions pinned by Django. Also add `black` as a dependency so that migrations which are created when using these images will be formatted. --- Containerfile | 4 ++-- README.md | 4 ++-- compose.yml | 13 +++++++++++-- requirements.txt | 4 ++-- 4 files changed, 17 insertions(+), 8 deletions(-) diff --git a/Containerfile b/Containerfile index 29bcb11..c75ffeb 100644 --- a/Containerfile +++ b/Containerfile @@ -25,8 +25,8 @@ ENV TERM="xterm-256color" # Create user and prepare directories. RUN < Date: Fri, 6 Dec 2024 17:37:34 +0000 Subject: [PATCH 40/48] Added linting tools and workflow for this repository. --- .github/workflows/lint.yml | 40 ++++++++++++++++++++++++++++ .hadolint.yml | 5 ++++ .pre-commit-config.yaml | 53 ++++++++++++++++++++++++++++++++++++++ 3 files changed, 98 insertions(+) create mode 100644 .github/workflows/lint.yml create mode 100644 .hadolint.yml create mode 100644 .pre-commit-config.yaml diff --git a/.github/workflows/lint.yml b/.github/workflows/lint.yml new file mode 100644 index 0000000..13f8e2c --- /dev/null +++ b/.github/workflows/lint.yml @@ -0,0 +1,40 @@ +--- +name: Lint + +on: # yamllint disable-line rule:truthy + pull_request: + push: + branches: + - main + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +permissions: + contents: read + +env: + COLUMNS: '120' + FORCE_COLOR: '1' + +jobs: + pre-commit: + runs-on: ubuntu-latest + name: pre-commit + steps: + - name: Checkout + uses: actions/checkout@v4 + with: + persist-credentials: false + - name: Set up Python + uses: actions/setup-python@v5 + with: + python-version: '3.13' + cache: pip + - name: Install packages + run: python -m pip install --upgrade pip pre-commit + - name: Run linting tools + run: pre-commit run --all-files + env: + PRE_COMMIT_COLOR: always diff --git a/.hadolint.yml b/.hadolint.yml new file mode 100644 index 0000000..1e55b63 --- /dev/null +++ b/.hadolint.yml @@ -0,0 +1,5 @@ +--- +ignored: + - DL3013 # Don't complain when upgrading to latest version of pip + - DL3022 # Due to use of additional_contexts in compose.yaml + - DL3042 # Caching is desired with RUN --mount=type=cache diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml new file mode 100644 index 0000000..1aa3f4c --- /dev/null +++ b/.pre-commit-config.yaml @@ -0,0 +1,53 @@ +--- +ci: + autoupdate_schedule: weekly +repos: + - repo: meta + hooks: + - id: check-hooks-apply + - id: check-useless-excludes + - repo: https://github.com/pre-commit/pre-commit-hooks + rev: v5.0.0 + hooks: + - id: check-added-large-files + - id: check-builtin-literals + - id: check-case-conflict + - id: check-docstring-first + - id: check-executables-have-shebangs + - id: check-merge-conflict + - id: check-shebang-scripts-are-executable + # - id: check-toml + - id: check-vcs-permalinks + - id: check-yaml + - id: debug-statements + - id: detect-private-key + - id: end-of-file-fixer + - id: file-contents-sorter + args: [--unique] + files: ^(?:packages|requirements)\.txt$ + - id: fix-byte-order-marker + - id: fix-encoding-pragma + args: [--remove] + - id: requirements-txt-fixer + - id: trailing-whitespace + - repo: https://github.com/astral-sh/ruff-pre-commit + rev: v0.8.2 + hooks: + - id: ruff + args: [--exit-non-zero-on-fix, --fix] + - id: ruff-format + args: [--check] + - repo: https://github.com/adrienverge/yamllint.git + rev: v1.35.1 + hooks: + - id: yamllint + args: [--strict] + - repo: https://github.com/hadolint/hadolint + rev: v2.12.0 + hooks: + - id: hadolint-docker + name: hadolint + - repo: https://github.com/woodruffw/zizmor-pre-commit + rev: v0.9.2 + hooks: + - id: zizmor From 224942cabe4348b7a805bc008f56582cc7dac930 Mon Sep 17 00:00:00 2001 From: Nick Pope Date: Sat, 7 Dec 2024 21:59:30 +0000 Subject: [PATCH 41/48] Added more entries to `.gitignore` file. --- .gitignore | 3 +++ 1 file changed, 3 insertions(+) diff --git a/.gitignore b/.gitignore index 9f11b75..6ee304b 100644 --- a/.gitignore +++ b/.gitignore @@ -1 +1,4 @@ +__pycache__/ .idea/ +.mypy_cache/ +.ruff_cache/ From 2f3a8c13be4b49f35b55675419a31109c88aad6d Mon Sep 17 00:00:00 2001 From: Nick Pope Date: Sat, 7 Dec 2024 21:59:53 +0000 Subject: [PATCH 42/48] Reordered entries in the `.env` file. --- .env | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/.env b/.env index d8445b5..0ad48cd 100644 --- a/.env +++ b/.env @@ -1,7 +1,7 @@ PYTHON_IMPLEMENTATION=python PYTHON_VERSION=3.10 -POSTGRESQL_VERSION=13 -POSTGIS_VERSION=3.0 -MYSQL_VERSION=8 MARIADB_VERSION=10.5 +MYSQL_VERSION=8.0 ORACLE_VERSION=23.5.0.0 +POSTGRESQL_VERSION=13 +POSTGIS_VERSION=3.0 From 2927b5345b5dab1ea4c2a1d07b6e6d71071883ee Mon Sep 17 00:00:00 2001 From: Nick Pope Date: Sat, 7 Dec 2024 22:00:47 +0000 Subject: [PATCH 43/48] Updated the charset for MariaDB/MySQL. Recently this was updated to use `utf8mb4` for Django 5.2 --- settings.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/settings.py b/settings.py index c5f3aad..40cd183 100644 --- a/settings.py +++ b/settings.py @@ -19,7 +19,7 @@ def _build_databases_setting(): } if engine.endswith(".mysql"): - entry["TEST"] = {"CHARSET": "utf8"} + entry["TEST"] = {"CHARSET": "utf8mb4"} if engine.endswith(".oracle"): entry |= { From 168aa916e18bffe4c469bdae6d20649a703b4f1e Mon Sep 17 00:00:00 2001 From: Nick Pope Date: Sat, 7 Dec 2024 22:01:43 +0000 Subject: [PATCH 44/48] Removed stale container image cache location. Hopefully we'll reinstate this in the future, but make use of GitHub Container Registry. It'll be easier to maintain and we should be able to publish all versions on a regular basis from a GitHub Actions workflow. --- compose.yml | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/compose.yml b/compose.yml index 98536ed..3bb91e4 100644 --- a/compose.yml +++ b/compose.yml @@ -1,15 +1,13 @@ --- x-base: &base - image: djangobox/django-docker-box:${PYTHON_IMPLEMENTATION}-${PYTHON_VERSION} + image: django-docker-box:${PYTHON_IMPLEMENTATION}-${PYTHON_VERSION} build: context: . dockerfile: ${PWD}/Containerfile args: - PYTHON_IMPLEMENTATION=${PYTHON_IMPLEMENTATION} - PYTHON_VERSION=${PYTHON_VERSION} - cache_from: - - djangobox/django-docker-box:${PYTHON_IMPLEMENTATION}-${PYTHON_VERSION} additional_contexts: src: ${DJANGO_PATH:-../django} volumes: From ca67e3b7044ba6874eea0a63a6b2c027bd591647 Mon Sep 17 00:00:00 2001 From: Nick Pope Date: Sat, 7 Dec 2024 22:03:00 +0000 Subject: [PATCH 45/48] Reworked commands for building documentation, etc. --- compose.yml | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/compose.yml b/compose.yml index 3bb91e4..b24d496 100644 --- a/compose.yml +++ b/compose.yml @@ -385,12 +385,6 @@ services: # Commands: Other - docs: - <<: *base - entrypoint: ["make", "spelling"] - working_dir: /tests/django/docs - user: root - pre-commit: <<: *base entrypoint: pre-commit run --all-files @@ -399,3 +393,10 @@ services: # XXX: Disable eslint due to issues finding dependencies. # See https://github.com/django/django/pull/18162 SKIP: eslint + + sphinx: + <<: *base + entrypoint: make + working_dir: /django/source/docs + environment: + BUILDDIR: /django/output/docs From ea7b4a61d69c621082d9eed4f0d8e7c498605f15 Mon Sep 17 00:00:00 2001 From: Nick Pope Date: Fri, 13 Dec 2024 20:26:06 +0000 Subject: [PATCH 46/48] Bumped database versions to those supported on Django main. --- .env | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.env b/.env index 0ad48cd..e77a955 100644 --- a/.env +++ b/.env @@ -3,5 +3,5 @@ PYTHON_VERSION=3.10 MARIADB_VERSION=10.5 MYSQL_VERSION=8.0 ORACLE_VERSION=23.5.0.0 -POSTGRESQL_VERSION=13 -POSTGIS_VERSION=3.0 +POSTGRESQL_VERSION=14 +POSTGIS_VERSION=3.1 From dd318875e7ad55b769391dabfc710dddb8006e1a Mon Sep 17 00:00:00 2001 From: Nick Pope Date: Fri, 13 Dec 2024 20:26:28 +0000 Subject: [PATCH 47/48] Removed reference to `${PWD}` from `compose.yaml`. This isn't necessary - we can just use the current directory as there is a lot of other stuff already doing that. Using `${PWD}` makes using this on Windows more awkward as this environment variable is undefined by default. --- compose.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/compose.yml b/compose.yml index b24d496..fc65ac7 100644 --- a/compose.yml +++ b/compose.yml @@ -4,7 +4,7 @@ x-base: &base image: django-docker-box:${PYTHON_IMPLEMENTATION}-${PYTHON_VERSION} build: context: . - dockerfile: ${PWD}/Containerfile + dockerfile: ./Containerfile args: - PYTHON_IMPLEMENTATION=${PYTHON_IMPLEMENTATION} - PYTHON_VERSION=${PYTHON_VERSION} From 06aed4a7b1c0af8bce3aecd4590faee2e9bad216 Mon Sep 17 00:00:00 2001 From: Nick Pope Date: Sat, 7 Dec 2024 22:03:28 +0000 Subject: [PATCH 48/48] Reworked documentation in `README.md`. Add much more guidance on using the tools within this repository. --- README.md | 317 ++++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 273 insertions(+), 44 deletions(-) diff --git a/README.md b/README.md index 929959e..a64b14e 100644 --- a/README.md +++ b/README.md @@ -1,82 +1,311 @@ -# django-docker-box +# Django Toolbox + +Tooling and test execution support for [Django][0] :unicorn: + +:heart: Support Django development by [donating][1] to the Django Software Foundation. + + +## Highlights + +- :test\_tube: Test supported core database backends — MariaDB, MySQL, Oracle, PostgreSQL, SQLite +- :earth\_africa: Test supported core geospatial backends — MariaDB, MySQL, Oracle, PostGIS, SpatiaLite +- :globe\_with\_meridians: Test user interfaces in different browsers using Selenium — Chrome, Edge, Firefox +- :snake: Test using different Python interpreters — CPython, PyPy +- :broom: Execute linting and formatting tools on the Django repository +- :books: Build the project documentation using Sphinx and run spelling and link checkers -Run the django test suite across multiple databases. ## Quickstart -Clone this repository somewhere. Also make sure you have docker and docker-compose installed. +1. Make sure that you have Docker installed. + +2. Clone this repository as well as the Django repository, e.g. + + ```console + $ mkdir ~/Sources + $ cd ~/Sources + $ git clone https://github.com/django/django.git + $ git clone https://github.com/django-docker-box/django-docker-box.git + $ cd django-docker-box + ``` + +> [!IMPORTANT] +> As long as the two repositories are adjacent the Django source repository will be discovered. +> A different path can be specified by setting the `DJANGO_PATH` environment variable. + +3. Build the image: + + ```console + $ docker compose build sqlite + ``` + +4. Run the tests: + + ```console + $ docker compose run --rm sqlite + ``` + + +## Running Tests + +All of the test commands detailed below can be passed additional arguments that +are provided to the `runtests.py` entrypoint. You can see a list of these +arguments by running the following command: + +```console +$ docker compose run --rm sqlite --help +``` + +### Standard Tests + +To run the standard set of tests you can use the following commands: + +```console +$ docker compose run --rm mariadb +$ docker compose run --rm mysql +$ docker compose run --rm oracle +$ docker compose run --rm postgres +$ docker compose run --rm sqlite +``` + +Each of the above commands will run the test suite for a different supported +database. + +More information about [running the unit tests][7] for Django can be found in +the documentation. + + +### Geospatial Tests + +To run tests on geospatial features you can use the following commands: -Ensure that the `DJANGO_PATH` variable points to the root of the Django repo: +```console +$ docker compose run --rm mariadb-gis +$ docker compose run --rm mysql-gis +$ docker compose run --rm oracle-gis +$ docker compose run --rm postgres-gis +$ docker compose run --rm sqlite-gis +``` -`export DJANGO_PATH=~/projects/django/` +Each of the above commands will run the test suite for a different supported +geospatial database. -If you see a docker-compose warning about it not being defined followed by an error ensure that is defined in the shell you are using. +> [!TIP] +> To only run the subset of tests for geospatial features, pass `gis_tests` as +> an argument to specify that only that folder of tests should be collected, +> e.g. +> +> ```console +> $ docker compose run --rm sqlite-gis gis_tests +> ``` -You can now either download the latest image used on the CI servers with the dependencies pre-installed: +More information about [running the GeoDjango tests][9] for Django can be found +in the documentation. -`docker-compose pull sqlite` -Or build it yourself: +### User Interface Tests -`docker-compose build sqlite` +To run tests on user interfaces you can use the following commands: -Then simply run: +```console +$ docker compose run --rm chrome +$ docker compose run --rm edge +$ docker compose run --rm firefox +``` -`docker-compose run --rm sqlite` +Each of the above commands will run the subset of user interface tests for a +different supported web browser. The tests are executed using Selenium. -All arguments are passed to `runtests.py`. Before they are run all specific dependencies are -installed (and cached across runs). +To capture screenshots of certain test cases used for comparison to avoid +regressions, the `--screenshots` flag can be passed. -## Different databases +More information about [running the Selenium tests][8] for Django can be found +in the documentation. -Simply substitute `sqlite` for any supported database: -`docker-compose run --rm postgres [args]` +## Running Tools -`docker-compose run --rm mysql [args]` -`docker-compose run --rm mariadb [args]` +### Linting & Formatting -And if you're mad you can run all the tests for all databases in parallel: +Django uses the following linting and formatting tools: `black`, `flake8`, +`isort`, and `eslint`. To ensure that the correct versions are used, Django +also supports using `pre-commit` which is the mechanism provided here: -`docker-compose up` +```console +$ docker compose run --rm pre-commit +``` -#### Database versions +You can run individual tools by passing them as an argument: -You can customize the version of the database you test against by changing the appropriate `[db]_VERSION` environment variable. See the Configuration section below for the available options and their defaults. +```console +$ docker compose run --rm pre-commit black +$ docker compose run --rm pre-commit blacken-docs +$ docker compose run --rm pre-commit isort +$ docker compose run --rm pre-commit flake8 +$ docker compose run --rm pre-commit eslint # XXX: Currently not working. +``` -## Different Python versions +More information about Django's [coding style][5] can be found in the +documentation. -The `PYTHON_VERSION` environment variable customizes which version of Python you are running the tests against. e.g: +### Building Documentation -`PYTHON_VERSION=3.10 docker-compose run --rm sqlite` +Documentation for Django is built using Sphinx. Run the following to see the +available commands: -You can also pull the pre-built image in the same way: +```console +$ docker compose run --rm sphinx +``` -`PYTHON_VERSION=3.10 docker-compose pull sqlite` +You may find the following builders particularly useful when working on +documentation improvements: -## Utilities +```console +$ docker compose run --rm sphinx dirhtml +$ docker compose run --rm sphinx spelling +$ docker compose run --rm sphinx linkcheck +``` -To run the docs spellchecker: +The `BUILDDIR` environment variable has been set to generate output into the +`./output/docs` path under this repository instead of the usual location in the +Django source repository. You can alter this environment variable to generate +to a different path if required. -`docker-compose run --rm docs` +More information about [writing documentation][6] for Django can be found in +the documentation. -Or pre-commit: -`docker-compose run --rm pre-commit` +### Other -To enter a bash shell within the container, run: +To enter a shell within the container, run: -`docker-compose run --rm --entrypoint bash [database]` +```console +$ docker compose run --rm --entrypoint=bash sqlite +``` ## Configuration -| Environment Variable | Default | Description | -| --- | --- | --- | -| `DJANGO_PATH` | `../django` | The path to the Django codebase on your local machine | -| `PYTHON_IMPLEMENTATION` | `python` | One of `python` or `pypy` | -| `PYTHON_VERSION` | `3.10` | The python version to run tests against | -| `POSTGRESQL_VERSION` | `13` | The version of Postgres to use | -| `MYSQL_VERSION` | `8` | The mysql version to use | -| `MARIADB_VERSION` | `10.5` | The mariadb version to use | -| `ORACLE_VERSION` | `23.5.0.0` | The Oracle version to use | +The build of the container image can be customized by setting the following +environment variables: + +| Environment Variable | Default Value | Description | +| ----------------------- | ------------- | ---------------------------------------------------- | +| `DJANGO_PATH` | `../django` | Path to the Django repostory on your local machine | +| `PYTHON_IMPLEMENTATION` | `python` | Implementation of Python to use — `python` or `pypy` | +| `PYTHON_VERSION` | `3.10` | Version of Python container image to use | + +The versions of various backend services can be switched by setting these environment variables: + +| Environment Variable | Default Value | Description | +| ----------------------- | ------------- | ---------------------------------------------------- | +| `MARIADB_VERSION` | `10.5` | Version of MariaDB container image to use | +| `MYSQL_VERSION` | `8.0` | Version of MySQL container image to use | +| `ORACLE_VERSION` | `23.5.0.0` | Version of Oracle container image to use | +| `POSTGRESQL_VERSION` | `14` | Version of PostgreSQL container image to use | +| `POSTGIS_VERSION` | `3.1` | Version of PostGIS extension to use | + + +### Python Versions + +The `PYTHON_VERSION` environment variable controls which version of Python you +are running the tests against, e.g. + +```console +$ PYTHON_VERSION=3.10 docker compose run --rm sqlite +``` + +In addition, it's possible to select a different implementation of Python, i.e. +PyPy instead of CPython, by setting the `PYTHON_IMPLEMENTATION` environment +variable, e.g. + +```console +$ PYTHON_IMPLEMENTATION=pypy docker compose run --rm sqlite +``` + +Be warned, however, that support for PyPy is not as complete and there are more +restrictions with respect to the range of versions available. + +### Database Versions + +Most database container images are pulled from [Docker Hub][2]. Oracle database +is pulled from the [Oracle Container Registry][3]. + +You can switch the version of the database you test against by changing the +appropriate environment variable. Available options and their defaults can be +found in the [configuration section](#Configuration). + +> [!WARNING] +> Be aware that only a single version of a particular database may be running +> at one time, so you will need to ensure that you tear down the previously +> running instance before starting up the new one, e.g. +> +> ```console +> $ docker compose ps --format='{{.Image}}' postgresql-db +> postgres:13-alpine +> $ docker compose down postgresql-db +> [+] Running 1/1 +> ✔ Container django-docker-box-postgresql-db-1 Removed 0.2s +> $ POSTGRESQL_VERSION=17 docker compose up --detach postgresql-db +> [+] Running 1/1 +> ✔ Container django-docker-box-postgresql-db-1 Started 0.3s +> $ docker compose ps --format='{{.Image}}' postgresql-db +> postgres:17-alpine +> ``` +> +> Alternatively, run the following to tear down the whole stack before bringing +> up new containers running different versions: +> +> ```console +> $ docker compose down +> ``` + +> [!NOTE] +> +> Unlike other GIS database backends, for PostgreSQL with PostGIS you will need +> to specify both versions: +> +> ```console +> $ POSTGRESQL_VERSION=17 POSTGIS_VERSION=3.5 docker compose up --detach postgresql-gis-db +> ``` + +To determine what database versions can be used you can check the release notes +for the branch of Django that you have checked out, or alternatively there is +the [supported database versions][4] page on Django's Trac Wiki. + + +### Other Versions + +For the Memcached, Redis, and Selenium container images, the latest container +image tag is always used. + +Where possible, for backend services, we also use Alpine images where available +for smaller image size and sometimes improved performance. + + +## Roadmap + +The following list is a collection of ideas for improvements that could be made +with no promises that they'll be delivered: + +- Add a monthly scheduled full test matrix execution using GitHub Actions +- Add support for some third-party databases, e.g. CockroachDB, SQL Server +- Add support for test coverage execution and report generation +- Add support for running accessibility tooling and report generation +- Support report generation during monthly runs and publish to GitHub Pages +- Publish pre-built container images to the GitHub Container Registry +- Support testing against different versions of SQLite and SpatiaLite +- Support running with Podman in addition to Docker +- Support generating screenshots into `./output/screenshots/` + + +[0]: https://www.djangoproject.com/ +[1]: https://www.djangoproject.com/fundraising/ +[2]: https://hub.docker.com/search?badges=official&badges=open_source +[3]: https://container-registry.oracle.com/ords/ocr/ba/database/free +[4]: https://code.djangoproject.com/wiki/SupportedDatabaseVersions +[5]: https://docs.djangoproject.com/en/stable/internals/contributing/writing-code/coding-style/ +[6]: https://docs.djangoproject.com/en/stable/internals/contributing/writing-documentation/ +[7]: https://docs.djangoproject.com/en/stable/internals/contributing/writing-code/unit-tests/#running-the-unit-tests +[8]: https://docs.djangoproject.com/en/stable/internals/contributing/writing-code/unit-tests/#running-the-selenium-tests +[9]: https://docs.djangoproject.com/en/stable/ref/contrib/gis/testing/#geodjango-tests