Skip to content

Running a production WeBWorK server using Docker

Jump to bottom Edit New page
Nathan Wallach edited this page Jan 3, 2022 · 12 revisions

Introduction

It is possible to run a reasonable production WeBWorK server using Docker. It was already possible to do this using WeBWorK 2.14, but the 2.15 release will include a docker-compose.yml file and several sample configuration files to make setting up a production system much easier.

At present, this includes an R server, a MariaDB server (MariaDB 10.4 for WW 2.15), and a WeBWorK server running Apache. (In principle, it would also be possible to run a Docker system connecting to an external mysql-derived database server.)

In the future, the Docker system may be extended to also allow using lighthttpd to reduce the workload on Apache.

Main configuration

Configuration in the docker-compose.yml file and files it may mount to a running container:

  1. Before first starting your Docker based system you should change the default sample SQL passwords. Once the Maria DB data volume is created, you would need to modify those values by hand via direct mysql commands.
    • MYSQL_ROOT_PASSWORD is the root password for mysql, and appears only once in the file.
      • This is still set in docker-compose.yml.
    • The WEBWORK_DB_PASSWORD and WEBWORK_DB_USER are now set via the .env file and not via the docker-compose.yml file.
    • MYSQL_PASSWORD is the password used by the WeBWorK server to access the webwork database and the same value needs to also be set for WEBWORK_DB_PASSWORD: in the environment: section further down in the file.
      • These values are now both set via the .env file and not via the docker-compose.yml file.
  2. Consider where to store docker-compose.yml:
    • If you are using a shared webwork2 directory on shared storage to operate several VMs each running WeBWorK via Docker, you will want to store your docker-compose.yml per VM in a special per-VM directory, and have each one point to the central location of webwork2 using the settings in the build: section of the file. Sample lines are included.
    • Using a special directory to store docker-compose.yml also allows you to use modifications to the file to keep different Docker images of webwork available in parallel. This is helpful when you upgrade the Docker image (to get core OS upgrades, etc.) or build a different Docker image of WW for any other reason. Some comments on this are given below in the section on updating the Docker image.
  3. It is possible to mount webwork2/ and/or pg/ from outside the standard image.
    • Those options are intended for people doing development work, but is also useful is you need a customized version of the code.
    • Small customizations could be better made by mounting just specific modified files from outside the image.
      • By keeping locally customized files outside the main webwork2 tree and mounting them to the image using specific lines in docker-compose.yml you can easily get such local versions into upgraded WW2 Docker images.
  4. The location from which the courses directory is mounted should be set.
    • This can be set using the value of COURSES_DIRECTORY_ON_HOST in .env or directly set by making suitable changes to docker-compose.yml.
    • The current default setting for Docker use is to store it in ../ww-docker-data/courses namely, that a ww-docker-data directory will be in parallel to the webwork2 directory.
    • For production servers, a more appropriate location should be used.
  5. Several other directories and files should be mounted from a persistent location outside of the docker container.:
    • the WeBWorK log file directory webwork2/logs/,
    • the Apache logs directory /var/log/apache2 (possibly relocated by Apache config)
    • the WeBWorK webwork2/htdocs/tmp directory,
    • htdocs/my_site_info.txt
    • (optional) the OPL
    • local WeBWorK configuration files:
      • conf/localOverrides.conf
      • conf/site.conf
      • conf/authen_LTI.conf
  6. Local Apache configuration / default files should be created and mounted from appropriate locations:
  7. Set the hostname:
  8. Adjust the ports: for production vs. personal PC use. See the setting in .env and check if it should be changed/used.
  9. As needed modify the environment variables set in the .env file:
    • The SQL database password and user: WEBWORK_DB_PASSWORD and WEBWORK_DB_USER.
    • COURSES_DIRECTORY_ON_HOST which is used to provide a value to mount (by default) to /opt/webwork/courses in the default version of docker-compose.yml.
    • WEBWORK2_HTTP_PORT_ON_HOST sets the host port HTTP number used to which connects to the server.
      • By default is is 8080 as suitable for using Docker on a development machine.
      • For production use, it could be changed to 80, but it would be simpler to just switch which lines are commented/uncommented in the ports: section of docker-compose.yml.
  10. Set/adjust the environment variables section environment:
    • very important WEBWORK_DB_PASSWORD:
      • now set via .env so both settings have a single source.
      • If either is manually set then the same password must be used here and in MariaDB section (not the mysql root password).
    • SSL: 1 to turn on SSL
    • PAPERSIZE: size to change the default system paper-size (defaults to letter and a4 or something else may be desired)
    • ADD_LOCALES: can be used to set which locales which will be generated and available in the running container.
    • ADD_PACKAGES: can be used to have additional Ubuntu packages (ex vim) installed in the running container. (Such packages additions are not persistent and will be reinstalled each container start-up.)
    • SYSTEM_TIMEZONE: can set the server timezone of the running container. (The default is UTC.)
    • WEBWORK_ROOT_URL: can set the URL, and should be used in particular if you are using SSL.
    • WEBWORK_SMTP_SERVER:, WEBWORK_SMTP_SENDER: sets these environment variables used by the running container.
    • WEBWORK_TIMEZONE: sets the timezone WeBWorK uses by default.

Configuring max_connections for the database

Warning: I have found that the default setting of max_connections for the MariaDB container is too low for production use. As a result, I copied /etc/mysql/my.cnf out of a running container, and edit the value for max_connections and now mount the modified file into the MariaDB container using docker-compose.yml.

Symptoms of trouble: lines reporting error instantiating DB driver WeBWorK::DB::Driver::SQL in the Apache error.log file, and/or reports from students about error pages where that is reported.

There are quite a few discussions in the forums about the need to use a relatively large value of max_connections:

Backing up the SQL database outside the docker storage volume

It is recommended to backup the courses data and the SQL database data.

See: http://webwork.maa.org/wiki/Backup_and_Disaster_Recovery

Since the courses data is mounted from outside the Docker container, regular procedures can be used to back that data up.

However, the SQL database data is stored in a named storage volume.

Here is one possible approach using a modified method:

  1. Create an external "database backup" directory and the local file to be mounted to the container.
  2. Edit docker-compose.yml to mount the directory and files.
    • the external "database backup" directory
    • /root/.my.cnf from the relevant file with the correct password.
      • Make sure to fix the value of the password in the file.
    • /root/ww-mysqldump from a file containing the file below, modified as necessary, and made executable.
  3. Restart the container:
docker-compose down
docker-compose up -d
  1. Test it once from the command line using something like the line below. The line below assumes that you are running under "webwork2/" so that precedes the app_1 in the container name, and mounted the script to the location given.
docker container exec -i webwork2_app_1 /root/ww-mysqldump
  1. Set up a cron job (for a user in the docker group) by editing your crontab using crontab -e and adding a line like
30 1 * * * /usr/bin/docker container exec -i webwork2_app_1 /root/ww-mysqldump
  1. Set up some backup of the dump files on a different server.

Sample for /root/.my.cnf:

[client]
user=webworkWrite
password=passwordRW
host=db
port=3306
default-character-set=utf8mb4

Change the password to match what you set.

Sample for /root/ww-mysqldump:

#!/bin/bash
HOME=/root
# Some versions of mysqldump need "--column-statistics=0" below and others do not.
/usr/bin/mysqldump --column-statistics=0 --opt webwork | /bin/gzip -c > /database_backup/webwork.sql.gz
#/usr/bin/mysqldump --opt webwork | /bin/gzip -c > /database_backup/webwork.sql.gz

It seems that with mysqldump from Ubuntu 20.04 the extra --column-statistics=0 switch is needed when the actual database is on Maria DB 10.4.

Change the backup path to match what you set.

Sample mount lines to add to docker-compose.yml:

      # For mysql backup
      - "/your_path_to/root-my.cnf:/root/.my.cnf"
      - "/your_path_to/ww-mysqldump:/root/ww-mysqldump"
      - "/your_path_to/database_backup:/database_backup"

Change the paths to match where things are on your system.

Updating the Docker images - to get core OS upgrades, etc.

The Docker image for webwork is built on top of an ubuntu system image and with many additional packages installed.

Just as a real server needs to be maintained by updating packages, so does the WeBWorK image.

One benefit of using Docker is that by renaming the image, you can keep the prior image (for emergency rollback) when you create a new image.

Below are my notes on the process I used to do this.

Pull updated Docker images the system and build-process depend on

The lines below assume that WW is still using the noted versions of ubuntu and mariadb. Adjust as necessary:

docker image ls
docker pull ubuntu:18.04
docker pull alpine:latest
docker pull alpine/git
docker pull mariadb:10.4
docker image ls

Old and unneeded images can be removed using docker image rm hash.

Make sure you have the current version of WW (if necessary)

Assuming you want to use the current version of WW (and in particular of the Dockerfile) with any updates:

  1. First change into the location where you have your webwork2 directory (the one used when you built the images).
  2. Check if you have any local modifications to Dockerfile and if so save a copy for comparison.
    • If you have the local version of Dockerfile stored outside the main webwork2 tree - look at that file, and keep a backup copy if relevant.
  3. Save a copy of your local version of docker-compose.yml (from wherever you keep it).
    • You may need to git checkout -- docker-compose.yml before you can run git pull below if you store your locally modified file in the main webwork2 tree.
    • Keeping the different versions of docker-compose.yml which refer to different image builds in separate directories will make switching between which image to run easier.
    • When doing so, you should set the context: in the build: section to point to path to where your main (Git controlled) webwork2 directory is located.
    • If you need a custom Dockerfile it can be specified using dockerfile: in the build: section, and that allows keeping the locally modified file outside the Git controlled webwork2 tree.
    • Note: You will need to run the docker-compose commands in the directory where you have the relevant version of docker-compose.yml.
  4. Use git branch to check that you are on the expected branch (ex. master).
  5. If necessary change branch using git checkout branchname.
  6. Fetch and pull updates
git fetch origin
git pull
git status
  1. Merge in any local modifications to Dockerfile as necessary.
    • If such modifications are needed, I recommend storing the Dockerfile elsewhere and using the dockerfile: setting in the build: section of docker-compose.yml.

Build the new image

  1. Make sure you are in the location with the correct version of docker-compose.yml.
  2. Run docker-compose build
  3. Pay attention to the end of the output. If all went well there should be a message Successfully built with the new image hash ID, and possibly followed by one Successfully tagged with the image name you are using.
  4. Check what docker image ls shows.
  5. Most likely there will be some temporary images from the build process which can be removed using docker image rm.
  6. You probably will want to save the prior webwork and ubuntu images until you are sure that the new image is working properly.

Restart the container with the new image.

  1. Change into the location of the old docker-compose.yml file.
  2. docker-compose down
  3. Change into the location of the new docker-compose.yml file.
  4. docker-compose up -d
  5. Check that your container is working as expected.

Clean up unneeded Docker images

When you are certain you do not need the older Docker images, you can remove them.

Use docker image ls to see what images there are and docker image rm hash to remove unneeded images.

Warning: Do not rush to remove your old images until you are certain that you will not want to roll-back to the prior image.

Add a custom footer
Add a custom sidebar
Clone this wiki locally