Planned changes

[nigelgbanks]

Sandbox Ideas

• Introduction
• Repository layout
• Docker
  • Drupal
  • Solr
• Terraform
• Packer
  • OVA
  • Droplet
• Releases
• Development
• SSL Certificates

Introduction

The goal for this is to outline how I would setup automation around sandbox such
that minimal to no intervention is needed.

Repository layout

├── docker
│   ├── drupal
│   │   ├── Dockerfile  # Builds custom Drupal docker image extends `islandora/drupal`.
│   │   └── rootfs
│   │       ├── etc
│   │       │   └── cont-init.d
│   │       │       ├── 04-install-drupal.sh # Installs drupal site if not already installed.
│   │       │       └── 05-content-sync.sh # Performs content sync if not already performed.
│   │       └── var
│   │           └── www
│   │               └── drupal
│   │                   ├── assets
│   │                   │   ├── 0001-drupal-core-patch.patch # Patches for drupal, applied during `composer install`.
│   │                   │   ├── 0001-module-patch.patch # Appended to `default.settings.php` sets config from environment variables. Applied during `composer install`.
│   │                   │   └── default.settings.php # Patches for modules, applied during `composer install`.
│   │                   ├── composer.json
│   │                   ├── composer.lock
│   │                   ├── config
│   │                   │   └── sync # Drupal site configuration, `drush site-install` uses this config on install. During development can be updated with `drush cex`.
│   │                   ├── content # Content sync folder which is used to populated the site with content.
│   │                   └── web
│   │                       ├── modules # Custom modules (likely will remain empty).
│   │                       ├── profiles
│   │                       │   └── islandora_install_profile_demo # Git submodule linking to the respective repository.
│   │                       └── themes
│   │                           ├── islandora_base_theme # Git submodule linking to the respective repository.
│   │                           └── islandora_starter_theme # Git submodule linking to the respective repository.
│   └── solr
│       ├── Dockerfile  # Builds custom Solr docker image extends `islandora/solr`, basically only config.
│       └── rootfs
│           └── opt
│               └── solr
│                   └── server
│                       └── solr
│                           └── default
│                               ├── conf
│                               │   ├── accents_en.txt
│                               │   ├── accents_und.txt
│                               │   ├── elevate.xml
│                               │   ├── protwords_en.txt
│                               │   ├── protwords_und.txt
│                               │   ├── schema_extra_fields.xml
│                               │   ├── schema_extra_types.xml
│                               │   ├── schema.xml
│                               │   ├── solrconfig_extra.xml
│                               │   ├── solrconfig_index.xml
│                               │   ├── solrconfig_query.xml
│                               │   ├── solrconfig_requestdispatcher.xml
│                               │   ├── solrconfig.xml
│                               │   ├── solrcore.properties
│                               │   ├── stopwords_en.txt
│                               │   ├── stopwords_und.txt
│                               │   ├── synonyms_en.txt
│                               │   └── synonyms_und.txt
│                               └── core.properties
├── docker-compose.yml # Used for local development / deployment on OVA only, likely to be several files but one shown here for simplicity.
├── .github
│   └── workflows
│       ├── pull.yml # Runs lints / test etc.
│       ├── push.yml # Runs lints / test etc.
│       └── release.yml # Pushes Docker image, OVA file, Digital Ocean Droplet Image.
├── LICENSE
├── packer # Packer is used to create OVA sandbox image & Digital Ocean image.
│   ├── sandbox.pkr.hcl # Packer build file.
│   └── sandbox.service # Systemd unit file used to automatically start the docker images when the server starts.
├── README.md
└── terraform  # Terraform is used to create/remove Droplets and any other Digital Ocean resources.
    └── main.tf

Docker

This repository will produce two Docker images.

Drupal

This repository will produce a custom drupal docker image islandora/sandbox this
image, will contain all the content that is required.

• It will install Drupal automatically when started.
• If Docker volumes are used when restarted it will use the existing installation.

Solr

This repository will produce a custom solr docker image islandora/sandbox-solr
this image, will extend the islandora/solr but add the required configuration
so no additional steps on startup are required.

Terraform

Terraform will be used to deploy the Droplet image produced by Packer.

We will continue to use a single small droplet, with a reserved IP, no load balancer
etc, nothing will change here.

Packer

Currently OVA files are stored in Google Drive which is getting close to its
max size 15GB of storage. Ideally we can go with either AWS S3 (Cheapest) or
stick with Digital Ocean and use space for $5 a month. Might as well stick
with Digital Ocean $5 a month is pretty negligible, and we're not likely to
get exceed 250GB.

OVA

"Raking" will not be performed automatically.

Since users of this image may want to interact with the services through
docker we'll use a systemd service that wraps a call to docker compose to
ensure that islandora starts when the VM starts.

Volumes will be used as users will likely want to have their changes persist
across restarts.

Droplet

Volumes will not be used as we'll always want to start from a known clean
state.

docker composer won't be used instead we'll use systemd. This allows us to
force restart other containers when one goes down. For example if Drupal
crashes we'll want to also restart Solr and MariaDB since they would have
old data and we want to start from a clean state.

"Raking" the sandbox will no longer be performed by a script, we'll use
systemd to restart the containers at mid-night. Since they retain no data via
volumes we'll get a freshly installed site.

Releases

Releases versions will not track with Islandora alone, as we need to update the
site with minor fixes, updates to content etc.

Semantic versioning does not make much sense for this use case. As this is not
intended to be consumed like a library, where changes could break some
downstream consumer. I think [Calendar Versioning] makes the most sense here and
will be clearer for consumers of the OVA.

Each time we make a release there will be two artifacts produced:

• OVA Image
• Droplet Image

The OVA image will be pushed Digital Ocean Spaces. This bucket will be
publicly available for download only. The http://islandora.ca site will link
to this bucket instead of the Google Drive.

The Droplet Image will be pushed to Digital Ocean. Upon a successful upload the
old droplet will be stopped and a new one will be started.

All of the above will be triggered automatically be creating a release in
Github. No need to pull down the repository and run any commands manually.

Development

During development bind mounts will be used to the following directories which
will allow the use of drush config:export / drush content-sync:export, etc.
As well as changes to theme, profile, etc.

• docker/drupal/rootfs/var/www/drupal/assets
• docker/drupal/rootfs/var/www/drupal/config/sync
• docker/drupal/rootfs/var/www/drupal/content
• docker/drupal/rootfs/var/www/drupal/web/profiles/islandora_install_profile_demo
• docker/drupal/rootfs/var/www/drupal/web/themes/islandora_base_theme
• docker/drupal/rootfs/var/www/drupal/web/themes/islandora_starter_theme
• docker/drupal/rootfs/var/www/drupal/composer.json
• docker/drupal/rootfs/var/www/drupal/composer.lock

Updating the solr configuration will still be manually as one has to go into the
UI export the configuration and unpack it into this folder
docker/solr/rootfs/opt/solr/server/solr/default.

The basic workflow for changes to the sandbox will be something akin to this:

1. Clone the repository git clone
2. Bring up the images docker compose up --build
3. Visit the site https://islandora.traefik.me
   1. Alter configuration / content, etc.
4. Visit the editor https://ide.traefik.me
   • Export the config drush config:export or use the built in command CTRL-SHIFT-B and select Config Export.
   • Export the content drush content:export or use the built in command CTRL-SHIFT-B and select Content Export.
   • Update / add dependencies to composer composer update / composer require.
   • Make changes to any of the bind mounted theme / profile / modules files.
   • etc.
5. Test changes in fresh environment press CTRL-C in the terminal and rerun docker compose up --build.
6. If happy with the changes commit and push git commit.

SSL Certificates

Additionally on the Droplet we'll deploy an additional Docker image
certbot/certbot. This will allow us to automatically regenerate certificates
without any manual intervention. This will not be included in the OVA image.

[noahwsmith]

Thanks @nigelgbanks . I fully support this direction. CCing @g7morris, @Natkeeran and @dannylamb in case one of them have comments.

My only wish to improve on this plan would be to account for testing ISLE-DC with the sandbox. This is a pretty hard fork from there, so I think we'll have to set up separate flows to build and test isle-dc at another point. I support moving ahead without that because I also worry that trying to do too much at once makes everything needlessly complicated.

Green light from me.

[g7morris]

@nigelgbanks How will the server be maintained? Are you going to create a cronjob to do this monthly e.g. run apt-get updates etc?

[nigelgbanks]

@g7morris I was planning on using CoreOS there is no package manager to run updates, it's pretty much a Linux Kernel, Systemd and Docker. Nothing else is installed on the box.