How to upgrade existing isle-dc site to Drupal 10 / Drush 12 / PHP 8.x (meta-issue)

[ysuarez]

In case it helps, I wanted to create this meta-issue to keep track of open issues and PRs that will help us run an existing ISLE-DC site with Drupal 10 / Drush 12 / PHP 8.x (meta-issue).

Apologies in advance if you do not find this meta-issue helpful, and I welcome your feedback.

Available isle-buildkit releases:

https://github.com/Islandora-Devops/isle-buildkit/releases
https://github.com/Islandora-Devops/isle-buildkit/releases/tag/2.0.7

Related PRs:

• remove recast #346
• Fix php-fpm PID check for php8.1 #347
• Update docker-compose.cantaloupe.yml #360
• Add english OCR training data isle-buildkit#292
• update advanced search for new Drupal "once" function Islandora/advanced_search#41
• Fix to use Drupal once library Islandora-Labs/views_nested_details#5
• add mariadb-server-utils to mariadb image isle-buildkit#294
• added matomo upgrade instructions Islandora/documentation#2271

Documentation PRs:

• Update docker-maintain-isle.md Islandora/documentation#2264

Related issues:

• [CODE TASK] Tests need to be updated for new Drupal 10.2 Add Field page. Islandora/islandora#989
• Make starter_dev fails to build #354
• make demo fails to build #355
• 2.0.0 Release  isle-buildkit#261
• Matomo docker volume causes issues when upgrading Matomo's version #361
• Changes needed to install Drupal 10 islandora-playbook#269

Related PRs for upgrade to PHP 8.2:

• Add PHP 8.2 and Drupal 10.2 to build matrix. Islandora/openseadragon#58
• Add PHP 8.2 and Drupal 10.2 to testing matrix. Islandora/jsonld#75
• Update PHP and Drupal versions in testing matrix. Islandora/controlled_access_terms#108
• Add PHP 8.2 <strike>and Drupal 10.2</strike> to testing matrix Islandora/islandora#987
• Add php 8.2; remove php 7.4 from testing matrix Islandora/Crayfish#179

Related issues for upgrade to PHP 8.2:

• PHP 8.2 Compatibility Islandora/documentation#2262
• PHP 8.2 isle-buildkit#293
• [BUG] utf8_encode() has been deprecated. Islandora/islandora#988

@joshdentremont
@aOelschlager
@joecorall
@nigelgbanks

[aOelschlager]

Adding some reference material.
Solr upgrade docs: https://solr.apache.org/guide/8_0/major-changes
/-in-solr-8.html#major-changes-in-solr-8
Matomo upgrade docs: https://matomo.org/faq/on-premise/update-matomo/

[highermath]

Is anyone working on this?

[ysuarez]

@highermath my understanding is that @joecorall has provided two key PRs (see above) that need testing and merging to push this upgrade forward. I think there might be one or more additional tasks needed for this issue to be closed, but merging those two PRs are key.

[joshdentremont]

Solr Search API requires updating the Solr config when you update the module to 4.3.0, which should be the same process as when we upgrade Solr from 7 to 8.

I am hoping to try updating Isle to 2.0.2 soon, but in the meantime, here are the instructions that worked for me to update the Solr config when I upgrade the Search API module.

Download config.zip from Drupal UI at /admin/config/search/search-api/server/default_solr_server
Copy it into the solr container (docker compose cp ~/Downloads/solr_7.x_config.zip solr:/opt/solr/server/solr/ISLANDORA/conf)
SSH into container (docker compose exec solr with-contenv bash)
cd /opt/solr/server/solr/ISLANDORA/conf
unzip solr_7.x_config.zip
Replace all when prompted
rm solr_7.x_config.zip
exit
docker compose restart solr
make reindex-solr

[noahwsmith]

What happens if you run make solr-cores again? That seems very close to the steps that includes...

[joecorall]

@joshdentremont this works with TAG=main until we get a new 2.x tag with Islandora-Devops/isle-buildkit#290

@noahwsmith can we get a bumped tag?

[noahwsmith]

yep! here you go https://github.com/Islandora-Devops/isle-buildkit/releases/tag/2.0.3

[joshdentremont]

@noahwsmith good catch. I just updated the solr search API to 4.3.0 and ran make solr-cores and it updated the config for me. That's much easier than the manual steps.

[joshdentremont]

I was able to update my site to Buildkit 2.0.3 and Drupal 10 and everything seems to be working except Matomo, which seems to be on version 3.13.5 and not be compatible with PHP 8.

I did this update on my local dev site. I have a staging and production site as well so I will comment again once I get it working on the staging site.

Below are two sets of steps. One that I think will work, and that we should test and possibly add to the documentation, and the exact steps I took that did work, but could be simplified.

One thing to note: this required removing all my data from Solr and recreating the core, then reindexing everything. It seemed like my existing core had been corrupted after the update to 8.x. There may be a way to avoid this, but I think you probably want to reindex anyway, because we are generating a new Solr config for 8.x.

Steps for documentation:

1. git pull the latest isle-dc
2. Edit the TAG variable in your .env file to 2.0.3 (or higher)
3. make -B docker-compose.yml
4. make pull && make down && make up
5. docker compose exec -T solr with-contenv bash -lc 'rm -r server/solr/ISLANDORA'

• Note that this deletes all data indexed in Solr

6. make solr-cores
7. make reindex-solr
8. You should now have PHP 8 and Solr 8, but you will still be on Drupal 9
9. Update your composer requirements for Drupal 10 and Drush 12

• see https://github.com/Islandora-Devops/islandora-starter-site/blob/main/composer.json for an example

10. docker compose exec -T drupal with-contenv bash -lc 'drush updb'

These are the exact steps I took that worked for me:

• Edit .env to specify TAG=2.0.2
• Make -B docker-compose.yml
• Make down
• Make pull
• Make up
• Make solr-cores
• docker compose exec solr with-contenv bash
  • cd server/solr
  • rm -r ISLANDORA
  • exit
• Make solr-cores
• Make reindex-solr
• Make down
• Update TAG to 2.0.3
• Make -B docker-compose.yml
• Make pull
• Make up
• Make solr-cores
• Make reindex-solr
• Update composer.json
  • to have the version numbers from starter site (https://github.com/Islandora-Devops/islandora-starter-site/blob/main/composer.json)
  • some namespaces changed from Islandora to Drupal since I created my site, so I changed those too
• Drush Database updates

[joshdentremont]

On Staging/Production:

1. git pull the latest isle-dc
2. update .env to have TAG=2.0.3 (same as you have locally)
3. make -B docker-compose.yml
4. make pull && make down && make up
5. docker compose exec -T drupal with-contenv bash -lc 'drush updb'
6. docker compose exec -T solr with-contenv bash -lc 'rm -r server/solr/ISLANDORA'
7. docker compose restart solr
8. make solr-cores
9. make reindex-solr

On both my dev site and staging site, Solr was acting up after the upgrade. On staging I had to remove the ISLANDORA directory, then restart Solr for it to clear out. Without doing that I was not able to run make reindex-solr

This also assumes you are using the Makefile to build, push, and pull your custom Drupal image. If you are not doing it that way, step 4 might be a little different, but the rest should be the same.

[jefferya]

Regarding Matomo, for the isle-buildkit TAG v1 to v2 upgrade, here's what locally, we found as the problem in the past.

isle-buildkit uses /var/www/matomo as a working directory to download and install Matomo during image creation (details). The Matomo version is v4 in isle-buildkit TAG v2

isle-dc creates a volume at mounted at /var/www/matomo (details).

    volumes:
      - matomo-config-data:/var/www/matomo

What is stored in the local storage volume overrides the Matomo files embedded in the isle-buildkit image TAG v2. Thus, if you start with TAG v1 and update to v2 isle-buildkit, you will get Matomo from your local volume (e.g., Matomo v3). However, if starting with a clean (i.e., empty volumes) then one would get the version of Matomo embedded in the isle-buildkit image. The documentation behind the behaviour.

Locally, we've moved away from Matomo before digging into a solution so I don't have solution. One thought, untested, update Matomo via the Matomo web interface which will update the contents of the storage volume as a first step (assumes there is a version of Matomo that works with both PHP 7 and 8). This might be off-base so use with caution.

[g7morris]

We have found changing the volume bind mount for matomo to

    volumes:
    - matomo-config-data:/var/www/matomo/config:rw

Works well. One just needs to copy the config.ini.php file manually to upgrade.

• Before making the changes, docker cp matomo-container-name:/var/www/matomo/config/config.ini.php to a location on the host disk
• shutdown containers
• push changes
• docker cp config.ini.php matomo-container-name:/var/www/matomo/config/

[joshdentremont]

@aOelschlager and I found another issue where Cantaloupe was not working with Isle and buildkit 2.0.0 I just put in a PR for this #360

The instructions are the same once this PR goes through. The first 2 steps of pulling isle-dc and running make -B docker-compose.yml will pull in those changes for Cantaloupe as well

[joecorall]

Hypercube running on islandora/hypercube:2.0.3 is not generating OCR for TIFF's Islandora/Crayfish#176

Update: Islandora-Devops/isle-buildkit#292 resolves the hypercube:2.x issue.

[joshdentremont]

Now that there is a fix for advanced search compatibility (Islandora/advanced_search#41) I think everything is ready to go, except Matomo. I have created an issue for Matomo, at #361

I also have a documentation PR being tested to give instructions on how to do the upgrade: Islandora/documentation#2264

I am running Drupal 10 in staging now and planning to update production next week and everything seems to be working well.

I think we can probably close this once the documentation PR is merged, and the Matomo issue is dealt with. My only concern is that it seems most of us are not using Matomo, so I'm not sure how to address that issue. I think someone who has an existing site, that uses matomo, might need to do some testing of the options g7morris and jeffreya suggested above.

[joshdentremont]

Another issue that will affect anyone upgrading from 1.x to 2.x is mariadb errors in the logs. This doesn't seem to cause any major issues, but it does really clutter up the logs. I have a PR to fix this in Buildkit and have updated the instructions in the documentation PR above for how to deal with this.

See the discussion in slack: https://islandora.slack.com/archives/CM6F4C4VA/p1696525259478829

[joshdentremont]

The Mariadb issue is now fixed with buildkit 2.0.5. Upgrade instructions have been updated to include how to fix the database errors in the logs: https://islandora.github.io/documentation/installation/docker-maintain-isle/#version-1x-to-2x

[joshdentremont]

fcrepo imports are failing for me after switching to 2.x.

See Islandora-Devops/isle-buildkit#304 for more details.

This is easily fixed by editing the Makefile. I have included an issue in isle-dc so we can discuss this at the next tech call. (#368)