Add alternate container engine support to Justfile

[chris-t-jansen]

Overview

PR for #7324. Adds alternate container engine support to the justfile for TWiR. This PR was written with Podman in mind and was tested with that, but should work with any drop-in Docker replacement. Use it with any recipe that previously called Docker like so:

$ just <recipe> <container engine>

For example:

$ just website podman

$ just email podman

Warning

Though I feel that what I've written is sound, additionally testing is immensely appreciated just to be sure, especially by the TWiR editors since they are the people who will use it the most.

Docker by default

Docker remains the default container engine, so passing in no arguments (i.e. just website, just email) will use Docker as it always has. This is achieved by having Docker set as a global Just variable here:

default-container-engine := "docker"

Which then gets used as the default value for the newly added container-engine parameter on any recipe that used to call docker. For example:

# Build container engine image (Docker by default)
container-build container-engine=default-container-engine:
	cd .. && {{container-engine}} build -t twir -f publishing/Dockerfile . && cd -

Cons

A refactor like this usually comes with a downside or two, and this one is no exception. The obvious one here is that the justfile is fairly more verbose, with the inclusion of parameters and default values for most recipes. This makes it less easily readable, which may hurt maintainability in the long run.

The other downside has to do with the optimize-email recipe. This recipe originally just called a shell script (create_optimized_email.sh), which itself explicitly called docker. To make it engine-agnostic, I inlined the script into the justfile as a script recipe. Originally I implemented this as a shebang recipe, but due to noexec issues on WSL (see here and here), I had to opt for the unstable [script(COMMAND)] attribute instead. The default COMMAND of sh -eu didn't work for me in testing on WSL, so I opted for bash instead. This could cause issues with portability on machines that don't include or symlink a bash executable, but I had no trouble when testing it on my Windows machine with WSL and with my macOS machine.

The optimize-email recipe issue could also be resolved by refactoring the original script to support command line arguments (e.g. create_optimized_email.sh -c podman), but keeping things in the justfile seemed a more elegant solution to me.

Concerns

If you have any questions, comments, concerns, etc., feel free to let me know. As I said in the original issue I opened, I think this kind of compatibility support is important, but totally understand if the TWiR editors aren't interested in it.

[nellshamrell]

Thank you so much! I will review this this weekend!

[nellshamrell]

Hi there! Just tested this with Docker and WSL and I'm getting this error when running "just website"

Is there a dependency of some sort that I need to install?

[chris-t-jansen]

My guess is that you need a newer version of just. just --version should tell you what version you are using, and I believe you'll need to use version 1.33.0 or newer. I was using 1.43.1 on my macOS machine, and updating for me was as simple as running cargo install just again (God bless cargo).

unstable is a just-specific setting that allows the use of experimental features within the script (see here). The unstable feature here is the [script(COMMAND)] attribute (see here), which I used because of issues with WSL. You can read the detailed explanation in the "Cons" section of the PR, but if it ends up being a blocker for the PR being merged, I can get around it by refactoring the create_optimized_email.sh script to take commandline args.

[chris-t-jansen]

Looks like the clean-website recipe got removed from the just website workflow in a recent commit (9101984), but I wasn't sure if that was intentional so I resolved the conflict for this PR by keeping the workflow as it was. If it should get removed to align better with the current justfile, I'm happy to change it.

[nellshamrell]

Good catch! I'm going to check with that maintainer to see whether that removal was intentional. Also I apologize for taking so long to review this, life got busy, but I will get it done soon.

[nellshamrell]

that was indeed an accident, it's back now!

[chris-t-jansen]

that was indeed an accident, it's back now!

Glad I could help!

I apologize for taking so long to review this, life got busy, but I will get it done soon.

Truly, it's no problem! This is a quality-of-life PR more than anything, it kinda defeats the purpose if it makes your life more difficult. I imagine that when working on a weekly-release publication, it's tough to make time for something that doesn't contribute to the next release. Whenever it gets reviewed is great, it's not urgent.

[nellshamrell]

This overall looks good to me. @tzilist, since you are using Podman, would you be interested in testing this one out?

[chris-t-jansen]

I realized that the "Building" section of the README should also be updated to include instructions about using Docker alternatives, so I added a commit updating that section with instructions using Podman as an example alternative.

As I was updating the "Building" section, I noticed a couple other bits that seemed outdated or slightly misleading, so I ended up updating those as well, though that was perhaps outside the scope of this PR. If it's preferred, I could move that to its own PR.

[nellshamrell]

Ty so much! I do believe that is in the scope of this PR - I've designated some time tomorrow to do a review of this :)

[tzilist]

@chris-t-jansen most of this seems to be working thankfully! Unfortunately, when i run just generate-email podman or just generate-website podman, I am running into this bug

podman run -it -e USE_EMAIL_THEME=1 -v /Users/tzilist/projects/this-week-in-rust/publishing/output-email-format:/usr/twir/output twir:latest
[03:40:50] CRITICAL FileNotFoundError: [Errno 2] No such file or directory: '/usr/twir/output/atom.xml'                       __init__.py:550
error: Recipe `generate-email` failed on line 94 with exit code 1

Would you mind taking a look when you have a sec? Thank you for all of your work on this so far, this is quite nice!

[chris-t-jansen]

@tzilist That's interesting! And unfortunate, given that those are kind of the core functions of the justfile. Sadly, I'm not seeing that on either of my machines, which makes it tough to troubleshoot. It looks like it's Pelican that's throwing the error, as you can see the line it's referencing here, but the question is what changed with the justfile that caused it. Would you mind trying a couple things to help me narrow it down?

1. Do just generate-email podman and just generate-website podman throw the same CRITICAL FileNotFoundError: [Errno 2] error, and do they throw it every time consistently?

2. Will you try commenting out the feed-generating settings in pelicanconf.py and see if it still happens?

DEFAULT_LANG = u'en'

'''
FEED_DOMAIN = SITEURL
FEED_ALL_ATOM = 'atom.xml'
FEED_ALL_RSS = 'rss.xml'
FEED_MAX_ITEMS = 4
CATEGORY_FEED_ATOM = 'categories/{slug}/atom.xml'
CATEGORY_FEED_RSS = 'categories/{slug}/rss.xml'
RSS_FEED_SUMMARY_ONLY = False
'''

DEFAULT_PAGINATION = 10

3. Will you separately try modifying the Dockerfile's CMD instruction with --verbose and sharing the output of the just commands? Should look something like this:

CMD ["pelican", "--delete-output-directory", "--verbose", "content"]

Apologies for the laundry-list of requests! Hopefully it sheds some light on what's going on, because after that my next best idea is to try bumping the Pelican version to see if it's something that got fixed later, but that's more of a hopeful solution than a logical one.