Smoke test the FTL upgrade path

[safeer]

On merge into main (or part of the manual release process?), we should have a GitHub action that will:

1. create a kube cluster
2. clone/download and run the previous latest release of FTL
3. deploy the exemplar
4. run the exemplar's smoke test (?)
5. upgrade FTL to the latest main
6. run the exemplar's smoke test
7. re-deploy the latest exemplar
8. re-run the smoke tests

If this is run on merges to main, we'll have to figure out a way to be notified of failures. If it's run as part of the manual release process, failures should block the release.

[stuartwdouglas]

You can probably do something like this using a similar approach to the existing kube test:

ftl/backend/controller/scaling/kube_scaling_integration_test.go

Line 21 in a416149

func TestKubeScaling(t *testing.T) {

There is nothing in there about upgrades but I don't think it would be that much work. It's not quite the same as running on a 'real' cluster but is a lot lighter weight in terms of providing a smoke test.

[safeer]

Smoke testing the upgrade path from the last FTL release is a bit of a chicken and egg, especially for the first upgrade test, but also because the smoke tests themselves can be updated. We also can't just grab the latest smoke test, since it may refer to newly added FTL features / test cases in the exemplar. My current thinking is to:

1. clone the FTL repo at the last tagged release
2. run that release's smoke tests on that release's exemplar (what if it doesn't have it?)
3. clone the FTL repo at the current HEAD
4. upgrade FTL without redeploying the exemplar modules (is this possible?)
5. run the previous smoke tests on that release's exemplar

For (2), we skip since we can't test upgrade and the integration test will cover the non-upgrade path.
For (4), seems like the GHA will need to deploy the modules separately from running the smoke test on them, so that it can upgrade FTL and skip the deploy part.
For (5), we need to check out the latest HEAD but the latest tagged release's smoketest directory.

[safeer]

The above tests that a new release is still compatible with old modules, but it doesn't test the upgrade path the way FTL client projects would upgrade. We'll need to use the long-living FTL cluster for that.