Development & Release Process

[urwrstkn8mare]

We should discuss on how we develop the plugins and release them (incl. versioning). I gave an idea in: GOG-Nebula/galaxy-integration-uplay#2 (comment), but again I don't even know if its a good idea. Just looking to hear your guys' ideas. And hopefully land on something we can stick to.

[urwrstkn8mare]

@AndrewDWhite @UncleGoogle Don't mean to disturb you just in case you weren't aware of the organisation discussions tab.

[UncleGoogle]

Oh nice, I didn't know about Discussions 👍

Let me continue the thread.

Yeah we could use tasks.py to create a build script for the repo using buildtools and then the Github action could use that to create win and mac packages.

👍

Maybe we should have a dev branch. I'm not quite sure about automating a version bump (not sure how to implement it and may cause issues). We could implement a system where we have a dev branch where we merge our PRs to. Each PR would include a pre-release version bump (if we follow semver.org it would look something like 0.55.6-alpha.2 or 0.55.6-beta or 0.55.6-rc.3. See semver/semver#114 (comment) for the meanings). After we think the rc release is ready we can create a PR to merge dev into master bumping the version in dev from 0.55.6-rc.3 to 0.55.6 and master from 0.55.5 to 0.55.6. This would become tedious if we were working together on a feature or fix and we could setup org rules that allow us to create branches on the repo which we have write access to (e.g. feature/achievements) and then we would create a PR to merge that into dev. However, this isn't automated and is possibly too tedious or complicated practically but just putting the idea out there.

👎 seems over complicated. We don't have enough traffic to have it profitable and we don't have manpower to maintain/manage this in case of bigger traffic.
The fewer branches the fewer problems with synchronization. Git flow is passe. The most recent trends and practice for continues delivery approach finds out simple "truncate" master a good approach.

We can mark tested releases with tags, and use current_version.json as a public build (at least for plugins shown in Galaxy) - we should treat it as tested and ready to use build and master as a beta branch for devs/testers/enthusiassts (with optional automated pre-releases on each master commit)

(I've tried to reuse this mechanism in my plugin to replease links from current_version.json to on own "update channel" but it doesn't work - probably they don't allow for a different sources than stated in cfg repo.)

If I understand how versioning in plugins works we have constraints we should guard from violation:

1. version in manifest.json matches version passed to Plugin class
2. link in current_version.json should point to ready-to-use build (zip attached to a release) that Galaxy can download and unpack to Galaxy plugins directory
3. version from manifest.json of the build linked in current_version.json should match version given in current_version.json

The first can be done by always reading manifest.json inside python code like that: https://github.com/UncleGoogle/galaxy-integration-humblebundle/blob/master/src/plugin.py#L44

2 and 3 are good candidates to be put in tasks.py and automated by github workflows.

Reading the history of FoG plugins they did:

• commit(s) with changes to master
• release published with zip files for Windows and Mac (in github API those can be 2 separate steps: release and attaching files)
• update current_version.json in separate commit (probably to ensure that release already exists, transaction like)

As for automatic version bump for master by @AndrewDWhite -- it is good. 0.55.6.123133 is easier to than than 0.55.6.-rc.3 which has to be done manually.

In humble plugin I follow yet another approach - create pre-release only on tags. Then do real release manually.

From the "GOG constrains" perspective we can trigger workflows job on mnifest.json change. Then:

prev_version = parse_version(json.load(fetch_file_from_master("src/manifest.json"))
version = parse_version(json.load("src/manifest.json"))
if version > prev_version:
     if version.without_build_part() > version.without_build_part():
          release()
     else:
          pre_release()
else:
    loggging.info("no version change in manifest.json. Skipping automatic release")

k
But I would treat it as an improvement, after we have working semi-manual release flow

What do you think?

[urwrstkn8mare]

We can mark tested releases with tags, and use current_version.json as a public build (at least for plugins shown in Galaxy) - we should treat it as tested and ready to use build and master as a beta branch for devs/testers/enthusiassts (with optional automated pre-releases on each master commit)

Yeah this is a better idea. More simple as well.

As for automatic version bump for master by @AndrewDWhite -- it is good. 0.55.6.123133 is easier to than than 0.55.6.-rc.3 which has to be done manually.

Agreed

In humble plugin I follow yet another approach - create pre-release only on tags. Then do real release manually.

This also seems like a good idea. But I'm happy with either one. For now we could probably stay with @AndrewDWhite's method as it works well.

[urwrstkn8mare]

@UncleGoogle in regard to ur comments in GOG-Nebula/galaxy-integration-steam#1 (comment)

I think we could employ a combination of both actually. we could use a combination of both. once we can setup automatic release builds we can move the current_version.json to the plugin repo and just delete it from the cfg repo. this won't have any apparent affect as you would just change the link to the current_version.json in cfg

in regards to the next comment:

Can someone confirm what it is enough to have "write" permission to publish /update release by hand?

does seem so: docs.github.com/en/rest/releases/assets#update-a-release-asset.

there is github API to update assets in a given release 🗡️

yikes almost every sort of 'package manager' validates downloads with a hash so i completely forgot this. this could be temporarily mitigated by just storing the zips in a repo (cfg or otherwise) you can get a download link with a hash in the link that will only point to that file in the state it was at that commit.

[UncleGoogle]

I think we could employ a combination of both actually. we could use a combination of both. once we can setup automatic release builds we can move the current_version.json to the plugin repo and just delete it from the cfg repo. this won't have any apparent affect as you would just change the link to the current_version.json in cfg

yes, we should only make sure that link in the manifest.json is not used, otherwise weird things can happen on GOG side if there will be 2 different links. Maybe I'll just sit on release action today...

by just storing the zips in a repo (cfg or otherwise) you can get a download link with a hash in the link that will only point to that file in the state it was at that commit.

yes, great solution. I was also thinking about adding hashes to this file for GOG to use, seems to be clean solution (and standard one as you pointed out) but can't find any advantage on storing latest releases in the repo or cfg. And it requires some work on GOG side (not much and they should be interested but still)

[urwrstkn8mare]

yes, we should only make sure that link in the manifest.json is not used, otherwise weird things can happen on GOG side if there will be 2 different links. Maybe I'll just sit on release action today...

👍

yes, great solution. I was also thinking about adding hashes to this file for GOG to use, seems to be clean solution (and standard one as you pointed out) but can't find any advantage on storing latest releases in the repo or cfg. And it requires some work on GOG side (not much and they should be interested but still)

yes if GOG checks a hash given in a dedicated key in current_version.json then thats perfect 👌 . but incase they don't we can preserve security by version controlling the .zip release file so it can't be changed after the link to the file is put in current_version.json. Just putting the hash in the filename doesn't actually prevent the file from being different from the hash. Someone could hypothetically just edit the release with a new file with the same hash in its filename.

i think temporarily while we try to get support/cooperation from GOG and push out releases we could do that just temporarily while we get better procedures and structures in place.

[UncleGoogle]

i think temporarily while we try to get support/cooperation from GOG and push out releases we could do that just temporarily while we get better procedures and structures in place.

right.

Just putting the hash in the filename doesn't actually prevent the file from being different from the hash. Someone could hypothetically just edit the release with a new file with the same hash in its filename.

it should be hash of the all zip binary content, not filenames