Publish channel.json

[braver]

The new crawler's channel.json should be communicated on the site to make it easy for everyone to switch over.

@deathaxe suggested to serve the channel.json via the site as well on https://packages.sublimetext.io/channel.json. I like that because it's a short URL, and it decouples where the channel is generated in a way that's easy to maintain over time.

In my mind there are 2 things we can do:

1. curl the channel.json during build time just like we do with the workspace.json, and build the site more often (e.g. hourly instead of nightly)
2. put the site in a gh-pages branch on the crawler repo, and build it every time the crawler completes

I think both could work and there is something to be said for either approach. I think I kind of prefer having the site in a separate repo and would lean towards option 1, even though that does mean updates from the crawler reach users in a slightly delayed manner.
@kaste perhaps you have an opinion that could sway us one way or the other?

[kaste]

I thought the site generation lives also besides the crawler. Basically, it is a view of the crawler's database. Just like the channel. Primarily, however, because it is easier to trigger new builds. The crawler knows when actually something changed, so we have that trigger. It only generates a new channel if something changed, and only then it also needs to build the page new.

Currently, the the crawler generates two channels, one for ST4 and one for ST3, the libraries are included. So PC, which should get an update anyway and split up (aka leave behind regarding new features) ST4 and ST3, ships either of these channels.

Basically, I just don't know any good signaling, events passing between repos. I would go for two or more orphaned branches (for now) on one repo, e.g. thecrawl. (But these repos can be renamed freely over time too. Maybe "packagecontrol/thepackages", "packagecontrol/theregistry".)

With git worktrees this works very easy and convenient. You have branches like "thecrawl", "thesite", "thechannel". And you can flip via "default_branch" which one is shown at GitHub's homepage. In the README we link to the other branches. That's a bit novel and I think that can work out nicely, esp. because I don't see thousands of commits except for "thechannel" which gets these constant updates where people register their packages actually. (If we eventually move away from wbond.) It is like a monorepo but with very clear separation of concerns still. Very little friction and inter-group communication needed. (We're in a group and still don't need to see and be informed about what the others do because everyone has their own root.)

[braver]

It's actually a very common pattern to have an orphaned gh-pages branch on a repo with the website code in it that's not otherwise present in the default branch. And depending on how things go we can prioritize whichever branch is visible on the repo homepage.

So I like the idea of moving this site repo into the crawler repo. Another benefit is that all issues for the project are collected in one space and users don't have to think about whether a problem is a website problem or a crawler problem or something else.

Ok, let's just wrap up some ongoing development and move stuff around. @FichteFoll once we do that, do we need a DNS change for that? I haven't set that up for my own projects recently enough to remember how it works 😅

[braver]

Currently, the the crawler generates two channels, one for ST4 and one for ST3, the libraries are included

Just for my understanding, if you're using ST4, would you still be interested in the ST3 channel?

[kaste]

Just for my understanding, if you're using ST4, would you still be interested in the ST3 channel?

No, just one channel. The legacy or the standard ST4 one. Libraries included, so we have one download/channel in the PC settings.

[deathaxe]

Ideally the crawler would just directly resolve asset based releases and pypi libraries the instead of merging in the channel_v4 from packagecontrol/channel repo.

The overall goal would be to

1. merge https://github.com/deathaxe/package_control_channel/tree/four-point-oh into https://github.com/packagecontrol/channel
2. and replace existing package_control based crawler by the logic of "thecrawl"
3. to use https://github.com/packagecontrol/channel of new single source of truth and deploy channel data for PC clients from there or via "packages.sublimetext.io/channel.json" if sanely doable. The suggestion was just about shipping channel via nice URL. Current channel_v4 URL could also be re-used.

[braver]

We’re looking at pulling everything into the thcrawl repo. It already has the channel repository. That makes it super easy to rebuild the site if needed and publish the channel jsons via nice URLs at packages.sublimetext.io.

Until the wbond situation is resolved keeping the new stuff separate and opt-in based makes sense. I kind of hope we eventually gain control over the og domain name and channel repository repo, because that would make for a smooth transition for end users. We have some things to build still before we position this as a whole sale replacement, so that gives us some time to see how that pans out.

What that means for libraries and their repo is currently beyond my understanding.

[kaste]

There is code for crawling dependencies in the repo actually. I just deleted that as it was unused. Everything is in the lib crawler already. The lib crawler is a bit different than the packages crawler. (1) the lib registry is tiny and different from the main registry, (2) the update cadence is slow, it just runs nightly in one go, (3) the features are different, the metadata isn't important and the main Hub is PyPi. That made it clear that this is not the same function/script/file but a separate one and then a combine/merge both things step is required. (Easy enough, because one crawler provides the packages_cache and the other one the libraries_cache. This confirms the observed separation actually.)

For Pypi, I then had the other, IMO more user-friendly, idea to parse Pypi data differently. (Probably a hybrid approach is best but I proposed only the client code sublimehq/package_control#1705.) I don't know the importance of gate-keeping here. IMO we should be open and nice. Anyhow, the PR allows "any" Pypi package to be installed on the client side, but also on the server I think the registry API should allow just

	"details": "https://pypi.org/project/aiohttp/"

and the crawler should figure that out.

tldr; The lib crawler works and does not have or had issues like the main crawler (which works even worse than what people think, in reality the main wbond crawler removes (forgets) and re-adds packages during a day, each day!); plus with the ideas around Pypi, there is quite a chance we see some improvements API wise too which would require a code change of that crawler.

[braver]

Ah yes, thanks. That’s clear and reasonable. Let’s leave libraries out of scope then, since that status quo seems acceptable. I mean, I’m sure work can or needs to be done there, but that’s not this project right now.

[braver]

@kaste shall I set up the orphan "gh-pages" branch on the crawler repo, push the main branch from here into there, and then try to get a workflow set up? Not sure if the subdomain can be easily switched to point to that repo, but we can always fix that later.

[kaste]

Feel free to use a cool name for the branch. What should gh-pages be?

[braver]

“gh-pages”, short for GitHub Pages, is the default recommended name for “the branch that holds the website for this project”, and as such will be recognizable and easy to find for potential contributors. Loads of repos have a branch like that.

[kaste]

Ah, okay. Never called this GitHub pages when it is not on their domain. Some advertisement for them. 😁 (I mean: everything is free, but then the robots come and read how we resolved the permalinks yesterday, gold for them. Unpaid labor.)

[deathaxe]

A gh-pages branch is the classic/legacy way of distributing Github pages.

With state of the art actions the workflow however doesn't differ from publishing releases without cluttering the repo with a branch of build artefacts.

Step 1: Upload the site after building it:

https://github.com/packagecontrol/channel/blob/da8d62e4c7b197c8111797a7a273508e96c1f16f/.github/workflows/deploy.yml#L62-L67

Step 2: Publishing the site:

https://github.com/packagecontrol/channel/blob/da8d62e4c7b197c8111797a7a273508e96c1f16f/.github/workflows/deploy.yml#L99-L101

[braver]

GitHub pages when it is not on their domain. Some advertisement for them.

Well you know, it’s still their servers. And of course we pay by feeding the robots 😉