Add WrapDB to Repology

[stephanlachnit]

It would be super useful to have WrapDB support in Repology. The upstream issue is repology/repology-updater#1138.

The requirements for inclusion to Repology are listed here: https://repology.org/docs/requirements

Essentially, WrapDB is almost there, it supports:

• Name and version in a single request via https://wrapdb.mesonbuild.com/v2/releases.json
• Recipe URL via https://github.com/mesonbuild/wrapdb/releases/tag/<wrap_name>_<wrapdb_version> (would be nice to have a nginx redirect like https://wrapdb.mesonbuild.com/v2/<wrap_name>/<wrapdb_version>/release)

What is missing is support for a homepage. This is essentially discussed in also #292.

My suggestion would be to simply introduce a homepage field to releases.json now to support Repology and postpone displaying this information on the WrapDB website to a different pull request.

[eli-schwartz]

I think that this would effectively constitute making the wrapdb worse for everyone (including the primary use case) just to appease a frankly ridiculous repology rule. The homepage will always come from some linked project anyway, surely?

Adding a great deal of unavoidable, uncompressed text to everyone's queries doesn't seem good to me, and there's no justification of "but that might be what they are looking for". There is no part of the meson wrap command line that would end up using a homepage.

[bgilbert]

The project homepage has some value as a unique identifier for the origin of the upstream project we're shipping. Neither the wrap name nor the source_url really provide this today. When I set up the Anitya integration, I had to do a bunch of manual digging to ensure I was mapping wraps to the correct upstreams.

[stephanlachnit]

I think that this would effectively constitute making the wrapdb worse for everyone

Sorry, I don't see how adding a homepage makes wrapdb worse? This has been requested already since three years (see #292), and you even claimed yourself that this might be useful here.

I also often find my self looking through wrapdb, thinking this project might be interesting, and then going the repo and finding the wrap file to figure the website (which doesn't even always work). Having this on the WrapDB website is absolutely useful, and if I'm not mistaken it uses releases.json.

And Repology is a real improvement for users. I regularly use it to check if a dependency is commonly available. WrapDB is notably missing and I always need to check it separately. Literally all distributions, PyPI, crates.io etc manage to provide this information. I don't see how this that big of a problem for WrapDB.

Adding a great deal of unavoidable, uncompressed text to everyone's queries doesn't seem good to me, and there's no justification of "but that might be what they are looking for".

Sorry, but I think the extra used space is just a bad argument. I listed good arguments unrelated to Repology already above and they are also listed in #292.

A typical homepage might be 40 characters, which would add 56 bytes to the JSON per entry. This would be less than 2kB for the entire JSON file (which is 67kB). This is less than 3%, and it doesn't even scale linearly since adding new releases will reduce this fraction.

If space is really that big of a concern, how about:

• providing/storing a minified version of the JSON
• using a more efficient binary format like msgpack

There is no part of the meson wrap command line that would end up using a homepage.

Hard disagree. meson wrap info is there, and this would be useful information, and again you said so yourself here. This could also be displayed in meson wrap search to quickly allow checking the documentation before installing the subproject.

[eli-schwartz]

Sorry, I don't see how adding a homepage makes wrapdb worse? This has been requested already since three years (see #292), and you even claimed yourself that this might be useful here.

Hard disagree. meson wrap info is there, and this would be useful information, and again you said so yourself here. This could also be displayed in meson wrap search to quickly allow checking the documentation before installing the subproject.

I would like to factually correct you on this. I did not claim it might be useful. I replied to the PR in order to demonstrate the shape of what a well-documented proposal should look like, including questions designed to jog the author into providing answers, where the answers could serve to guide a decision.

Please don't use me as evidence that this is a good idea. All I did was ask whether the author was motivated "for example" by such and such a use case. Because if so, the author should state that directly.

Then, and now, I wasn't convinced that it was useful in such a way. I was simply trying to make the point that if one does want X, it's better to request X than to implement Y and leave people trying to guess how one plans to use Y.

If space is really that big of a concern, how about:

• providing/storing a minified version of the JSON

We... probably should. The original wrapdb was a server-side process that only returned what you ask for, even, but it was seen as an overly complex maintenance burden, which was part of the motivation to move to the current monorepo.

• using a more efficient binary format like msgpack

Whatever we use must be in the stdlib for python 3.7, so that meson can consume it. :(

[eli-schwartz]

Essentially, WrapDB is almost there, it supports:

• Name and version in a single request via https://wrapdb.mesonbuild.com/v2/releases.json

The simplest and most straightforward way to solve this would be akin to how we have ci_config.json for information that CI uses, but meson wrap doesn't use. I know that the repology DFL doesn't like compromise, working with others, or taking responsibility for data accuracy, so if that's a dealbreaker then it disincentivizes me to care more about working with him than he cares about working with me.

[stephanlachnit]

Sorry for the misunderstanding then.

How about:

• we add the homelage filed in the repository as I suggest
• we implement a CI job which publishes a minified version without the homepage and an extended version with the homepage via GitHub pages?

This way, we reduce the space for most requests but provide an extended version for the website and sites like Repology or Anitya?

[bgilbert]

Anitya doesn't retrieve any data from us. I manually populate its database with mappings from its upstream project records to our wrap names.

If space is really that big of a concern, how about:

• providing/storing a minified version of the JSON

We... probably should. The original wrapdb was a server-side process that only returned what you ask for, even, but it was seen as an overly complex maintenance burden, which was part of the motivation to move to the current monorepo.

raw.githubusercontent.com supports Content-Encoding: gzip. It shouldn't be too hard to add support for that in Meson.

Data	Size (bytes)
releases.json	66522
Minified releases.json	40893
releases.json with Content-Encoding: gzip	10368
Minified releases.json with Content-Encoding: gzip	9608 (estimate)

[bgilbert]

PR to add gzip Content-Encoding support: mesonbuild/meson#14552