feat(hex): use registry v2 for packages rather than hex.pm http API #29756

bryannaegele · 2024-06-19T00:42:15Z

Changes

use hex registry v2 for package information
use hex.pm package API for supplemental package metadata when package is hosted there.
support registryAliases for private repo/registry mapping
run mix.repo add in mix manager for private repos

Context

This changes the hex datasource to use hex registry v2 for package management. Hex registries are not required to implement an http API like hex.pm has, so the prior implementation prevented updating of packages in private repos such as oban pro. Additionally, due to the way mix works when updating dependencies, mix.lock files could not be updated when such a dependency existed and the mix manager ran mix deps.update <<updated packages>>.

In order to keep as much parity with the prior implementation, hex.pm is used for supplemental package metadata where possible.

Documentation (please check one with an [x])

I have updated the documentation, or
No documentation update is required

How I've tested my work (please select one)

I have verified these changes via:

Code inspection only, or
Newly added/modified unit tests, or
No unit tests but ran on a real repository, or
Both unit tests + ran on a real repository

bryannaegele · 2024-06-19T02:29:20Z

Leaving this as a draft until there's an answer for this #29622 (comment)

I am no TS expert, so please feel free to jump in and fix up whatever can be done better.

viceice

reduce fixtures, we don't need all versions for testing, ~3 are enough. ensure the tgz file is at minimum too

lib/modules/datasource/hex/package.spec.ts

lib/modules/manager/mix/artifacts.ts

lib/modules/datasource/hex/signed.ts

bryannaegele · 2024-06-20T03:20:12Z

reduce fixtures, we don't need all versions for testing, ~3 are enough. ensure the tgz file is at minimum too

Is this strictly necessary? The tgz has the registry version of the package wrapped in a signed wrapper. I don't have any way to create a version with less releases in it that would have a valid signature.

I can try to find a different package as an example but this is one of the rare ones with retirements.

bryannaegele · 2024-06-20T04:10:40Z

Mix changes split to #29775

bryannaegele · 2024-07-26T19:57:53Z

Alright. Made the following requested changes:

remove codegen from CI and setup
commit generated files
add readme with info on how to generate the proto files
renamed script name in package per @zharinov's suggestion

I have kept the generation script in tools so that if anyone should need to regenerate the files, it's as simple as possible.

esambo · 2024-08-13T17:53:59Z

@bryannaegele FYI, you have merge conflicts

bryannaegele · 2024-08-13T18:00:28Z

@bryannaegele FYI, you have merge conflicts

Are y'all ready to merge this and the other PR? I'd like to not keep resolving the conflicts unless it's going to get merged.

# Conflicts: # package.json # pnpm-lock.yaml

lib/modules/datasource/hex/index.ts

viceice · 2024-08-26T10:53:55Z

lib/modules/datasource/hex/index.ts

-  override readonly releaseTimestampSupport = true;
-  override readonly releaseTimestampNote =
-    'The release timestamp is determined the `inserted_at` field in the results.';
+  override readonly releaseTimestampSupport = false;


that's a bad step back. isn't it possible to still support release time stamps?

this is really a breaking change and a new show stopper

Timestamps are not a piece of metadata in the registry metadata. They are only a piece of metadata that hex.pm supports via its HTTP API which is not a requirement to run a registry.

Available fields.

inserted_at field is preserved but that field is only available for a package served from hexpm.

Can you explain why it's a showstopper? I understand it's a potentially breaking change but the original implementation was technically incorrect. I'm not sure how the releaseTimestampSupport is used. If we set it to true and the value isn't available for some packages, what would happen?

lib/modules/datasource/hex/index.ts

viceice · 2024-08-26T11:07:07Z

tools/generate-hex-protos.ts

+function generateProto(protos_path: string, file: string): Promise<string> {
+  return new Promise((resolve, reject) => {
+    console.log(`Current directory: ${process.cwd()}`);
+    fs.readdirSync(process.cwd()).forEach((file) => {


use fs/promises api

Co-authored-by: Michael Kriese <[email protected]>

bryannaegele · 2024-09-04T17:15:32Z

Handing to Maintainers to Finish

I leave this PR and the Mix PR #29775 to the maintainers to complete. My limited time is better spent on my maintainer duties elsewhere.

These PRs are required to Elixir/Hex support for companies to adopt Renovate in non-trivial setups. First-order support for the hex registry is necessary to support the Hex package manager. This was unfortunately missed during original implementation in the good faith effort to support Hex/Mix.

Reasons for the handoff:

compatibility concerns and solutions are best left to maintainers
maintainer expertise on testing techniques employed in the project will provide better coverage
strong opinions on code style, organization, and nits are best resolved by codeowners

Three months and >100 hours greatly surpasses the level of effort I thought this would take when I volunteered to investigate and take a stab.

Renovate is a great project. I hope you can find resources to complete the work @rarkins so we can consider using it for Elixir projects again someday.

Thanks

#29622

rarkins · 2024-09-04T18:17:42Z

@bryannaegele thanks for your efforts. I've closed this if you're unable to continue and hope someone can pick it up in future.

bryannaegele · 2024-09-04T18:33:06Z

@viceice should have enough to wrap up. I'd hate to see all this work go to waste.

zharinov · 2024-09-04T20:20:31Z

@bryannaegele Thanks for your effort, it definitely isn't easy one 🤝

I'll take a look if I could reuse it as series of smaller changes.

bryannaegele marked this pull request as draft June 19, 2024 00:42

bryannaegele changed the title ~~Use hex registries for packages rather than hex.pm http API~~ Use hex registry v2 for packages rather than hex.pm http API Jun 19, 2024

bryannaegele changed the title ~~Use hex registry v2 for packages rather than hex.pm http API~~ feat/Use hex registry v2 for packages rather than hex.pm http API Jun 19, 2024

bryannaegele changed the title ~~feat/Use hex registry v2 for packages rather than hex.pm http API~~ feat(hex) use registry v2 for packages rather than hex.pm http API Jun 19, 2024

bryannaegele changed the title ~~feat(hex) use registry v2 for packages rather than hex.pm http API~~ feat(hex): use registry v2 for packages rather than hex.pm http API Jun 19, 2024

viceice reviewed Jun 19, 2024

View reviewed changes

bryannaegele mentioned this pull request Jun 20, 2024

feat(mix): Support private hex repos/registries #29775

Open

6 tasks

bryannaegele marked this pull request as ready for review June 24, 2024 17:22

bryannaegele changed the base branch from main to v38 July 8, 2024 19:43

bryannaegele changed the base branch from v38 to main July 8, 2024 19:43

bryannaegele force-pushed the feat/29622-hex-registry-support branch from 6f0e4ee to 97ceb1d Compare July 8, 2024 19:55

bryannaegele and others added 16 commits July 8, 2024 14:27

Use hex registries for packages rather than hex.pm http API

655b0df

Use hex registries for packages rather than hex.pm http API

cb17695

linter fixes

7ac3521

Update pnpm-lock.yaml

d5199ab

Update pnpm-lock.yaml

12e4b75

Update pnpm-lock.yaml

a56add3

Update pnpm-lock.yaml

b587b32

Update pnpm-lock.yaml

f0f8062

Update pnpm-lock.yaml

eca0292

run prettier on generated files

564591f

PR feedback

bd64da4

Split mix manager changes to separate PR

5b7512f

generate ts protos in build

33bfdc2

ignore generated files

ab217a4

updated lock file

527f322

Fix up generate script

13ea8ed

bryannaegele force-pushed the feat/29622-hex-registry-support branch from 453dc5a to 2e0ea0a Compare July 26, 2024 16:16

rarkins changed the title ~~feat(hex)!: use registry v2 for packages rather than hex.pm http API~~ feat(hex): use registry v2 for packages rather than hex.pm http API Jul 26, 2024

bryannaegele and others added 5 commits July 26, 2024 13:23

Remove CI changes, commit files, add gen docs

f36e415

Update codecov ignore paths

4dd2ba4

Merge branch 'main' into feat/29622-hex-registry-support

9909320

Lint fix

ca9b584

set codecov skip in generated files

310a8a8

rarkins requested a review from viceice August 24, 2024 06:19

Merge branch 'main' into feat/29622-hex-registry-support

b152481

# Conflicts: # package.json # pnpm-lock.yaml

viceice requested changes Aug 26, 2024

View reviewed changes

bryannaegele and others added 11 commits September 3, 2024 16:51

Update lib/modules/datasource/hex/index.ts

50ee499

Co-authored-by: Michael Kriese <[email protected]>

Update lib/modules/datasource/hex/index.ts

480817c

Co-authored-by: Michael Kriese <[email protected]>

make base url static

06ab7c4

make hexRepoName a local var

13cb6c5

replace maps with for of loops

ca51f26

move decompressBuffer to utils

2784b34

increase test coverage

1f04b07

Merge branch 'main' into feat/29622-hex-registry-support

e8047e7

fix broken lock file

c5e53a2

eslint errors

3197b93

more eslint

3f40c6f

rarkins closed this Sep 4, 2024

zharinov self-assigned this Sep 4, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(hex): use registry v2 for packages rather than hex.pm http API #29756

feat(hex): use registry v2 for packages rather than hex.pm http API #29756

bryannaegele commented Jun 19, 2024

bryannaegele commented Jun 19, 2024

viceice left a comment

bryannaegele commented Jun 20, 2024

bryannaegele commented Jun 20, 2024

bryannaegele commented Jul 26, 2024

esambo commented Aug 13, 2024

bryannaegele commented Aug 13, 2024

viceice Aug 26, 2024

bryannaegele Sep 3, 2024

viceice Aug 26, 2024

bryannaegele commented Sep 4, 2024

rarkins commented Sep 4, 2024

bryannaegele commented Sep 4, 2024

zharinov commented Sep 4, 2024

feat(hex): use registry v2 for packages rather than hex.pm http API #29756

feat(hex): use registry v2 for packages rather than hex.pm http API #29756

Conversation

bryannaegele commented Jun 19, 2024

Changes

Context

Documentation (please check one with an [x])

How I've tested my work (please select one)

bryannaegele commented Jun 19, 2024

viceice left a comment

Choose a reason for hiding this comment

bryannaegele commented Jun 20, 2024

bryannaegele commented Jun 20, 2024

bryannaegele commented Jul 26, 2024

esambo commented Aug 13, 2024

bryannaegele commented Aug 13, 2024

viceice Aug 26, 2024

Choose a reason for hiding this comment

bryannaegele Sep 3, 2024

Choose a reason for hiding this comment

viceice Aug 26, 2024

Choose a reason for hiding this comment

bryannaegele commented Sep 4, 2024

rarkins commented Sep 4, 2024

bryannaegele commented Sep 4, 2024

zharinov commented Sep 4, 2024