Add entry_point plugin support for user defined HELM run_specs

[dmjoy]

Temporary workaround until this HELM PR lands: stanford-crfm/helm#3916

May consider using similar approach for other MAGNET/HELM plugins.

[dmjoy]

@Erotemic I guess we're stuck with failing tests until the boolq scenario is fixed upstream?

[Erotemic]

We could:

1. disable that one test
2. switch to a different dataset.
3. use a patched fork of HELM on the CI

[dmjoy]

Closing as the upstream PR has landed.

[dmjoy]

Re-opening as the entrypoint updates here to plugin our custom run-specs/scenarios are still relevant. Removing the magnet_helm_run.py wrapper script as it's no longer needed.

[dmjoy]

@Erotemic seems like CI doesn't like the URL dependency for helm, but I did confirm that our test pass locally. Do you think it makes more sense to pin to a particular commit (vs. the kitware-main branch)? If we're going to make somewhat frequent / potentially breaking changes there maybe pinning to a commit is a good idea?

[Erotemic]

I think pinning to a branch makes sense. Frequent changes can be pushed to a branch other than kitware-main, which I want to be: commits that we think probably should go upstream if it was still maintained.

It looks like git urls are not really supported in the project.dependencies section of pyproject.toml as they break pypi deployment. That means making a pypi package like kitware-helm might be the only way to have a truly robust install mechanism where the user doesn't have to think about it.

https://docs.astral.sh/uv/pip/compatibility/?utm_source=chatgpt.com#transitive-url-dependencies

To make CI pass so we can continue to develop I recommend making a requirements/runtime.txt adding the git url line to that and adding a pip install -r requirements/runtime.txt before we pip install the magnet package. Then make the dependency plain without the git url.

This unblocks us while we think of a better way to handle this in general. (I think it will converge on a new kitware-helm package)

[dmjoy]

I think I roughly follow you and the link is helpful, I've made some tweaks to that end (again mostly to get the CI passing). I do have reservations about maintaining dependencies it two distinct places (seems like a footgun). I'm with you that we'll likely converge on needing a PyPI kitware-helm package)

[dmjoy]

@Erotemic is this the place (and only place) I needed to add this?

[Erotemic]

You also need it in the test_purepy_wheels section after python -m uv pip install --prerelease=allow "aiq-magnet[$INSTALL_EXTRAS]==$MOD_VERSION" -f wheelhouse. There are tow paths for testing the source dist and the wheel dist.

[dmjoy]

Alright so maybe the requirements/runtime.txt bit is working now with CI, but running into a different error with the tests (in that it looks like the latest symlink isn't being generated for the demodata so the _coerce_from_patterned_paths piece isn't passing). Running the tests locally it was passing, but then I cleared out my local cached demodata directory, and tried re-running the tests. It doesn't seem like any demo data is being downloaded or something, because a whole hosts of tests failed after clearing that local cache. My assumption was the demo data would get downloaded as a part of the tests. Am I missing something @Erotemic ?

[Erotemic]

You should be correct. I'm taking a look. Tests that rely on network items will always have some sort of issue like this. I've got the issue reproduced locally.

[Erotemic]

Ah, I see the issue. Upstream HELM removed the "latest" symlink, which honestly is probably a good design decision. See: 130e41ae10c305fa83df6d3158e3153b188b74c8

That does invalidate some tests, but it just means we need to update the "num_expect" in that string. I'll make the change and push it up.

[Erotemic]

@dmjoy The tests are passing. Note that the original run had failures to resolve the IPFS url, but rerunning the tests passed. This is part of the tradeoff when using a distributed system, it can be a bit slow to warm up. What probably happened is we hit the IPFS gateway with the URL, it didn't have the data cached, so it started searching for the content, but timed and returned an error. But it kept trying to find that content behind the scenes so when we tried again, it found it and everything worked.

I was hoping it would be a bit more seamless, but it looks like there are still some rough edges.

[dmjoy]

Roger that, thanks for pushing up fixes. Any objections to merging this as-is now?

[Erotemic]

Let's merge.

We might need to add in a longer timeout, but let's see what it looks like for other PRs.