Re-think caching install scripts

[Submandarine]

It happens that multiple cm scripts are run in succession which need different versions of packages e.g. numpy. With the current way the cache works this can cause issues when a script fails and in the next run the execution e.g. of the installation of a specific numpy version is cached, but the installed version is different because of an install later in the script. In other words, cache entries should not try to guarantee states outside of the cache.

[arjunsuresh]

Hi @Submandarine I'm not able to fully understand the problem here. The state of a cached entry only describes the state at which it got created right?

[arjunsuresh]

Got it. So, is your concern only applicable for packages installed via pip?

We currently do have a problem there and thinking of a good solution. The problem should be resolved if the cache entry is invalidated when the installed version is somehow removed right? Normally we use DEPENDENT_CACHED_PATH for verifying if a path exists and if not invalidate that cache entry. I'm thinking if we should do a "detect-version" for pip installed packages and if the version is not matching, invalidate it.

[Submandarine]

I don't know if something like this can also happen to generic-sys-utils packages, i find it quite hard to get familiar with the structure and internal workings of the scripts.
Checking the version would probably solve the problem, however i don't get why get-generic-python-lib has a cache at all. Wouldn't it be enough if the script checks the current version and if it is missing or incompatible installs the correct one?

[arjunsuresh]

currently the purpose of caching the pip packages is to speed up the run by avoiding repeated package detection. But if we add a version detection to the cached entries, then this advantage will be gone and we surely can consider no-caching.

But in get-generic-python-lib we also have non-standard pip packages like "torch_rocm" for example where we also install all the dependencies for it - if we remove the caching - these dependencies will also get executed (they might be using cache) every time we call the package.

this can also happen to generic-sys-utils packages - it is possible if a user uninstalls a package via package manager. So, we probably need to add a detection command for that also. Removing the cache will cause package manager to be called every time - is fine if version detection works without issues.

[Submandarine]

For python packages detection should work very consistently and also be very fast. I think this is the more urgent thing to fix since it can break without the user messing with packages. For the sys-utils i think there is already a version detection but not every package has a detection command yet. I could start adding those. Which branch should i work on in general? It seems like dev is pretty far behind main?

[arjunsuresh]

@Submandarine This is an implementation for validation support for cache entries. The idea is that we put a "validate_cache.sh" file in the script folder and while processing the cache - we run this file and we ignore the cache entry if there is a failure. This is currently a draft implementation but the idea is simple.

It is safe to work on a fork of the "mlperf-inference" branch and give PRs to it as we have tests running on that branch.

Even though we invalidate the cache like this, I think the proper way to support a CM workflow where a part of it needs a different python env will be to use a different python venv for that CM script dependency. This issue is discussed here.