Use D: for Windows installations on Azure

[jaimergp]

Checklist

• Added a news entry
• Regenerated schema JSON if schema altered (python conda_smithy/schema.py)

The D: drive is supposed to be faster, so let's see if that's true!

[jaimergp]

Yes, this cuts the Miniforge installation time in half. Just installing Miniforge (not counting the mamba updates we do on top later) takes 1m10s - 1m30s (seeing times for this run of the pyside2 feedstock). Changing C: to D: means we can do it in 30s.

The install times for base are:

For C, 1m40s:

2024-10-03T14:17:59.8193862Z Installing dependencies
2024-10-03T14:19:40.0923480Z Setting up configuration

For D, 54s:

2024-10-03T15:02:08.0620513Z Installing dependencies
2024-10-03T15:03:02.2620718Z Setting up configuration

So, in total:

• C takes: 1m17s (Miniforge install) + 1m40s: 2m57s
• D takes: 37s (Miniforge install) + 54s (deps): 1m31s

[wolfv]

Can we also run builds and all on D:/? E.g. set CONDA_BLD=D:/bld?

[jaimergp]

That's already the case :D See TEST END: D:\bld\win-64\nsis-3.10-h9ead6bc_log_0.conda from https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=1045477&view=logs&j=b0f7521e-dadf-5339-5101-26345e949aac&t=8017811c-a576-520e-3ed1-402a7f560f6d.

[isuruf]

For context, we used C: for Miniforge so that pkgs_dirs ends up in C: due to C: having more space and D: for conda-build PATH because D: is faster. We also have a switch to build in C: for feedstocks that run out of space in D:.

Not sure if switching everything to D: would make builds run out of space for more feedstocks.

[jaimergp]

Hm, I feared as much. Maybe this should be configurable. I doubt most feedstocks are running out of space, but for sure all of them are installing the same base environment. We can shave a few minutes of those with D: + micromamba.

As long as this is documented next to the "free_disk_space" options I think we should be fine, wdyt? If that's agreeable, then I'll put some settings.

[jaimergp]

I'm also thinking, having the cache in C: and BLD in D:... wouldn't that cause copies across drives? I don't think Windows can hardlink across drives 🤔 That would also cause slowdowns and diminish the benefits of saving space. Maybe it makes sense in some extreme cases with multi-outputs only, not sure.

[isuruf]

Yeah, that makes sense. The only issue here would be that feedstocks that customized CONDA_BLD_PATH to C: now has to customize MINIFORGE_HOME too. Maybe we do that in a migrator first?

[jaimergp]

There are only 12 of them, according to the search. Could be faster to just send them manually, and I'm happy to do so :D

[isuruf]

Thanks. Note that those feedstocks are the heaviest loads, so keep an eye on the CI if you can.

[jaimergp]

If only Azure respected the [skip ci] stuff :( We can cancel the runs on Azure, but I don't have admin permissions there, so if you see it's a problem, please feel free to cancel the heaviest ones as you see fit! Thanks and sorry!

[beckermr]

We should add short circuit skips based on the commit message to the azure templates.

[jaimergp]

See #2077

[jaimergp]

There are only 12 of them, according to the search.

There we go:

• MNT: set MINIFORGE_HOME: C:\Miniforge descarteslabs-feedstock#73
• MNT: set MINIFORGE_HOME: C:\Miniforge scipy-feedstock#286
• MNT: set MINIFORGE_HOME: C:\Miniforge intel-compiler-repack-feedstock#50
• MNT: set MINIFORGE_HOME: C:\Miniforge google-cloud-cpp-feedstock#186
• MNT: set MINIFORGE_HOME: C:\Miniforge flang-feedstock#75
• MNT: set MINIFORGE_HOME: C:\Miniforge deno-feedstock#137
• MNT: set MINIFORGE_HOME: C:\Miniforge occt-feedstock#117
• MNT: set MINIFORGE_HOME: C:\Miniforge onnxruntime-feedstock#133
• MNT: set MINIFORGE_HOME: C:\Miniforge qt-main-feedstock#297
• MNT: set MINIFORGE_HOME: C:\Miniforge lancedb-feedstock#21
• MNT: set MINIFORGE_HOME: C:\Miniforge qt-webengine-feedstock#66
• MNT: set MINIFORGE_HOME: C:\Miniforge qgis-feedstock#465

[jakirkham]

FWIW in PR ( #1966 ) has switched to using PowerShell to download Miniforge, which helped a little bit. Though the main push of that PR was to free more space on the image (as space is a constant challenge). Also please see issue ( #1949 ) about space limitations

[jaimergp]

FWIW in PR ( #1966 ) has switched to using PowerShell to download Miniforge, which helped a little bit.

I think PowerShell has a non-negligible start time, right? We can compare against the certutil hack though. The important bit is to have it in the bat script though; the download tool itself is not as important.

[jakirkham]

I don't think so. The end-to-end time of the PowerShell download was pretty fast

In any event we are currently calling Python to do that download, which has its own startup time

Also not sure what kind of security measures are applied on Azure. Depending on what they are, these can add overhead to startup, runtime, downloads, etc.

If we are especially concerned with startup time, we could use Batch for downloading. The syntax is a little unpleasant. Though hopefully it is write once

If the goal here is just generally to improve CI startup time, it might be worth revisiting Windows Docker images ( conda-forge/docker-images#209 ). With that we could do all of the download, configuration, etc. in the image. Then CI need only download the image and perhaps update before building. If Azure has a particular registry it likes, we could also push to that registry for faster downloading (and maybe opt-in to earlier scanning to avoid this cost in CI jobs)

[jaimergp]

I don't think so. The end-to-end time of the PowerShell download was pretty fast

In any event we are currently calling Python to do that download, which has its own startup time

Also not sure what kind of security measures are applied on Azure. Depending on what they are, these can add overhead to startup, runtime, downloads, etc.

If we are especially concerned with startup time, we could use Batch for downloading. The syntax is a little unpleasant. Though hopefully it is write once

Yea, using certutil is not that bad: https://github.com/conda-forge/conda-smithy/pull/2075/files#diff-69e88e5a41c617697582fa7d0b6a32d712995f682e7cac24b517d4b8cc4ff9b8R28

[jakirkham]

Yea, using certutil is not that bad: https://github.com/conda-forge/conda-smithy/pull/2075/files#diff-69e88e5a41c617697582fa7d0b6a32d712995f682e7cac24b517d4b8cc4ff9b8R28

That looks nice! I would be happy with that 😄