CI: Plugin Verification tasks randomly failing to download IDEs

[WarningImHack3r]

What happened?

The Run Plugin Verification tasks task (./gradlew runPluginVerifier) in the verify job periodically fails to download (some?) IDE versions.

From what I can tell, it's due to a lack of disk space, despite the step to maximize space.

I support a wide range of IDE versions in my plugins, which may be why I'm more likely to fall into this kind of problem. The thing I fail to explain is why it's so periodical (sometimes no problem for months, then problems again for a week or 2 consistently): probably because the different EAP IDEs may be taking a lot more than usual.

The fear I have is the more IDE versions you guys release, the more I'll likely encounter this issue, meaning it won't get any better soon in my opinion. There will likely be a point when I would never again be able to get my CI to work past some amount of IDE versions.

I have no clue what and where the code for your Gradle tasks is, but my suggestion is the following: run each verification (download + test against this version) in parallel (if it's not already the case), then after each test immediately delete that version. Additionally, try again 1 or 2 times in case of download failure (if it's not caused by disk space) to avoid failing jobs as much as possible.
It seems to be the most scalable solution to me.

Thank you!

Relevant log output or stack trace

Run ./gradlew runPluginVerifier -Dplugin.verifier.home.dir=~/.pluginVerifier
  
Starting a Gradle Daemon (subsequent builds will be faster)
Calculating task graph as no cached configuration is available for tasks: runPluginVerifier
> Task :checkKotlinGradlePluginConfigurationErrors
> Task :initializeIntelliJPlugin
> Task :downloadAndroidStudioProductReleasesXml
> Task :downloadIdeaProductReleasesXml
> Task :patchPluginXml
> Task :verifyPluginConfiguration
> Task :processResources
> Task :listProductsReleases
> Task :compileKotlin
> Task :compileJava NO-SOURCE
> Task :classes
> Task :instrumentCode
> Task :jar
> Task :instrumentedJar
> Task :prepareSandbox
> Task :verifyPlugin
2024-06-22 17:27:01,078 [    295]   WARN - llij.ide.plugins.PluginManager - id redefinition ([row,col,system-id]: [2,3,"product classpath"]) 
> Task :buildSearchableOptions
> Task :compileTestKotlin
> Task :classpathIndexCleanup SKIPPED
> Task :buildSearchableOptions
Jun 22, 2024 5:27:04 PM java.util.prefs.FileSystemPreferences$1 run
INFO: Created user preferences directory.
Jun 22, 2024 5:27:04 PM java.util.prefs.FileSystemPreferences$6 run
WARNING: Prefs file removed in background /home/runner/.java/.userPrefs/prefs.xml
Starting searchable options index builder
2024-06-22 17:27:05,096 [   4313]   WARN - l.NotificationGroupManagerImpl - Notification group CodeWithMe is already registered (group=com.intellij.notification.NotificationGroup@2dc67902). Plugin descriptor: PluginDescriptor(name=Code With Me, id=com.jetbrains.codeWithMe, descriptorPath=plugin.xml, path=~/.gradle/caches/modules-2/files-2.1/com.jetbrains.intellij.idea/ideaIU/2021.3/3597c84ef4f0d240f4f9d9eacba7509e01674c4d/ideaIU-2021.3/plugins/cwm-plugin, version=213.5744.223, package=null, isBundled=true) 
2024-06-22 17:27:22,987 [  22204]   WARN - ellchecker.SpellCheckerManager - Couldn't load dictionary 'svelte.dic' with loader 'class dev.blachut.svelte.lang.spellchecker.SvelteSpellcheckingDictionaryProvider' 
2024-06-22 17:27:24,296 [  23513]   WARN - #com.intellij.ui.jcef.JBCefApp - JCEF is manually disabled in headless env via 'ide.browser.jcef.headless.enabled=false' 
Found 355 configurables
Searchable options index builder completed
> Task :jarSearchableOptions
> Task :buildPlugin
> Task :runPluginVerifier
[gradle-intellij-plugin :runPluginVerifier] Cannot download 'IU-2022.1.4' from 'release' channel: https://cache-redirector.jetbrains.com/download.jetbrains.com/idea/ideaIU-2022.1.4.tar.gz. Run with --debug option to get more log output.

Steps to reproduce

Run the CI in https://github.com/WarningImHack3r/intellij-shadcn-plugin (preferred) or https://github.com/WarningImHack3r/npm-update-dependencies

Gradle IntelliJ Plugin version

1.17.4

Gradle version

8.8

Operating System

None

Link to build, i.e. failing GitHub Action job

https://github.com/WarningImHack3r/intellij-shadcn-plugin/actions/runs/9605368413/job/26492877373