github-action: re-usability of distrib cache

[th0ma7]

Description

Improves the GitHub Actions cache strategy for the distrib folder to reduce redundant downloads and speed up builds.

The current cache configuration uses github.run_id as the cache key, which creates a unique cache entry for every workflow run. This means:

• Cache is never reused between runs
• All source dependencies are re-downloaded every time
• Build times are unnecessarily long

Additionally, GitHub automatically evicts caches that haven't been accessed in 7 days, which can result in a completely empty cache after periods of low activity.

Proposed Solution

1. Updated cache strategy in build.yml

• Use content-based cache keys with a hash of file paths and sizes
• Same content = same hash = no duplicate cache entries
• Uses restore-keys: distrib- fallback to always find the most recent available cache
• Conditional save: only saves a new cache if the content hash doesn't already exist
• Cache keys are stable and reusable across runs, eliminating redundant downloads

2. New distrib-cache-maintenance.yml workflow

A scheduled workflow that runs twice weekly (Monday and Thursday) to:

• Refresh the cache: Prevents GitHub's 7-day eviction policy by regularly accessing the cache
• Clean corrupted files: Removes files marked as .wrong by checksum verification
• Clean obsolete files: Removes source files from distrib/ that haven't been modified in 180 days
• Cleanup stale cache entries: Keeps only the 5 most recent distrib-* cache entries (other caches like toolchain-* are not affected)

3. Updated download.sh

• Added retry logic for checksum failures on cross/native packages
• If a cached file fails checksum verification (renamed to .wrong), the download is automatically retried once
• Python wheels are excluded from retry logic (no checksum verification)
• Prevents build failures due to outdated cached files

Benefits

1. Cumulative cache: Dependencies accumulate over time instead of being discarded
2. Faster builds: Most source files are already cached, reducing download time significantly
3. Efficient storage: Content-based hashing prevents duplicate cache entries; old unused files are cleaned up after 180 days
4. Resilient: Cache survives periods of inactivity thanks to scheduled refreshes
5. Self-healing: Corrupted cached files are automatically detected, removed, and re-downloaded

Testing

1. Go to aciton -> Build -> Run workflow
2. Select this PR branch
3. Select a package to build (I tried with jellyfin)
4. Look for a "Cache restored" or "cache hit" such as:

Run actions/cache/restore@v4
  
Cache hit for restore-key: distrib-e9b45af652d4bfde67b556fc7f74d2e422c48ed7848777d0645b0d79d47b1d82
Received 226492416 of 12330[10](https://github.com/th0ma7/spksrc/actions/runs/19602310044/job/56135581549#step:6:10)495 (18.4%), 216.0 MBs/sec
Received 478150656 of 12330[10](https://github.com/th0ma7/spksrc/actions/runs/19602310044/job/56135581549#step:6:11)495 (38.8%), 227.8 MBs/sec
Received 687865856 of 1233010495 (55.8%), 217.7 MBs/sec
Received 939524096 of 1233010495 (76.2%), 223.2 MBs/sec
Received 1207959552 of 1233010495 (98.0%), 227.9 MBs/sec
Received 1233010495 of 1233010495 (100.0%), 226.8 MBs/sec
Cache Size: ~[11](https://github.com/th0ma7/spksrc/actions/runs/19602310044/job/56135581549#step:6:12)76 MB (1233010495 B)
/usr/bin/tar -xf /home/runner/work/_temp/6ee36446-93a6-4edd-9ec4-529d0bad6e70/cache.tzst -P -C /home/runner/work/spksrc/spksrc --use-compress-program unzstd
Cache restored successfully
Cache restored from key: distrib-e9b45af652d4bfde67b556fc7f74d2e422c48ed7848777d0645b0d79d47b1d82

6. If there where downloaded files then the sha256sum used as cache identifier will differ:

Run # Compute a hash based on file paths and sizes (not content) for performance.
Distrib cache hash: 7869e838014134fa084a2d95927724c2225b1e2a27d3be92bd589ca748601f9a

Run # Check if a cache with this hash already exists to avoid duplication
New cache content, will save

Run actions/cache/save@v4
/usr/bin/tar --posix -cf cache.tzst --exclude cache.tzst -P -C /home/runner/work/spksrc/spksrc --files-from manifest.txt --use-compress-program zstdmt
Sent 0 of 1418040293 (0.0%), 0.0 MBs/sec
Sent 100[6](https://github.com/th0ma7/spksrc/actions/runs/19602310044/job/56135581549#step:10:7)632960 of 1418040293 (71.0%), 479.8 MBs/sec
Sent 1418040293 of 1418040293 (100.0%), 478.2 MBs/sec
Cache saved with key: distrib-7869e838014134fa084a2d9592[7](https://github.com/th0ma7/spksrc/actions/runs/19602310044/job/56135581549#step:10:8)724c2225b1e2a27d3be92bd589ca748601f9a

7. Once completed re-test, no downloads should then be necessary (SUCCESS!!!)

Fixes #6806

Checklist

• Build rule all-supported completed successfully
• New installation of package completed successfully
• Package upgrade completed successfully (Manually install the package again)
• Package functionality was tested
• Any needed documentation is updated/created

Type of change

• Bug fix
• New Package
• Package update
• Includes small framework changes
• This change requires a documentation update (e.g. Wiki)

[th0ma7]

A few tips & trick using gh client:

Install necessary plugin:

spksrc@spksrc13:~/distrib-reuse/spksrc$ gh extension install actions/gh-actions-cache
✓ Installed extension actions/gh-actions-cache

List all caches:

spksrc@spksrc13:~/distrib-reuse/spksrc$ gh actions-cache list -R th0ma7/spksrc
Total caches size 8.07 GB

Showing 23 of 23 cache entries in th0ma7/spksrc

distrib-19557510383                                                                            737.47 MB  refs/heads/master              a day ago
toolchain-aarch64-7.1-v3-59537071c31d077f771592eb7b0de5c99be590dfb55363d8dc73d42233039efc      134.22 MB  refs/heads/master              a day ago
toolchain-x64-6.2.4-v3-f8de238a18e2d4b06aad98aacf7e2ca7f4c4d273efce08853601e2e4a3514237        78.87 MB   refs/heads/master              a day ago
toolchain-aarch64-6.2.4-v3-a57e870859c9a6fe42b600413a82ef48801da5b244fe8b7744e356684e536cc2    51.54 MB   refs/heads/master              a day ago
toolchain-x64-7.1-v3-e5eefe04312f0af2d4bded960b9b59facb105c594e5ec992a0ab4ec3f3647f0c          143.08 MB  refs/heads/master              a day ago
toolchain-comcerto2k-7.1-v3-0e99514f7447562c04feb0858e378822c69fbea8e086cbdee291c11dbd30b304   58.64 MB   refs/heads/master              a day ago
toolchain-evansport-7.1-v3-d10dbdc9f73abba3e1b2ee0b56a7e289be36c416181bca30511e0fd9d8747640    130.57 MB  refs/heads/master              a day ago
toolchain-hi3535-6.2.4-v3-9107395ec582710cf6e84258f240138f92d28c79344abb2523441418c8b40b61     69.30 MB   refs/heads/master              a day ago
toolchain-88f6281-6.2.4-v3-eb224ada40ce43984cbaa28241cbc1da577bf2f08f587ec265a25172ad66ed77    49.62 MB   refs/heads/master              a day ago
toolchain-armv7-7.1-v3-7aba151bb05c37bf74de01cc73f736ff3a8efe7765d8fb65da06289eb54b4f6f        121.06 MB  refs/heads/master              a day ago
toolchain-armv7-6.2.4-v3-81deb0356544d5bbe894943d72460602e104c277c6cb4469931dbf36fa53a387      58.68 MB   refs/heads/master              a day ago
toolchain-evansport-6.2.4-v3-cf383e07591dd5f281abf24b909e39bad34c6abab79a0b44f1e143b161b0fcd4  65.49 MB   refs/heads/master              a day ago
distrib-19543320817                                                                            922.85 MB  refs/heads/tvh-crash           2 days ago
distrib-19535651506                                                                            862.42 MB  refs/heads/master              2 days ago
distrib-19522811003                                                                            862.42 MB  refs/heads/chromaprint-update  2 days ago
distrib-19486673088                                                                            862.61 MB  refs/heads/chromaprint-update  3 days ago
distrib-19486439755                                                                            312.91 MB  refs/heads/znc-build-fix       3 days ago
distrib-19485440026                                                                            737.70 MB  refs/heads/tvh-crash           3 days ago
distrib-19485238137                                                                            139.34 MB  refs/heads/master              3 days ago
toolchain-x64-7.2-v3-184d22938a15c4f1b9610eddffce2ec62b4d56fea2590b67754a7af9e90d3fd7          158.86 MB  refs/heads/dotnet-fix          4 days ago
toolchain-aarch64-7.2-v3-922cf7c6262246d64e238593c93152e77682a835ebd3cc870375d07da7234480      157.35 MB  refs/heads/dotnet-fix          4 days ago
toolchain-armv7-7.2-v3-8f6eb73e66e9df4916709b7aa74631213050036a6b071c70407b888e23760632        135.50 MB  refs/heads/dotnet-fix          4 days ago
distrib-19466399403                                                                            984.62 MB  refs/heads/dotnet-fix          4 days ago

List distrib only caches:

spksrc@spksrc13:~/distrib-reuse/spksrc$ gh actions-cache list -R th0ma7/spksrc --key "distrib-"
Showing 9 of 9 cache entries in th0ma7/spksrc

distrib-19557510383  737.47 MB  refs/heads/master              a day ago
distrib-19543320817  922.85 MB  refs/heads/tvh-crash           2 days ago
distrib-19535651506  862.42 MB  refs/heads/master              2 days ago
distrib-19522811003  862.42 MB  refs/heads/chromaprint-update  2 days ago
distrib-19486673088  862.61 MB  refs/heads/chromaprint-update  3 days ago
distrib-19486439755  312.91 MB  refs/heads/znc-build-fix       3 days ago
distrib-19485440026  737.70 MB  refs/heads/tvh-crash           3 days ago
distrib-19485238137  139.34 MB  refs/heads/master              3 days ago
distrib-19466399403  984.62 MB  refs/heads/dotnet-fix          4 days ago

[th0ma7]

@mreid-tt and @hgy59 I was able to test the cache changes which worked great! As no chache existed for this branch it falled back to latest cache avaiable from master, from there executed the download of all dependencies, completed the build, then all set.

Now comes the interesting part: I re-ran this same PR, this time it re-used the previous cache, and finally it noticed that all downloads where already present in cache and simply checked digests!

This looks like working well and should avoid tons of false-positives caused by download errors. The only caveat is that I cannot test the cache-maintenance.yml script until this gets merged.

Another pair of 👀 would be nice before pushing this in.

[th0ma7]

I've now also fixed something anoying relatively to *.wrong files created, further when re-using a previously existing cache directory with changes in digests. It now does the validation of the digests and if failed and as to .wrong, re-try once to download the file. Here's a case I tested:

  -> cross/chromaprint-fftw: download-all then checksum
make: Entering directory '/home/runner/work/spksrc/spksrc/cross/chromaprint-fftw'
===>  Downloading files for chromaprint
===>    File chromaprint-1.6.0.tar.gz already downloaded
===>  Verifying files for chromaprint
===>    Checking sha1sum of file chromaprint-1.6.0.tar.gz
===>      Wrong sha1sum for file chromaprint-1.6.0.tar.gz
===>      Renamed as chromaprint-1.6.0.tar.gz.wrong
===>      Download cookie removed to trigger the download again
make: *** [../../mk/spksrc.checksum.mk:42: checksum_target] Error 1
make: Leaving directory '/home/runner/work/spksrc/spksrc/cross/chromaprint-fftw'
  -> Checksum failed due to outdated cached file, retrying download for cross/chromaprint-fftw...
make: Entering directory '/home/runner/work/spksrc/spksrc/cross/chromaprint-fftw'
===>  Downloading files for chromaprint
===>    wget --secure-protocol=TLSv1_2 --timeout=30 --tries=3 --waitretry=15 --retry-connrefused --max-redirect=20 --content-disposition -nv -O chromaprint-1.6.0.tar.gz -nc https://github.com/acoustid/chromaprint/releases/download/v1.6.0/chromaprint-1.6.0.tar.gz
2025-11-22 13:12:59 URL:https://release-assets.githubusercontent.com/github-production-release-asset/1625818/a155672a-7393-4ccd-8185-5bd9fe9be829?sp=r&sv=2018-11-09&sr=b&spr=https&se=2025-11-22T14%3A02%3A29Z&rscd=attachment%3B+filename%3Dchromaprint-1.6.0.tar.gz&rsct=application%2Foctet-stream&skoid=96c2d410-5711-43a1-aedd-ab1947aa7ab0&sktid=398a6654-997b-47e9-b12b-9515b896b4de&skt=2025-11-22T13%3A02%3A10Z&ske=2025-11-22T14%3A02%3A29Z&sks=b&skv=2018-11-09&sig=Ivt%2FMsVbSBaOKAeybztiShIpN6ecibbDQNrZYX3T%2BzA%3D&jwt=*** [1577695/1577695] -> "chromaprint-1.6.0.tar.gz.part" [1]
===>  Verifying files for chromaprint
===>    Checking sha1sum of file chromaprint-1.6.0.tar.gz
===>    Checking sha256sum of file chromaprint-1.6.0.tar.gz
===>    Checking md5sum of file chromaprint-1.6.0.tar.gz
/home/runner/work/spksrc/spksrc/cross/chromaprint-fftw/../../distrib/chromaprint-1.6.0.tar.gz
make: Leaving directory '/home/runner/work/spksrc/spksrc/cross/chromaprint-fftw'

The outcome is as expected and this allowed to re-use all other ffmpeg7 pre-downloaded files and the prepare took only a few seconds instead of 10+ minutes!!!

[hgy59]

@th0ma7 great, if this works, we also shouldn't reach the 10GB cache limit (but if we do, I don't know wheter this redesign will work over the time (probably we have to ensure the limit is not reached an omit further caching ?)

AFAICS the toolchains are not that big (50 to 100 MB) that is typically about 1GB in total (including DSM 7.2?).
But what about kernel sources? I guess we can't cache those due to the size.

Sorry, can't check, currently AKF

[th0ma7]

Thnx for your feedback, will look at how to consider them in the solution. I believe I'm already not that far from having something to play with, preliminary tests looked good but still a WIP.

[mreid-tt]

@th0ma7 congrats on this effort. If you are this deep in the sauce you may want to fix this line in the code which seems to be a syntax error:

spksrc/mk/spksrc.wheel.mk

Line 46 in 600bd55

ifneq ($(and $(WHEEL_NAME),$(or (WHEEL_VERISON),$(WHEEL_URL))),)

[th0ma7]

@mreid-tt and @hgy59 I believe its ready, by that I mean I tested all I could from within the PR. Things worth noting:

1. the distrib-cache-maintenance.yml cannot be tested until merged to master.
2. Caveat is that there may be a tipping point where PR not rebased against master might be continuing generating PR-based distrib caches that may end-up being cleaned-up or using unecessary space. GitHub will automatically kill older cache when reaching 10GB, so in theory these older cache will simply disapear.

[th0ma7]

Friendly reminder @hgy59 @mreid-tt , are we good at trying this out? We can later revert if we find issues beyond the transition phase.

[th0ma7]

Friendly reminder @hgy59 @mreid-tt , are we good at trying this out? We can later revert if we find issues beyond the transition phase.

[mreid-tt]

Some of this is beyond my expertise but it seems good to go.

[hgy59]

AFAICS you limit the cache to 2GB.
Just downloaded all in cross and native folder.
The sources of cross are about 6GB and of native about 1.9GB
without the largest (and rarly or unused) googlfonts, mono*, tessdata thare are 4.1 GB in cross
overall it is about 6GB.

probably we need a strategy to avoid caching the largest source files (even those would save most download time)
like those mentioned above plus: kavita, boost*, openjdk*, jdk*, dotnet-sdk,

further needs
toolchains
6.2.4 1.3 GB
7.1 2.72 GB
7.2 ???

To save some more space we could limit the toolchain caches to generic archs.

[hgy59]

let's give it a try....

[th0ma7]

I'm sure this will need adjustments... Probably on the cleaning script to be more aggressive in the removal of certain areas such as toolchains unused since little while for instance. But we can revert it as needed.

[th0ma7]

@hgy59 and @mreid-tt now merged, let's see how it behave in the comming days...

[th0ma7]

@hgy59 and @mreid-tt Just so you know, there is now a new Distrib Cache Maintenance option in Action where you can run a workflow that will clean-up old files on the latest distrib cache. The setting is currently at 180 days "unused" ... pretty sure it will require fine-tuning, wait to be seen. In the meantime this will run on a weekly basis to unclutter the lastest distrib cache, which normally should contain all is needed for on-going PR.