Actions/cache: Cache not being hit despite of being present

[RJ722]

Hi,

My workflow looks like this:

- name: Set Poetry Config
      run: |
        poetry config virtualenvs.create true
        poetry config cache-dir
- uses: actions/cache@v1
  name: Poetry Cache
  with:
    path: ~/.cache/pypoetry/virtualenvs
    key: poetry-${{ runner.os }}-${{ hashFiles('poetry.lock') }}
    restore-keys: |
      poetry-${{ runner.os }}-

- name: Install python dependencies
  run: make installdeps

The corresponding logs showed:

poetry config virtualenvs.create true
poetry config cache-dir
shell: /bin/bash -e {0}
env:
DEEPSOURCE_DSN: ***
pythonLocation: /opt/hostedtoolcache/Python/3.8.2/x64
/home/runner/.cache/pypoetry
Cache not found for input keys: poetry-Linux-10e15406c5ba2cd9517ecdbad0c577bdaeb25da4c86ef997a3082522bfa0d618, poetry-Linux-.

Run actions/cache@v1

with:

path: ~/.cache/pypoetry/virtualenvs

key: poetry-Linux-10e15406c5ba2cd9517ecdbad0c577bdaeb25da4c86ef997a3082522bfa0d618

restore-keys: poetry-Linux-
env:

DEEPSOURCE_DSN: ***

pythonLocation: /opt/hostedtoolcache/Python/3.8.2/x64

Cache not found for input keys: poetry-Linux-10e15406c5ba2cd9517ecdbad0c577bdaeb25da4c86ef997a3082522bfa0d618, poetry-Linux-.

The post job logs show that cache was stored successfully:

Post job cleanup.
/bin/tar -cz -f /home/runner/work/_temp/d1cb4b2a-f400-44ba-ac17-8a863c548c95/cache.tgz -C /home/runner/.cache/pypoetry/virtualenvs .
Cache saved successfully

However, on the second run, the cache is missed again. Logs here:

Cache not found for input keys: poetry-Linux-10e15406c5ba2cd9517ecdbad0c577bdaeb25da4c86ef997a3082522bfa0d618, poetry-Linux-.
Run actions/cache@v1
  with:
    path: ~/.cache/pypoetry/virtualenvs
    key: poetry-Linux-10e15406c5ba2cd9517ecdbad0c577bdaeb25da4c86ef997a3082522bfa0d618
    restore-keys: poetry-Linux-
env:

DEEPSOURCE_DSN: ***

pythonLocation: /opt/hostedtoolcache/Python/3.8.2/x64

Cache not found for input keys: poetry-Linux-10e15406c5ba2cd9517ecdbad0c577bdaeb25da4c86ef997a3082522bfa0d618, poetry-Linux-.

Does anyone have an idea about why the cache is being missed in the second run? Also, would it be apt to open an issue on https://github.com/actions/cache?

Thanks!

[BrightRan]

@rj722 ,

I tested and tried as the demo you shared, but I did not reproduce the problem. Not sure what causes this problem.

I have helped you report this ticket in the repository of the cache action:

actions/cache#231

You can follow this issue ticket, and add you comment here.

Of course, actually any user can directly open an issue this repository, that will allow you to directly interact with the appropriate engineering team, and make it more convenient for the engineering team to help you solve the problems occur when you use this action, and also can collect and categorize your suggestions for this action. I think this is why the team will set the repository to be public.

[david-molnar-oculai]

This just makes no sense. Please fix this.

[amirasyraf]

That's absurd. How would this work for pull requests?

What's the point of deciding this for the users as opposed to letting them decide? That's what the cache key is for.

[rene-hermenau]

It’s recommended that you have a workflow run on your default branch (usually master ) so that worklows on other branches will have a cache to restore from.

This makes no sense for us! We are building our dist of a branch, run workflow generated tests on the same branch, when they pass, we merge into master, not the opposite around.

[1ukastesar]

This is not working for any project that uses PRs and branching. In these projects, the cache hit almost never occurs, despite its key matching entirely.

It is just stupid; please add at least configurable option to turn that feature off.

[kamzil]

This should not be the marked answer.

[nroose]

This doesn’t make sense to me. Seems like the cache should be dependent on the repo and the keys. Not on the branch. And in my experience, it only gets a cache hit on the same commit when I re-run the workflow. It never seems to get a hit even on the same branch. This is pretty silly.

[partydrone]

Keeping things isolated by branch makes sense when you don't want the cache in one branch polluting the cache in another branch. For example, if I have a node project and I open a PR that adds, updates, or deletes a new node module to the project, then someone else on my team opens a PR that triggers the same workflow, a repo-level cache would likely cause issues.

[mmbarness]

This can be prevented if the engineer dealing with this writes out a cache key specific enough to eliminate other branches. Why is, instead, github making it impossible for us to make our own decisions about this.

[matinzd]

Handling cache hits at the repository level seems like a more sensible approach. We can easily customise cache keys by incorporating the branch's reference name into the key. The current default behaviour can be a bit confusing and unconventional or at least there should be a way to override this behaviour.
I am wondering how tags work in this scenario.
This can significantly reduce build times but it seems that Github wants to charge us for it.

[branberry]

@partydrone but at least in that case a developer would know which cache is problematic, and like @mmbarness mentions, that could be handled by creating a more distinguishable cache key. It's so much more confusing and frustrating having a cache miss even though the cache key matches exactly. It just doesn't make any intuitive sense.

[nroose]

Oh, so it’s scoped to the specific tag for runs triggered by a tag push as well. This is even more silly. It means that no tag build can use cache. Why do they scope it to the branch???

[mihaerzen]

Hey. How did you figure out it is scoped to the tag as well?

[kostyaplis]

  <a href="https://github.com/actions/cache/issues/556" target="_blank" rel="noopener nofollow ugc">github.com/actions/cache</a>

using cache on tags does not work

<div class="github-info">
  <div class="date">
    opened <span class="discourse-local-date" data-format="ll" data-date="2021-03-27" data-time="11:31:57" data-timezone="UTC">11:31AM - 27 Mar 21 UTC</span>
  </div>


  <div class="user">
    <a href="https://github.com/hq9000" target="_blank" rel="noopener nofollow ugc">
      <img alt="hq9000" src="https://user-images.githubusercontent.com/53430368/181101539-7c55ed4c-5cad-4ed7-bdbd-ea6e43b09a88.jpeg" class="onebox-avatar-inline" width="20" height="20">
      hq9000
    </a>
  </div>
</div>

<div class="labels">
</div>

hi!

I have a cache created on a non-default branch, all works fine except wh…en the workflow is run on a tag, the cache isn't hit

when it's not a tagged commit, and my github.ref is "...heads" it starts working again.

I guess it's related to https://github.com/actions/cache#cache-scopes

Essentially, it looks like there is no way to use cache in runs on tagged revisions, or is it? (I don't have any caches on my master yet). Is it an intended behavior?

At least, it'd be good if it was described in readme in that "scopes" section.

[swushi]

Makes no sense to cause a cache miss if it's from a non-default branch. That makes the cache much less effective. This essentially causes any workflow that you develop and presumably cache in a PR immediately cache-miss when it's merged into the default branch.

The hash is guaranteeing a stable cache, regardless of branch.. I don't see the point for the branch protection as well.

[SvenStaehs]

well, I can see it as a failsafe measure. You say The hash is guaranteeing a stable cache, but is this really the case? Will you bet the stability of your delivery on the fact that nobody managed to break the cache for a given content of poetry.lock?

The thing is, the key does not guarantee anything. There are so many things that can go wrong, even if you don't take malice into account. First the key is generated, then the venv is filled, then some action runs using that venv, then the venv is published as cache. But the action can have modified the contents of what you want to cache, in this case something could have installed additional packages in the poetry venv, without any change in poetry.lock, thus not changing the key.
If two branches make the same changes in poetry.lock, but one of them breaks the venv in their action, the other branch (and even the main branch!) could get this broken cache.

As far as I understand, running the workflow on the same branch with the same poetry.lock file should restore the cache just fine, so it only has to run once and from then on will get the cached venv. It's a tradeoff, and confusing as heck because it's not prominently documented, but it's a tradeoff I can understand (with the power of hindsight)

I hope my explanation attempt wasn't too confusing, sorry 😉

[alambike]

Why not let the user tweak the cache scope? I mean, using the cache/save action, GitHub could allow you to define the scope of the cache to be stored.

In this way we can have safe defaults, but allow them to be altered for other use cases.

For example, when merging a PR against the default branch, I would like to be able to promote the last cache generated in the PR, to the scope of the default branch, so that it would be the most recent so new PRs could use it.

But the way the action currently behaves, I don't see a way to do it, without launching a new workflow that does the build again and publishes the cache, once the PR has been merged.

A waste of minutes and time since this new build is going to be exactly the same as the last build executed in the PR. Furthermore, if a new PR comes in while the default branch is being built, it will grab an older version from the cache (unless the cache key has been altered).

[nroose]

This is one is the main reasons GitHub actions didn't work for us. The cache key, assuming you can set it in a relatively dynamic/powerful way (hash, branch, etc.), is enough, without any additional scoping. And caching is important to make many workflows fast enough to be practical. Most of our workflows are not going to production.

[Cypher1]

This caching behavior should at minimum be configurable, if not straight up not branch specific.

[nroose]

Sure. That’s why we have cache keys. And fallbacks. Let the person configuring the cache do that.

…

On Fri, Mar 31, 2023 at 9:51 AM Andrew Porter ***@***.***> wrote: Keeping things isolated by branch makes sense when you don't want the cache in one branch polluting the cache in another branch. For example, if I have a node project and I open a PR that adds, updates, or deletes a new node module to the project, then someone else on my team opens a PR that triggers the same workflow, a repo-level cache would likely cause issues. — Reply to this email directly, view it on GitHub <#27059 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABLJIQBU5MIZMFMF6CIRJTW64DQXANCNFSM6AAAAAAQK2XZJI> . You are receiving this because you commented.Message ID: ***@***.***>

[btfshadow]

I really dont see any improvement to use a cache inside the same branch, and that don't work with the default branch either ... only bullshit.
If I make a cache I want to use every time I want ...

[RafidMuhymin]

Keeping things isolated by branch makes sense when you don't want the cache in one branch polluting the cache in another branch. For example, if I have a node project and I open a PR that adds, updates, or deletes a new node module to the project, then someone else on my team opens a PR that triggers the same workflow, a repo-level cache would likely cause issues.

@partydrone A repository-level cache would likely cause issues. But not all people use the action just to cache node modules. There are many valid use cases. For example, I might want to use a production cache for the main branch, and for all the other branches I might want to use another development cache.

And even if it's a security vulnerability, it's not so high that it won't be even configurable. The discussion is running for 3 years without any response.

[wind57]

We have a real use-case where again circleci got it right and github got it wrong.

We want to split tests by their times, so that each matrix runs more or less the same amount of time, think of it as a perfectly distributed tests per each instance of the matrix.

If I am on branch-A and I have 100 tests, this will generate some distribution of test-times per one matrix. I cache these results. If I add 10 more tests, and try to get the previous distribution... surprise! there is no cache. This leaves me speechless.

[cAstraea]

So what's the solution ? Is my understanding that for some arbitrary reason it won't run on

name: Deploy development

on:
  push:
    tags:
      - '[0-9]+.[0-9]+.[0-9]+-release*dev'

Is that correct ?

[MatMercer]

After doing everything fine, and even testing the cache manually in a test branch, the cache suddenly isn't being hit, what is going on?

The cache is saved in a first run:

Then, not hit for no reason:

The two caches exists:

And the action is configured like that:

      - name: Restore cached golangci-lint
        id: golangci-lint-restore
        uses: actions/cache/restore@v3
        with:
          path: ${{ env.GOLANGCI_LINT_CACHE }}
          # search for golangci-lint-{commit} OR restore latest golangci-lint-*
          # for the current branch or main branch
          key: golangci-lint-${{ github.sha }}
          restore-keys: |
            golangci-lint-

Anyone know what is going on? env.GOLANGCI_LINT_CACHE is a fixed value and won't change.

@BrightRan do you have any clues?

[MatMercer]

@BillyTom

No:

If cache key is not found on current branch it will search current branch cache using restore-keys. If there is also no match it will search default repository branch cache using key and then restore-keys. Here is some documentation

The cache is at my default branch, and the workflow is running from a pr event. The cache still isn't matching, and for the above comment I posted it started working randomly.

Here's another person that started working randomly: https://github.com/orgs/community/discussions/58583#discussioncomment-6235462

There's clearly a bug for how caches are matched in the GitHub action.

Cache was saved at the default branch

Workflow was triggered in a PR after the cache was saved

[MatMercer]

Now I'm just waiting for it to randomly work again, as before, but there's clearly something weird going on

[udondan]

About two weeks ago, I noticed it started working for me again. Not sure it always works everywhere but I have seen it working on some renovate PRs. I just did a manual test, for a branch that certainly never existed before and the cache was restored instantly.

My cache action: https://github.com/udondan/cdk-ec2-key-pair/blob/509e42c4fb9fd3d1667babffd11c3bfce0761b75/.github/workflows/test.yml#L35-L46

[MatMercer]

@udondan

Can it be because you are using actions/cache@v4 and I actions/cache/restore@v4? This is the only difference I see...

[AllDmeat]

Bumped into same issue: actions/cache/restore@v4 couldn't access cache created by actions/cache@v4. Had to modify workflow to always use actions/cache, that helped.

[pavelsaman]

Is there any solution to this? It seems like people would like to configure the scope for cache but it also seems it's not configurable in any way. I've recently encountered this limitation myself as well.

[david-molnar-oculai]

Classic GitHub stupidity. It doesn't work for releases as it's always has a different tag which means no cache hit. It makes no sense. I used the hash of package-lock.json in the cache key. It's safe to cache it and use it even for different branches.

[emschu]

Thank you for this gigantic resource waste by design

[vegidio]

Github clearly doesn't want cache to work in a way that makes sense because they can charge corporate users for using more minutes of Github Actions.

[rene-hermenau]

Due to this, We are seriously considering switching from GitHub to GitLab Ci or another CI.
We don't care about the costs, the CI minutes in GitHub are not much and are fair rated but we need a caching solution that makes sense and that saves us time. At the moment our tests eat up huge time because our builds are recreated on each of our 70 workflows and we are required to run them all for each release. It would save us tons of waiting time if we can share caches between different workflows.

If this decision was made by design it's not smart. You will loose customers Github! Once, we move our CI to another service, it's just a matter of time until we have to move our whole project. I don't want to, and love GH, but its starting to get a bad taste.

[bgdnvk]

interesting design choice!

[adrians5j]

Based on everything that's been discussed above, it does look like it might just be better not to use actions/cache, but implement your own, where you store caches in an S3 bucket or something. 🤷🏻

[ooaklee]

This is precisely what runs-on has done with their drop in version of the actions/cache action, https://runs-on.com/caching/s3-cache-for-github-actions/#introducing-runs-on-cache:~:text=for%20large%20caches.-,Introducing%20runs%2Don/cache,-Our%20drop%2Din

[nroose]

That’s bound to be much slower.

…

On Wed, Apr 3, 2024 at 6:25 AM Adrian Smijulj ***@***.***> wrote: Based on everything that's been discussed above, it does look like it might just be better not to use actions/cache, but implement your own, where you store caches in an S3 bucket or something. 🤷🏻 — Reply to this email directly, view it on GitHub <#27059 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABLJIRGJ3B536P6FRSOG73Y3P7LFAVCNFSM6AAAAAAQK2XZJKVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DSOJXGEZTI> . You are receiving this because you commented.Message ID: ***@***.***>

[ahmadabdul3]

ugh... why is it like this. I'm running into the same issue, but theres a difference.
I have a CI workflow when I create a PR which stores caches (per branch in some instances)
I have another workflow that runs manually when I want to deploy to specific staging environments.

The weird thing is, if I run the deploy workflow on the same branch that the PR workflow ran on, it still cant find the cache. I dont understand how its 'scoped to a branch' if both workflows are running on the same branch...

if the PR is branch-a -> master and the deploy workflow runs on branch-a directly, do those count as 2 different branches?
I do know that PR workflows in GH-actions run in a detached state. 🤦

this is very frustrating

[david-molnar-oculai]

...and of course, GitHub just ignores us, they have nothing to say.

[rene-hermenau]

I have a feeling that since this discussion was tagged as "Answered by BrightRan", no one from the core Github team will ever read our comments!

Guys, we need to open another discussion!

[ajfranzoia]

In my case, I found a workaround for my Yarn dependencies:

• create a compressed file of the cache dir and upload to a remote storage provider
• add the file public URL as an Actions secret
• in the first step of the workflow, download the file to the Yarn cache folder

This is sub optimal because it requires updating the cache file once in a while, but it's better than no cache at all

[mtsadler-branch]

If someone wants to make a universal action for this, I'm sure people would sponsor it

[kamzil]

Open issue in actions/cache#79

[jakallergis]

this works for me:

jobs:
  validate:
    name: Validate
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Use Node.js 18.x
        uses: actions/setup-node@v3
        with:
          node-version: "18.x"

      - uses: actions/cache@v4
        if: ${{ !env.ACT }}
        id: yarn-cache # use this to check for `cache-hit` (`steps.yarn-cache.outputs.cache-hit != 'true'`)
        with:
          path: ./node_modules
          key: ${{ runner.os }}-modules-${{ hashFiles('**/yarn.lock') }}
          restore-keys: |
            ${{ runner.os }}-yarn-

      - name: Install Node Modules
        run: yarn install --network-timeout 100000

      - name: Validate and run tests
        run: yarn validate

yarn install runs in les than 1s.
Screenshot for proof:

[kamzil]

Are you saying that you manage to achieve cache hit for stuff that wasn't cached in the main branch but only in feature branch? I can't pinpoint anything special in your example that would enable that.

[OscarVanL]

I just fixed this issue for myself.

Before I had a workflow that used:

• Absolute paths
• Templated path values from previous step (steps.xyz.outputs.foo)
• Cache paths prefixed with ./

I made some changes to use:

• Relative paths
• No path templating
• No ./ prefix

And miraculously I'm getting cache hits.

I unfortunately did not try these all separately so haven't figured out which specific (or combination) of these changes were necessary.

Before:

    - name: Get yarn cache dir
      id: get-yarn-cache-dir
      shell: bash
      run: |
        echo "dir=$(cd src/frontend && yarn config get cacheFolder)" >> $GITHUB_OUTPUT
    - name: Cache webpack
      uses: actions/cache@v4
      env:
        CACHE_NAME: frontend-webpack
      with:
        # Cache yarn and webpack (node_modules/.cache)
        path: |
          ${{ steps.get-yarn-cache-dir.outputs.dir }}
          ./src/frontend/node_modules/.cache
        key: ${{ env.CACHE_NAME }}-${{ hashFiles('src/frontend/yarn.lock') }}
        restore-keys: |  # Failing an exact match, match the most recent cache with the same prefix 
          ${{ env.CACHE_NAME }}-

After (working):

    - name: Cache webpack
      uses: actions/cache@v4
      env:
        CACHE_NAME: frontend-webpack
      with:
        path: |
          src/frontend/.yarn/cache
          src/frontend/node_modules/.cache
        key: ${{ env.CACHE_NAME }}-${{ hashFiles('src/frontend/yarn.lock') }}
        restore-keys: ${{ env.CACHE_NAME }}-

I'm convinced that what I had before was perfectly fine and correct, but for some reason GitHub Actions just was not happy with it.

[Nguyen-Ngoc-Phuoc]

You just save me from losing my mind ;) Please take 100 upvotes

After the change the path to relative, the stuff just works

[OscarVanL]

I'm glad to hear that worked! I guess that was the fix then 😁

[p1utoze]

For me, I was missing a trailing path slash / , I was confused at first then checked with colordiff command and noticed it. Even though the key hash is correct, the path must be EXACTLY same.
Cache generated with this path

...
path: |
    ${{ env.TF_WORKING_DIR }}/.terraform.lock.hcl
    ${{ env.TF_WORKING_DIR }}/.terraform

The workflow trying to restore was:

...
path: |
   ${{ env.TF_WORKING_DIR }}/.terraform.lock.hcl
   ${{ env.TF_WORKING_DIR }}/.terraform/

[oleg-andreyev]

Hit this problem today...

WITHIN SAME BUILD
same cache-key different results

[oleg-andreyev]

one more disc. https://github.com/orgs/community/discussions/163260

[deanlot]

I can confirm we have exactly the same issues today. I'm updating my workflows to fallback to running npm i / pnpm i on cache miss / fail as a work around, but this is sub optimal.

[deanlot]

Furthermore, we're also seeing this error sporadically:
Failed to save: Unable to reserve cache with key Linux-my_key-my_hash, another job may be creating this cache.
I am 100% certain that this is the only workflow running in my current branch / repository.

[tanin47]

We've been seeing the same issue since last night (several hours ago)

The cache is created by a prior job, and a later job doesn't find it. The restore-keys also doesn't work occasionally.

It was working before. It feels like there's an incident going on.

[tanin47]

Github has opened an incident an hour ago: https://www.githubstatus.com/

[Martin-Narvar]

This totally stupid design choice makes Github actions completely useless for the majority of use cases.
In my case my cache key is a hash of package-lock.json and the only thing we cache is node_modules. This cache can safely be shared by all branches and tags with no risk of incorrect behavior.
Despite this, I have to wait 10 mins or more for my PR checks that would take seconds if the cache was used.