Azure CI jobs sometimes fail with "fatal: couldn't find remote ref"

[traversaro]

In the last weeks, I have noticed a few cases in which some PR jobs start failing with errors like:

==============================================================================
Syncing repository: conda-forge/idyntree-feedstock (GitHub)
git version
git version 2.47.0
git lfs version
git-lfs/3.5.1 (GitHub; linux amd64; go 1.21.8)
git init "/home/vsts/work/1/s"
hint: Using 'master' as the name for the initial branch. This default branch name
hint: is subject to change. To configure the initial branch name to use in all
hint: of your new repositories, which will suppress this warning, call:
hint:
hint: 	git config --global init.defaultBranch <name>
hint:
hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
hint: 'development'. The just-created branch can be renamed via this command:
hint:
hint: 	git branch -m <name>
Initialized empty Git repository in /home/vsts/work/1/s/.git/
git remote add origin https://github.com/conda-forge/idyntree-feedstock
git config gc.auto 0
git config core.longpaths true
git config --get-all http.https://github.com/conda-forge/idyntree-feedstock.extraheader
git config --get-all http.extraheader
git config --get-regexp .*extraheader
git config --get-all http.proxy
git config http.version HTTP/1.1
git --config-env=http.extraheader=env_var_http.extraheader fetch --force --tags --prune --prune-tags --progress --no-recurse-submodules origin   +refs/heads/*:refs/remotes/origin/* +refs/pull/112/merge:refs/remotes/pull/112/merge
fatal: couldn't find remote ref refs/pull/112/merge
##[warning]Git fetch failed with exit code 128, back off 2.266 seconds before retry.
git --config-env=http.extraheader=env_var_http.extraheader fetch --force --tags --prune --prune-tags --progress --no-recurse-submodules origin   +refs/heads/*:refs/remotes/origin/* +refs/pull/112/merge:refs/remotes/pull/112/merge
fatal: couldn't find remote ref refs/pull/112/merge
##[warning]Git fetch failed with exit code 128, back off 8.175 seconds before retry.
git --config-env=http.extraheader=env_var_http.extraheader fetch --force --tags --prune --prune-tags --progress --no-recurse-submodules origin   +refs/heads/*:refs/remotes/origin/* +refs/pull/112/merge:refs/remotes/pull/112/merge
fatal: couldn't find remote ref refs/pull/112/merge
##[error]Git fetch failed with exit code: 128
Finishing: Checkout conda-forge/idyntree-feedstock@refs/pull/112/merge to s

See conda-forge/idyntree-feedstock#112 . In most cases to solve the problem restarting the CI was not effective, while making the bot open a new PR was effective. I am not sure what is causing this or if this is the right place to report the issue, but I prefer to have at least an open issue on this so that I have someting to cross-link whenever this happens.

[traversaro]

In most cases to solve the problem restarting the CI was not effective, while making the bot open a new PR was effective.

Also pushing new commits seems effective to fix the problem.

[traversaro]

See also:

• https://matrix.to/#/!SOyumkgPRWoXfQYIFH:matrix.org/$17295915824STUby:gitter.im?via=matrix.org&via=gitter.im&via=ifisc.uib-csic.es
• [bot-automerge] qdax v0.4.0 qdax-feedstock#8
• linux-aarch64 / osx-arm64 builds torchaudio-feedstock#4

[traversaro]

fyi @Tobias-Fischer

[beckermr]

please see conda-forge/status#188

I am guessing this is a duplicate of that one.

[traversaro]

please see conda-forge/status#188

I am guessing this is a duplicate of that one.

I am not sure. This problem predates that one (as you can see in conda-forge/qdax-feedstock#8), and also the high level error is different: in one case the Azure CI jobs did not start, in this one it starts and it fails after a few seconds. However they may definitely be connected. If this does not occur anymore in a few days, I think we can close it.

[mfansler]

Can you try an empty commit? Close-reopen has been insufficient on all my tries, but I saw a staged-recipes PR clear up after a merge from main. I'm still waiting to see if my bot-reruns resolve it for some bot version bump PRs.

[mfansler]

@danielnachun and I were also seeing this since at least Friday.

[traversaro]

Can you try an empty commit? Close-reopen has been insufficient on all my tries, but I saw a staged-recipes PR clear up after a merge from main. I'm still waiting to see if my bot-reruns resolve it for some bot version bump PRs.

I tried a non-empty commit, and it fixed the problem.

[Tobias-Fischer]

I’ve tried non empty commits (rerender for example) and that didn’t resolve the issue in my case

[traversaro]

I’ve tried non empty commits (rerender for example) and that didn’t resolve the issue in my case

I vaguely recalled that, but I had no reference so I was not sure. In my case, the non-empty commit was done by me today (not a bot) and it worked. Not sure if the difference was the committer or simply that the problem today is different from a few days ago.

[traversaro]

Another instance: conda-forge/smirnoff-plugins-feedstock#11 .

[mattwthompson]

Yeah I was running into this earlier today, but an empty commit seemed to do this trick when re-opening the PR didn't. I didn't look into it any more closely than that.

[mattwthompson]

Getting this again today. No clue why.

[mattwthompson]

Empty commit seems to work again. Still confusing and a little frustrating

[jakirkham]

Are we still seeing this issue? If so, can you please share links to any recent examples?

Thanks in advance! 🙏