Skip to content

ci: unshallow submodule fetch to fix intermittent libbacktrace checkout failures#1946

Closed
tonibohnlein wants to merge 1 commit into
hw-native-sys:mainfrom
tonibohnlein:ci/unshallow-submodule-fetch
Closed

ci: unshallow submodule fetch to fix intermittent libbacktrace checkout failures#1946
tonibohnlein wants to merge 1 commit into
hw-native-sys:mainfrom
tonibohnlein:ci/unshallow-submodule-fetch

Conversation

@tonibohnlein

Copy link
Copy Markdown
Contributor

Summary

Fixes an intermittent actions/checkout failure that fails CI jobs before any build or test runs — most visibly dist-system-tests / system-tests:

/usr/bin/git -c protocol.version=2 submodule update --init --force --depth=1
Error: fatal: Needed a single revision
Error: Unable to find current revision in submodule path '3rdparty/libbacktrace'

Root cause

actions/checkout@v4 defaults to fetch-depth: 1, and it fetches submodules at the same depth — i.e. git submodule update --init --force --depth=1, which fetches only each submodule's branch tip. 3rdparty/libbacktrace is pinned (via .gitmodules: branch = macho-bundle-support) to a commit that is not the current tip of that branch, so the shallow fetch grabs the tip and can't resolve the pinned commit → "Needed a single revision."

It's intermittent because it depends on whether the persistent self-hosted runner's submodule cache already contains the pinned commit; when the cache is cold/cleaned, the shallow re-fetch fails.

Fix

Add fetch-depth: 0 to every submodules: true checkout (8 in ci.yml, 3 in daily_ci.yml). With full history, git submodule update fetches the pinned libbacktrace commit unconditionally. No submodule-pointer or .gitmodules change — this is checkout config only, so it can't alter what's built.

Trade-off: a full clone instead of a shallow one. On the persistent self-hosted runners (where this fails) the fetch is incremental/cached; on the GitHub-hosted unit-tests jobs it's a modest one-time cost for a small repo.

Scope

This is one of a few recurring infra flakes on the device CI. For the record, the others are handled separately:

actions/checkout@v4 defaults to fetch-depth: 1, which fetches submodules
with --depth=1 (only each submodule's branch tip). 3rdparty/libbacktrace is
pinned (via .gitmodules branch = macho-bundle-support) to a commit that is
NOT the current branch tip, so the shallow fetch cannot resolve it:

  git submodule update --init --force --depth=1
  fatal: Needed a single revision
  Error: Unable to find current revision in submodule path '3rdparty/libbacktrace'

This surfaces intermittently (runner-cache dependent) as an actions/checkout
failure in dist-system-tests / system-tests, before any build or test runs.

Set fetch-depth: 0 on every `submodules: true` checkout so submodules are
fetched with full history and the pinned commit always resolves. No submodule
pointer or .gitmodules change.
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Note

Gemini is unable to generate a review for this pull request due to the file types involved not being currently supported.

@coderabbitai

coderabbitai Bot commented Jul 3, 2026

Copy link
Copy Markdown

Warning

Review limit reached

@tonibohnlein, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 11 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: f0260d1d-04aa-4f1b-9348-659ef7a91b24

📥 Commits

Reviewing files that changed from the base of the PR and between 39a13a9 and 3446dd9.

📒 Files selected for processing (2)
  • .github/workflows/ci.yml
  • .github/workflows/daily_ci.yml

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@tonibohnlein

Copy link
Copy Markdown
Contributor Author

Closing — this approach backfires, so it's a net negative.

fetch-depth: 0 makes actions/checkout do a full git fetch --unshallow origin +refs/heads/*:... +refs/tags/*:... of the entire repo history (all branches + tags). On the runners' flaky GnuTLS/network that large transfer drops mid-stream and fails deterministically:

git -c protocol.version=2 fetch --prune --no-recurse-submodules --unshallow origin ...
error: RPC failed; curl 56 GnuTLS recv error (-9): Error decoding the received TLS packet.
fatal: early EOF / fetch-pack: invalid index-pack output

toolchain and clang-tidy (which pass on the shallow default) failed all 3 retries this way; the submodules were never even reached.

Reassessment — there are two independent infra issues, and the change traded a small one for a bigger one:

  1. libbacktrace --depth=1 shallow-fetch can't reach its off-branch-tip pinned commit (cache-dependent; the original dist flake).
  2. GnuTLS / early-EOF on git transfers (runner TLS/network instability) — amplified, not fixed, by fetching ~100× more data.

A safe repo-side fix for (1) alone (deep submodules without unshallowing the main repo) is entangled with (2) and with the private runtime submodule (hw-native-sys/simpler) needing checkout's auth, so it isn't a clean workflow change. Both are best addressed runner-side — fix the GnuTLS/network (e.g. OpenSSL git build / larger http.postBuffer / HTTP/1.1), and/or pre-seed or update the libbacktrace pin in the runner cache. Flagging to the CI/runner owners.

@tonibohnlein tonibohnlein deleted the ci/unshallow-submodule-fetch branch July 4, 2026 19:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

1 participant