pre-push always fail with "child zlib.h is a dangling symbolic link" error

[railwaycat]

Title: pre-push always fail with "child zlib.h is a dangling symbolic link" error

Description:
After Envoy migrated to Bazel 7.*, the envoy pre-push check consistently fail with error child zlib.h is a dangling symbolic link

The full error message as:

INFO: Analyzed target //tools/proto_format:proto_sync (553 packages loaded, 15098 targets configured).
ERROR: /home/xxxx/factory/envoy_xxx/bazel/foreign_cc/BUILD:414:12: error while validating output tree artifact bazel/foreign_cc/zlib/include: child zlib.h is a dangling symbolic link
ERROR: /home/xxxx/factory/envoy_xxx/bazel/foreign_cc/BUILD:414:12: Foreign Cc - CMake: Building zlib [for tool] failed: not all outputs were created or valid
Target //tools/proto_format:proto_sync failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 7.050s, Critical Path: 2.49s
INFO: 287 processes: 181 internal, 106 linux-sandbox.
ERROR: Build did NOT complete successfully
ERROR: Build failed. Not running target

I saw a related discussion under Bazel: bazelbuild/bazel#20886, which suggests adding build option --noincompatible_sandbox_hermetic_tmp for Bazel. But I'm not sure where to add for pre-push verifications.

I can reproduce this issue on ubuntu 22.04.5 and debian stable 12.8.

[optional Relevant Links:]

Any extra documentation required to understand the issue.

[phlax]

@railwaycat can you try using the build container - i think this may be related to llvm version or setup

[phlax]

hmm - actually this is the same error that istio hit (cc @zirain)

in that case they worked around the problem by increasing the bazel version - not sure on what the actual cause is

[railwaycat]

Thank you @phlax!