Cannot cross-compile Gazelle extension

[linzhp]

🐞 bug report

Affected Rule

The issue is caused by the rule: Gazelle extension

Is this a regression?

Yes, before #1895, we were able to cross-compile Gazelle extension by disabling cc toolchain resolution with --noincompatible_enable_cc_toolchain_resolution (#1825).

Description

The cross-compilation of Gazelle extension is broken after #1895, due to the introduction of dependency on a C library .

The issue can be partly mitigated by using hermetic_cc_toolchain, which supports cross-compiling into Linux, but we still cannot cross-compile from Linux into any macOS, or from darwin_arm64 into darwin_amd64.

This is technically not a problem specific to the Gazelle extension, but a problem of the lack of C toolchain support. However, are we better off by replacing a Python dependency with a C dependency?

🔬 Minimal Reproduction

From the gazelle directory of this rules_python repo on a Linux or Apple M1 machine:

bazel build --platforms=@io_bazel_rules_go//go/toolchain:darwin_amd64_cgo //python:gazelle_binary

🔥 Exception or Error

ERROR: /home/user/.cache/bazel/_bazel_zplin/973b3f124c212b9dd687dfdeb379d909/external/rules_go~/BUILD.bazel:86:17: While resolving toolchains for target @@rules_go~//:cgo_context_data (d965f1d): No matching toolchains found for types @@bazel_tools//tools/cpp:toolchain_type.
To debug, rerun with --toolchain_resolution_debug='@@bazel_tools//tools/cpp:toolchain_type'
If platforms or toolchains are a new concept for you, we'd encourage reading https://bazel.build/concepts/platforms-intro.

🌍 Your Environment

Operating System:

• Linux
• macOS

Output of bazel version:

7.1.2

Rules_python version:

730a2e3

Anything else relevant?
We can register a hermetic_cc_toolchain by adding this to MODULE.bazel file:

bazel_dep(name = "hermetic_cc_toolchain", version = "3.1.0")

toolchains = use_extension("@hermetic_cc_toolchain//toolchain:ext.bzl", "toolchains")
use_repo(toolchains, "zig_sdk")

register_toolchains(
    "@zig_sdk//toolchain:linux_amd64_gnu.2.31",
    "@zig_sdk//toolchain:linux_arm64_gnu.2.31",
    "@zig_sdk//toolchain:darwin_arm64",
    "@zig_sdk//toolchain:darwin_amd64",
)

Then from a Linux machine:

$ bazel build --platforms=@zig_sdk//platform:darwin_arm64 //python:gazelle_binary
INFO: Analyzed target //python:gazelle_binary (133 packages loaded, 15547 targets configured).
ERROR: /home/user/go-repos/src/github.com/bazelbuild/rules_python/gazelle/python/BUILD.bazel:92:15: GoLink python/gazelle_binary_/gazelle_binary failed: (Exit 1): builder failed: error executing GoLink command (from target //python:gazelle_binary) bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/go_sdk/builder_reset/builder link -sdk external/go_sdk -installsuffix darwin_arm64 -arc ... (remaining 95 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
external/go_sdk/pkg/tool/linux_amd64/link: running external/hermetic_cc_toolchain~~toolchains~zig_sdk/tools/aarch64-macos-none/c++ failed: exit status 1
error: unable to find framework 'CoreFoundation'. searched paths:  none
error: unable to find framework 'Security'. searched paths:  none

link: error running subcommand external/go_sdk/pkg/tool/linux_amd64/link: exit status 2
Target //python:gazelle_binary failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 16.068s, Critical Path: 14.64s
INFO: 52 processes: 2 internal, 50 processwrapper-sandbox.
ERROR: Build did NOT complete successfully

[aignas]

@hunshcn, @linzhp, should we revert the PR until we can find a fix for this? What do you think? I guess we should add a test in the CI to check that we can cross-compile.

[linzhp]

Yeah, I am planning to revert our internal diff that upgraded the Gazelle extension. Sorry for not catching this earlier. I thought it was a "pure golang helper", but it turned out to depend heavily on C

[hunshcn]

I don't know much about cross-compilation, but I tried hermetic_cc_toolchain, and it seems to work normally (compiling linux_amd64 target on my m1 mac).

hunshcn@a52bd0f

bazel build --enable_bzlmod --announce_rc  //python:gazelle_binary --platforms @zig_sdk//platform:linux_amd64 --extra_toolchains @zig_sdk//toolchain:linux_amd64_musl

(If the target platform is darwin_amd64, we may need to make some changes to gazelle_binary. like https://github.com/uber/hermetic_cc_toolchain/blob/13b27b4e1fa461c57ef12d4e84e0ef34d0e697fb/examples/bzlmod/BUILD.bazel#L23C1-L29C8)

[linzhp]

Thanks @hunshcn for pointing out... @zig_sdk//platform:linux_amd64 works but @io_bazel_rules_go//go/toolchain:linux_amd64 doesn't, because the rules_go version disables cgo. We need to use @io_bazel_rules_go//go/toolchain:linux_amd64_cgo instead.

Due to the limitation of hermetic_cc_toolchain, cross-compiling C libraries into any darwin platform won't work. Let me update the issue.

[linzhp]

@aignas Updated the description. So the main question here is:

are we better off by replacing a Python dependency with a C dependency?

[hunshcn]

The deepest motivation for me to raise that PR is that py_binary is not hermetic.
I have to exit the conda environment to run gazelle, otherwise I will receive an error.

Maybe darwin build can be supported by uber/hermetic_cc_toolchain#10 (comment) ?

[aignas]

So I guess there are a few alternatives here:

1. Maintain two codebases where the user can chose if they want the pure C tree-sitter implementation + cgo solution or the Python interpreter solution. I personally think that this is not ideal as it makes us maintain 2 implementations.
2. Explain users how they can cross-compile with cgo, but as @linzhp suggests that is non-trivial either.
3. Revert feat(gazelle): pure golang helper #1895 until we have a better way to do the cross-compilation.

The dependence on the system python is only to launch the interpreter that is bundled with the gazelle plugin, but #1599 needs to be addressed before it can work in venv. The dependence on the system python could be also removed if we solve #691.

I am inclined to go with 3. unless there is a better way to avoid the added dependencies here. @hunshcn, do you know if there are other ways to work around this issue?

[hunshcn]

Just retell the current situation. Can't cross-compile to darwin-any, other systems are ok. Because hermetic_cc_toolchain does not support sysroot (even on macos machines). This may be a future direction, but no one provides support. I'm interested, but it's beyond my knowledge. It would be best if someone could achieve this.
If you need to compile the binary of the whole platform, you need at least one darwin_amd64 and one darwin_arm64.

I think rules_python decides how much cross-compilation should be supported. If the answer is that we need complete cross-compilation ability, then we may need revert.

1 is also an option, but it can be a user operation (I mean patch by the user himself, this part of the code is relatively simple, and I think it is feasible. If revert, I will do it.)

[aignas]

A related issue from the upstream bindings package is here: smacker/go-tree-sitter#120

This issue is a tough one, the fact that the cross-compilation does not work means that it is not entirely hermetic on all platforms. So I think we should revert the #1895 especially since #1929 is in the works.

[aignas]

As discussed in #1931, the main idea to address this would be to use https://github.com/go-python/gpython to try and use a pure go implementation to fix this issue.

Within the maintainers meeting today we discussed that this issue is affecting a small subset of rules_python users and that subset who needs to cross-compile gazelle most likely have access to required hardware or know how how to build the plugin in the mean time.

So to summarize:

• For cross-compiling gazelle plugin for windows you may need a windows machine or a properly setup CC toolchain.
• For building gazelle for linux, you will need a sysroot and a properly setup toolchain, usually doable with hermetic_cc_toolchain and others.
• Fo building gazelle for mac, you will need a mac machine.

I'll pin this issue as it is a known issue.

PR's are welcome for this.

[alexeagle]

FYI that the Aspect CLI has a configure command which is a pre-compiled Gazelle. This way end-users aren't responsible for setting up a toolchain to build a local Gazelle binary themselves. see https://docs.aspect.build/cli/commands/aspect_configure

Here's a demo for a python project, setup from scratch:
https://asciinema.org/a/663610