How do I instruct the go_sdk to use the version specified in `go.mod`? #4292

jaqx0r · 2025-03-13T09:16:40Z

I have a new dependency in a project, which has a toolchain dependency higher than the local toolchain.

When I run bazel run //rules_go:go -- mod tidy I get:

bazel run @rules_go//go -- mod tidy
Computing main repo mapping: 
Loading: 
Loading: 0 packages loaded
bazel: Entering directory `/home/jaq/.cache/bazel/_bazel_jaq/d654f9f0bd3eddeb1a4864cf68fcdd6b/execroot/_main/'
Analyzing: target @@rules_go+//go:go (0 packages loaded, 0 targets configured)
Analyzing: target @@rules_go+//go:go (0 packages loaded, 0 targets configured)

INFO: Analyzed target @@rules_go+//go:go (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
bazel: Leaving directory `/home/jaq/.cache/bazel/_bazel_jaq/d654f9f0bd3eddeb1a4864cf68fcdd6b/execroot/_main/'
Target @@rules_go+//go/tools/go_bin_runner:go_bin_runner up-to-date:
  bazel-bin/external/rules_go+/go/tools/go_bin_runner/bin/go
INFO: Elapsed time: 0.173s, Critical Path: 0.00s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Running command line: bazel-bin/external/rules_go+/go/tools/go_bin_runner/bin/go <args omitted>
go: finding module for package github.com/google/go-containerregistry/pkg/v1/layout
go: toolchain upgrade needed to resolve github.com/google/go-containerregistry/pkg/v1/layout
go: github.com/google/[email protected] requires go >= 1.23.0 (running go 1.22.7; GOTOOLCHAIN=local)

Compilation exited abnormally with code 1 at Thu Mar 13 20:10:26, duration 3.03 s

The go.mod file contains this at the top:

module github.com/jaqx0r/blts

go 1.22

toolchain go1.24.1

The hint is GOTOOLCHAIN=local in the error.

I haven't set an explicit toolchain in MODULE.bazel because I assumed it would figure it out from go.mod:

module(name = "blts")

bazel_dep(name = "rules_go", version = "0.53.0")
bazel_dep(name = "gazelle", version = "0.42.0")

go_sdk = use_extension("@rules_go//go:extensions.bzl", "go_sdk")

# https://github.com/bazel-contrib/rules_go/blob/master/go/nogo.rst
# Like golangci-lint, but integrated into the build.
go_sdk.nogo(nogo = "//:nogo")

# Update dependencies with
# `bazel run //:gazelle`.
# and then
# `bazel run @rules_go//go -- mod tidy`
go_deps = use_extension("@gazelle//:extensions.bzl", "go_deps")
go_deps.from_file(go_mod = "//:go.mod")
use_repo(go_deps, ...)

I expected the default behaviour to be hermetic, that is I would not be using the system Go SDK. BUt even that doesn't make sense:

% go version
go version go1.24.1 linux/amd64

so wherever this running go 1.22.7; GOTOOLCHAIN=local is coming from, I don't know.

slab% go env GOTOOLCHAIN 
auto

The go version is supposed to be defined in go.mod so how can I make the go_sdk.download call in MODULE.bazel read this from go.mod so I don't have to repeat myself?

The text was updated successfully, but these errors were encountered:

fmeum · 2025-03-13T12:59:08Z

The default Go SDK is downloaded hermetically, but it currently defaults to a version specified per rules_go release (see rules_go's MODULE.bazel file).

Extending Go SDK with a go_sdk.from_file(go_mod = ...) tag would be a great feature. PRs are very welcome and I would happily support anyone who picks this up. It would still require opt-in in this way as we can't just read arbitrary files without the user specifying a dependency on them.

jaqx0r · 2025-03-13T21:14:35Z

OK so calling go_sdk.from_file(go_mod = "//:go.mod) provieds the opt in mechanism.

from_file should parse the given mod using the same method as go_deps.from_file, and extract the toolchain version, and if go.mod is lacking a toolchain, the go version.

Then it should call the go_download_sdk method directly to implement the fetch of the SDK.

Does that sound correct?

fmeum · 2025-03-13T23:36:08Z

Yes, except that we should use the version we read from the file for the go_default_sdk repo, which is already backed by go_download_sdk but at the moment has a version that's manually specified by us.

This change copies parts of the `go_mod.bzl` parser from the Gazelle repository: https://github.com/bazel-contrib/bazel-gazelle/blob/master/internal/bzlmod/go_mod.bzl A test based on the `go_download_sdk` test sets up a repo with a `go.mod` and checks that the version specified in it is used to run the test in that repo-under-test. Fixes bazel-contrib#4292

jaqx0r · 2025-03-22T05:34:06Z

I have gotten most of the way there but I'm having trouble with the last step. Can you please take a look at jaqx0r@3c7ed95 and advise me how I should be overriding the default sdk?

In my test, the version reported is not the one I am expecting.

INFO: From Testing //tests/core/from_go_mod_file:from_go_mod_file_test:
==================== Test output for //tests/core/from_go_mod_file:from_go_mod_file_test:
--- FAIL: Test (9.15s)
    --- FAIL: Test/toolchain (8.27s)
        from_go_mod_file_test.go:97: Starting local Bazel server and connecting to it...
            Computing main repo mapping: 
            Computing main repo mapping: 
            Computing main repo mapping: 
            Computing main repo mapping: 
            Loading: 
            Loading: 0 packages loaded
            DEBUG: /home/jaq/.cache/bazel/_bazel_jaq/7a087131b1e4a89e6bfe21cd9cecf749/external/rules_go~/go/private/extensions.bzl:288:18: Got version 1.24.1
            DEBUG: /home/jaq/.cache/bazel/_bazel_jaq/7a087131b1e4a89e6bfe21cd9cecf749/external/rules_go~/go/private/extensions.bzl:308:10: toolchains: [struct(goarch = "", goos = "", sdk_repo = "go_sdk", sdk_type = "remote", sdk_version = "1.24.1"), struct(goarch = "", goos = "", sdk_repo = "go_default_sdk", sdk_type = "remote", sdk_version = "1.23.6"), struct(goarch = "amd64", goos = "darwin", sdk_repo = "rules_go__download_0_darwin_amd64", sdk_type = "remote", sdk_version = "1.23.6"), struct(goarch = "arm64", goos = "darwin", sdk_repo = "rules_go__download_0_darwin_arm64", sdk_type = "remote", sdk_version = "1.23.6"), struct(goarch = "arm64", goos = "linux", sdk_repo = "rules_go__download_0_linux_arm64", sdk_type = "remote", sdk_version = "1.23.6"), struct(goarch = "amd64", goos = "windows", sdk_repo = "rules_go__download_0_windows_amd64", sdk_type = "remote", sdk_version = "1.23.6"), struct(goarch = "arm64", goos = "windows", sdk_repo = "rules_go__download_0_windows_arm64", sdk_type = "remote", sdk_version = "1.23.6")]
            Analyzing: target //:version_test (1 packages loaded, 0 targets configured)
            Analyzing: target //:version_test (1 packages loaded, 0 targets configured)
            
            Analyzing: target //:version_test (91 packages loaded, 918 targets configured)
            
            INFO: Analyzed target //:version_test (107 packages loaded, 5383 targets configured).
            FAIL: //:version_test (see /home/jaq/.cache/bazel/_bazel_jaq/7a087131b1e4a89e6bfe21cd9cecf749/execroot/_main/bazel-out/k8-fastbuild/testlogs/version_test/test.log)
            [15 / 16] 1 / 1 tests, 1 failed; Testing //:version_test; 0s linux-sandbox
            INFO: From Testing //:version_test:
            INFO: Found 1 test target...
            Target //:version_test up-to-date:
              bazel-bin/version_test_/version_test
            INFO: Elapsed time: 8.158s, Critical Path: 0.21s
            INFO: 3 processes: 2 internal, 1 linux-sandbox.
            INFO: Build completed, 1 test FAILED, 3 total actions
            exit status 3
    --- FAIL: Test/go_only (0.89s)
        from_go_mod_file_test.go:97: Computing main repo mapping: 
            Loading: 
            Loading: 0 packages loaded
            DEBUG: /home/jaq/.cache/bazel/_bazel_jaq/7a087131b1e4a89e6bfe21cd9cecf749/external/rules_go~/go/private/extensions.bzl:288:18: Got version 1.23.0
            DEBUG: /home/jaq/.cache/bazel/_bazel_jaq/7a087131b1e4a89e6bfe21cd9cecf749/external/rules_go~/go/private/extensions.bzl:308:10: toolchains: [struct(goarch = "", goos = "", sdk_repo = "go_sdk", sdk_type = "remote", sdk_version = "1.23.0"), struct(goarch = "", goos = "", sdk_repo = "go_default_sdk", sdk_type = "remote", sdk_version = "1.23.6"), struct(goarch = "amd64", goos = "darwin", sdk_repo = "rules_go__download_0_darwin_amd64", sdk_type = "remote", sdk_version = "1.23.6"), struct(goarch = "arm64", goos = "darwin", sdk_repo = "rules_go__download_0_darwin_arm64", sdk_type = "remote", sdk_version = "1.23.6"), struct(goarch = "arm64", goos = "linux", sdk_repo = "rules_go__download_0_linux_arm64", sdk_type = "remote", sdk_version = "1.23.6"), struct(goarch = "amd64", goos = "windows", sdk_repo = "rules_go__download_0_windows_amd64", sdk_type = "remote", sdk_version = "1.23.6"), struct(goarch = "arm64", goos = "windows", sdk_repo = "rules_go__download_0_windows_arm64", sdk_type = "remote", sdk_version = "1.23.6")]
            Analyzing: target //:version_test (0 packages loaded, 0 targets configured)
            Analyzing: target //:version_test (0 packages loaded, 0 targets configured)
            
            INFO: Analyzed target //:version_test (1 packages loaded, 870 targets configured).
            FAIL: //:version_test (see /home/jaq/.cache/bazel/_bazel_jaq/7a087131b1e4a89e6bfe21cd9cecf749/execroot/_main/bazel-out/k8-fastbuild/testlogs/version_test/test.log)
            INFO: From Testing //:version_test:
            INFO: Found 1 test target...
            Target //:version_test up-to-date:
              bazel-bin/version_test_/version_test
            INFO: Elapsed time: 0.864s, Critical Path: 0.06s
            INFO: 2 processes: 1 internal, 1 linux-sandbox.
            INFO: Build completed, 1 test FAILED, 2 total actions
            exit status 3
FAIL
================================================================================

and the go_only test's test.log:

exec ${PAGER:-/usr/bin/less} "$0" || exit 1
Executing tests from //:version_test
-----------------------------------------------------------------------------
go1.24.0 X:nocoverageredesign
--- FAIL: Test (0.00s)
    version_test.go:15: got version "go1.24.0 X:nocoverageredesign"; want "go1.23.0"
FAIL

jaqx0r · 2025-03-28T06:07:49Z

@fmeum Could you take a look at the commit above please, when you get a moment, and point me in the right direction?

fmeum · 2025-03-28T09:09:48Z

Now that I think about it, you probably don't need to actually replace go_default_sdk. Since the tags of the root module are interpreted first, its toolchains will also be picked first.

But I would recommend to move the logic above the handling of the download tag and synthesize one instead. That way you benefit from the existing logic for that tag that adds SDKs for all common execution platforms.

jaqx0r · 2025-03-29T01:22:51Z

Thanks @fmeum. I'm still missing something though, because now it looks like I correctly set the first_host_compatible_toolchain to be my toolchain selected from my from_file tag, but it isn't the one being used to build and run the test code, which is the same problem in the output above.

I've uploaded a new commit, with some extra debug statements printing out state.

master...jaqx0r:rules_go:from-go-mod-file

with the test output:

bazel test //tests/core/from_go_mod_file/...
Computing main repo mapping: 
Loading: 
Loading: 0 packages loaded
bazel: Entering directory `/home/jaq/.cache/bazel/_bazel_jaq/906827e386e14cc47d13c9784c62deb4/execroot/io_bazel_rules_go/'
Analyzing: target //tests/core/from_go_mod_file:from_go_mod_file_test (0 packages loaded, 0 targets configured)
Analyzing: target //tests/core/from_go_mod_file:from_go_mod_file_test (0 packages loaded, 0 targets configured)

INFO: Analyzed target //tests/core/from_go_mod_file:from_go_mod_file_test (0 packages loaded, 0 targets configured).
[10 / 11] Testing //tests/core/from_go_mod_file:from_go_mod_file_test; 1s local
FAIL: //tests/core/from_go_mod_file:from_go_mod_file_test (see /home/jaq/.cache/bazel/_bazel_jaq/906827e386e14cc47d13c9784c62deb4/execroot/io_bazel_rules_go/bazel-out/k8-fastbuild/testlogs/tests/core/from_go_mod_file/from_go_mod_file_test/test.log)
[10 / 11] 1 / 1 tests, 1 failed; Testing //tests/core/from_go_mod_file:from_go_mod_file_test; 8s local
INFO: From Testing //tests/core/from_go_mod_file:from_go_mod_file_test:
==================== Test output for //tests/core/from_go_mod_file:from_go_mod_file_test:
--- FAIL: Test (7.43s)
    --- FAIL: Test/toolchain (7.43s)
        from_go_mod_file_test.go:97: Starting local Bazel server and connecting to it...
            Computing main repo mapping: 
            Computing main repo mapping: 
            Computing main repo mapping: 
            Computing main repo mapping: 
            Loading: 
            Loading: 0 packages loaded
            DEBUG: /home/jaq/.cache/bazel/_bazel_jaq/7a087131b1e4a89e6bfe21cd9cecf749/external/rules_go~/go/private/extensions.bzl:176:18: Got version 1.24.1
            DEBUG: /home/jaq/.cache/bazel/_bazel_jaq/7a087131b1e4a89e6bfe21cd9cecf749/external/rules_go~/go/private/extensions.bzl:323:10: toolchains: [struct(goarch = "", goos = "", sdk_repo = "main___from_file_0", sdk_type = "remote", sdk_version = "1.24.1"), struct(goarch = "", goos = "", sdk_repo = "go_default_sdk", sdk_type = "remote", sdk_version = "1.23.6"), struct(goarch = "amd64", goos = "darwin", sdk_repo = "rules_go__download_0_darwin_amd64", sdk_type = "remote", sdk_version = "1.23.6"), struct(goarch = "arm64", goos = "darwin", sdk_repo = "rules_go__download_0_darwin_arm64", sdk_type = "remote", sdk_version = "1.23.6"), struct(goarch = "arm64", goos = "linux", sdk_repo = "rules_go__download_0_linux_arm64", sdk_type = "remote", sdk_version = "1.23.6"), struct(goarch = "amd64", goos = "windows", sdk_repo = "rules_go__download_0_windows_amd64", sdk_type = "remote", sdk_version = "1.23.6"), struct(goarch = "arm64", goos = "windows", sdk_repo = "rules_go__download_0_windows_arm64", sdk_type = "remote", sdk_version = "1.23.6")]
            DEBUG: /home/jaq/.cache/bazel/_bazel_jaq/7a087131b1e4a89e6bfe21cd9cecf749/external/rules_go~/go/private/extensions.bzl:324:10: first : @main___from_file_0//:ROOT
            Analyzing: target //:version_test (1 packages loaded, 0 targets configured)
            Analyzing: target //:version_test (1 packages loaded, 0 targets configured)
            
            Analyzing: target //:version_test (91 packages loaded, 918 targets configured)
            
            INFO: Analyzed target //:version_test (107 packages loaded, 5383 targets configured).
            FAIL: //:version_test (see /home/jaq/.cache/bazel/_bazel_jaq/7a087131b1e4a89e6bfe21cd9cecf749/execroot/_main/bazel-out/k8-fastbuild/testlogs/version_test/test.log)
            INFO: From Testing //:version_test:
            INFO: Found 1 test target...
            Target //:version_test up-to-date:
              bazel-bin/version_test_/version_test
            INFO: Elapsed time: 7.335s, Critical Path: 0.15s
            INFO: 3 processes: 2 internal, 1 linux-sandbox.
            INFO: Build completed, 1 test FAILED, 3 total actions
            exit status 3
FAIL

and the content of the inner test's log:

exec ${PAGER:-/usr/bin/less} "$0" || exit 1
Executing tests from //:version_test
-----------------------------------------------------------------------------
go1.24.0 X:nocoverageredesign
--- FAIL: Test (0.00s)
    version_test.go:15: got version "go1.24.0 X:nocoverageredesign"; want "go1.24.1"
FAIL

So you can see in extensions.bzl:176 we've correctly read the toolchain 1.24.1 from the go.mod in test.

The full set of toolchains then printed out has our from_file toolchain first, and the first_host_compatible_toolchain has the correct sdk_repo prefix.

But when the inner test executes, it's runtime is reported as 1.24.0, not 1.24.1. Have I messed up the test?

jaqx0r · 2025-03-29T03:43:03Z

Curiously, this is working correctly when I use local_repo_override to use my rules_go changes from another project.

in MODULE.bazel

bazel_dep(name = "rules_go", version = "0.53.0")
local_path_override(
    module_name = "rules_go",
    path = "/home/jaq/src/rules_go",
)

at the top of go.mod:

module github.com/jaqx0r/blts

go 1.23.0

toolchain go1.23.0

then

% bazel build //...

...
[614 / 1,107] GoToolchainBinaryBuild external/rules_go++go_sdk+blts__from_file_0/builder [for tool]; 2s linux-sandbox
...

looks promising

% go version bazel-bin/cmd/s/s_/s
bazel-bin/cmd/s/s_/s: go1.23.0 X:nocoverageredesign

and changing the go.mod toolchain to 1.24.1 and another build:

slab% go version bazel-bin/cmd/s/s_/s
bazel-bin/cmd/s/s_/s: go1.24.1 X:nocoverageredesign

which is pretty conclusive to me.

So I've definitely messed up the test. I suspect it's because I've set it up as a bzlmod workspace, and the go_default_sdk_test setup uses WORKSPACE files; the output of the test is always something like:

-----------------------------------------------------------------------------
go1.24.0 X:nocoverageredesign
--- FAIL: Test (0.00s)
    version_test.go:15: got version "go1.24.0 X:nocoverageredesign"; want "go1.17"
FAIL

which really looks like the Go SDK from the rules_go workspace is leaking into the test workspace.

jaqx0r · 2025-03-29T09:54:16Z

Some more spelunking in the innards of the test sandbox with --sandbox_debug I think that

rules_go/go/tools/bazel_testing/bazel_testing.go

Line 520 in eef92f5

root_file = "@local_go_sdk//:ROOT",

is tripping me up.

I copied what go_download_sdk_test.go was doing to set the toolchain name in the test invocation, and now I'm getting good test outcomes, but it feels like a hack. I've added a note in the commit as such.

This change copies parts of the `go_mod.bzl` parser from the Gazelle repository: https://github.com/bazel-contrib/bazel-gazelle/blob/master/internal/bzlmod/go_mod.bzl A test based on the `go_download_sdk` test sets up a repo with a `go.mod` and checks that the version specified in it is used to run the test in that repo-under-test. Fixes bazel-contrib#4292

jaqx0r linked a pull request Mar 29, 2025 that will close this issue

Add go_sdk.from_file to read the SDK version from go.mod. #4305

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

How do I instruct the go_sdk to use the version specified in `go.mod`? #4292

How do I instruct the go_sdk to use the version specified in `go.mod`? #4292

jaqx0r commented Mar 13, 2025 •

edited

Loading

fmeum commented Mar 13, 2025

jaqx0r commented Mar 13, 2025

fmeum commented Mar 13, 2025

jaqx0r commented Mar 22, 2025 •

edited

Loading

jaqx0r commented Mar 28, 2025

fmeum commented Mar 28, 2025

jaqx0r commented Mar 29, 2025

jaqx0r commented Mar 29, 2025 •

edited

Loading

jaqx0r commented Mar 29, 2025

How do I instruct the go_sdk to use the version specified in go.mod? #4292

How do I instruct the go_sdk to use the version specified in go.mod? #4292

Comments

jaqx0r commented Mar 13, 2025 • edited Loading

fmeum commented Mar 13, 2025

jaqx0r commented Mar 13, 2025

fmeum commented Mar 13, 2025

jaqx0r commented Mar 22, 2025 • edited Loading

jaqx0r commented Mar 28, 2025

fmeum commented Mar 28, 2025

jaqx0r commented Mar 29, 2025

jaqx0r commented Mar 29, 2025 • edited Loading

jaqx0r commented Mar 29, 2025

How do I instruct the go_sdk to use the version specified in `go.mod`? #4292

How do I instruct the go_sdk to use the version specified in `go.mod`? #4292

jaqx0r commented Mar 13, 2025 •

edited

Loading

jaqx0r commented Mar 22, 2025 •

edited

Loading

jaqx0r commented Mar 29, 2025 •

edited

Loading