Mage is inserting indirect dependencies into the projects go.mod which the project itself will never call (even indirectly)

[owenwaller]

Bug Description

Adding the github.com/magefile/mage/sh package into a magefile causes github.com/magefile/mage v1.15.0 to be added as an indirect dependency in the projects go.mod.

But github.com/magefile/mage/sh is only required by the magefile itself. It's got nothing to do with, and will never be called by (either directly or indirectly) the project code itself.

Is there a way to give a mage based build it's own go.mod for just the magefile imports?

The mage.go magefile we are using is in a local magefiles directory.

What did you do?

Hopefully these shell lines should be sufficient to reproduce the issue.

$ mkdir mage-dummy
$cd mage-dummy/
$go mod init example.com/mage_build
$code main.go             # code in the cat below
$cat main.go 
package main

import "fmt"

func main() {
	fmt.Println("Hello World")
}

$cat go.mod 
module example.com/mage_build

go 1.22.1

$mkdir magefiles
$code magefiles/mage.go
$cat magefiles/mage.go
//go:build mage

package main

import (
	"fmt"

	"github.com/magefile/mage/sh"
)

func Build() error {
	fmt.Println("Mage Hello")
	return sh.Run("go", "install", "example.com/mage_build")
	return nil
}

$mage build
mage.go:8:2: no required module provides package github.com/magefile/mage/sh; to add it:
	go get github.com/magefile/mage/sh
Error: error compiling magefiles
$go get github.com/magefile/mage/sh
$cat go.mod
module example.com/mage_build

go 1.22.1

require github.com/magefile/mage v1.15.0 // indirect //<---- Why? This is the example.com/mage_build go.mod

What did you expect to happen?

The project's go.mod should be untouched. If mage needs a go.mod file then it should generate/update a private one (used only for the mage build) and not any project level go.mod.

Just to be clear I mean the projects (or project level) go.mod file to be the one at the root of the projects repository. So what Go would take as the module root IIRC.

What actually happened?

The project level go,mod was updated with an indirect dependency that will never be called directly or indirectly by the project itself.

Environment

$mage --version
Mage Build Tool v1.15.0-5-g2385abb
Build Date: 2024-04-26T17:53:06+01:00
Commit: 2385abb
built with: go1.22.1

$ go env # with some necessary redaction...
GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/home/XXX/.cache/go-build'
GOENV='/home/XXX/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/XXX/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/xxxx/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/home/xxx/sdk/go1.22.1'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/home/xxx/sdk/go1.22.1/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.22.1'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/tmp/mage-dummy/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build384345434=/tmp/go-build -gno-record-gcc-switches'

Additional context

A bit of background information....

Over at the vugu project we have been trying to update the build process for both maintainers and contributors, with a view to:

• We need to keep all of this cross platform, especially Windows.
• Having the same experience locally as we do via the Github Action CI/CD runs. So basically the same tooling, executing in the same sequence (as far as possible).
• Simplifying the wasm-test-suite tests that the project uses. Currently this involves two docker containers that have to be networked together, such that a locally running go test connects to one container (which has a chrome headless browser running in it) that then in turn connects to the second container (running nginx) that is serving the wasm object. We need to pass the generated wasm though a browser to see if the effect of the wasm is what we expect after the browser has executed it. We can't use a wasm runtime because the generated code will call the browsers DOM manipulation JS API's. The wasm code is never intended to run on a server via a JS runtime,

We've experimented with using Taskfile but basically run into a dead end with it. it's just to difficult to get to to loop over results from a command and writing shell in a YAML with added Go templates which is pretty painful experience.

We can't go for a simple shell script because we need this to be cross platform. Ditto a Makefile.

mage looks like it might be a very good fit for us as any small parts we need to be cross platform (e..g. removing all the generated files under a directory tree) we can easily rewrite in Go. But, still benefit for mage task dependency and better shell integration etc. It also looks like less code for us.

But at the minute we are seeing dependencies being inserted into the vugu projects go.mod file that vugu itself or and of the vugu commands (vugugen vugufmt`) will never ever call (directly or indirectly).

We'd really like to keep the Magefile(s) in the magefiles directory, so the vugu code and the build scrips are all in the same repo.

Is this just something we have to live with, or can this be fixed?

Thanks

[natefinch]

I think if you make the magefiles directory into its own go module (by running go mod init github.com/vugu/vugu/magefiles in that directory) then its dependencies won't creep into the main project's dependencies. And I don't think it'll change mage's behavior. I haven't tried it, but I think that should work.

Report back how it goes or ping me on the #mage channel on the gophers slack. I'd like to know if you can get it working, so we can make it a recommendation for people who want to keep the dependencies of their main project clean (which I totally understand).

[owenwaller]

@natefinch

Many thanks for the suggestion. I've just tried it and it does indeed work. Brilliant.

The critical part is to execute both go mod init inside the magefiles directory but also any go get ... commands to add any mage only dependencies inside the magefiles directory.

After that a simple mage ... will work.

Can we add this tip into the mage documentation? If you know where you would want this, I am happy to create a PR based on the above example.

By way of an example, following on from my original question.

$ cd magefiles # so from within the magefiles directory
$ go mod init example.com/mage_build/magefiles
$ go get github.com/magefile/mage/sh
$cat go.mod
module example.com/mage_build/magefiles

go 1.22.1

require github.com/magefile/mage v1.15.0 // indirect

$cd .. # to get back put of the magefiles directory
$cat go.mod # to check the project level go.mod is untouched
module example.com/mage_build

go 1.22.1
# now build as normal
$mage build
Mage Hello # this is the expected output given the magefile above
$mage_build # run the executable given the main.go above
Hello World