investigate using Go's -overlay flag to replace source files

[mvdan]

$ go help build | grep -A10 -- '-overlay file'
	-overlay file
		read a JSON config file that provides an overlay for build operations.
		The file is a JSON struct with a single field, named 'Replace', that
		maps each disk file path (a string) to its backing file path, so that
		a build will run as if the disk file path exists with the contents
		given by the backing file paths, or as if the disk file path does not
		exist if its backing file path is empty. Support for the -overlay flag
		has some limitations: importantly, cgo files included from outside the
		include path must be in the same directory as the Go package they are
		included from, and overlays will not appear when binaries and tests are
		run through go run and go test respectively.

This feature was mainly added for gopls, for the sake of supporting the use of commands like go list or go test on Go packages which are modified on an editor but not yet written to disk.

We currently write obfuscated files to temporary directories, and then replace the original paths fed to the toolchain with those temporary paths. This has worked OK for the most part, but it is also somewhat tricky. Here are some edge cases we've run into:

• Sometimes, the temporary paths are longer, and that can make the commands too long like in "filename or extension is too long" when building many source files #572.
• It appears that header files are read by the toolchain at multiple stages of a build, so it is hard to properly replace the files early enough. I have so far failed to resolve obfuscated names in header files included from assembly fail at link time #553 for this reason; it appears like the obfuscated build fails the first time, but the second time (when the cache kicks in) then it works.
• The toolchain sometimes hard-codes that source files must be in certain directories. We have so far worked around those restrictions carefully, but it would be better to not have to.

However, there is also one huge roadblock to using -overlay: it currently requires that we supply all file replacements upfront. Meaning that we would have to first load all the Go packages in a build, obfuscate them all at once, then perform the obfuscated build with one big -overlay JSON map. Which is certainly possible, but could also make some larger builds significantly slower due to the loss of incremental builds. Each of the three steps (initial load, obfuscation, obfuscated build) could itself be incremental and parallelized, but splitting the work into three steps would slow down many builds.

I think this is likely a dead end, given that we want to interactively and progressively obfuscate Go packages, not all of them at once, and that's not how -overlay works. But I thought laying out my thoughts here would be useful. I'll probably close this issue in a few weeks assuming I didn't miss anything.

We could request upstream to provide a form of -overlay that allows us to reactively provide file overlays as a build progresses, but I doubt that would be considered: it feels very niche, and someone who wants that much power can likely just use -toolexec like we do.

[mvdan]

Closing per the above. It was interesting to think about whether we could use this flag, but I don't think we can without severely compromising on performance.