reboot wit-bindgen-go

[dicej]

This re-introduces Go support, with a new implementation written from scratch. While the previous incarnation targeted TinyGo and reused the C generator for lifting and lowering, this one targets "big" Go and handles lifting and lowering itself. In addition, it supports idiomatic, goroutine-based concurrency on top of the component model async ABI, including streams and futures.

This implementation is also distinct from the
go-modules project, which has a number of limitations (e.g. no async support, not safe for use with "big" Go) and is no longer being actively maintained.

Note that the async support currently requires a small patch to the Go runtime library. I plan to work with the upstream project to make that patch unecessary in the future. Components that don't use async features should work with stock, unpatched Go.

One of the tricky parts about lowering values and passing pointers to e.g. {stream,future}.{read,write} is that we must tell the Go garbage collector to pin the pointers (i.e. mark the pointee as both immovable and uncollectable) for as long as the host has access to them, but then unpin them promptly afterward to avoid leaks. We do this using runtime.Pinner instances, most of which are locally scoped and easy to reason about. However, we must use a couple of globally-scoped pinners as well: one for cabi_realloc allocations by the host when it lowers parameters (which we unpin after lifting those parameters) and another for returning values from sync-lifted exports (which we unpin in post-return functions).

There are a couple of known missing features which I'll open GitHub issues for:

1. Resource handles are not being restored for unwritten items in the case of an incomplete stream or future write.
2. Importing and/or exporting multiple versions of the same package will cause name clashes.

In addition, I plan to expand the test coverage beyond the basics covered in this commit.

[dicej]

I just realized I need to make CI run the go tests. I'll push an update.

[alexcrichton]

Agreed we'll want to ge this running on CI, but overall looking great! I only skimmed codegen though and would mostly lean on tests to exercise bits and bobs

[dicej]

I was so focused on the runtime tests that I never ran the codegen ones. Looks like I have more work to do. I'll switch this to draft mode while I do that.

[ydnar]

Thanks to @dgryski for pointing me to this!

@dicej this looks great. We solved the multiple versions issue by optionally generating a versioned Go package, e.g. wasi/http/v0.2.1.

Does this require the WASI 0.1 adapter to function in big Go?

[dicej]

@dicej this looks great. We solved the multiple versions issue by optionally generating a versioned Go package, e.g. wasi/http/v0.2.1.

Good idea. Currently we generate package names like wasi_http_0_2_1_types because I'm a Go novice, but versioning the Go package makes more sense.

Does this require the WASI 0.1 adapter to function in big Go?

Yes, currently. I've opened golang/go#76775 as a baby step towards adding GOOS=wasip3 upstream. Meanwhile, you can use any p2 or p3 features you like by generating bindings from the WASI WIT files and using them directly.

BTW, here's an example that shows off concurrent outbound requests, etc. Eventually we'll want to do something like you did here to make that use net/http interfaces, etc.