Support BuildKit

[jshbrntt]

Would be nice to provide an easy way to enable BuildKit for faster and more efficient builds.

It's as easy as setting DOCKER_BUILDKIT=1 before using the Docker CLI, not sure what is required from the perspective of the Docker API usage.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Open a new issue if needed.

[mschout]

Realize this is old, but seems current situation is that the HTTP API does not support buildkit, even in 2022 :(

[cristianrgreco]

https://docs.docker.com/build/buildkit/

BuildKit is the default builder for users on Docker Desktop, and Docker Engine as of version 23.0.

This issue should be re-opened. Users are failing to build images using BuildKit features: testcontainers/testcontainers-node#571

[apocas]

docker/for-linux#1136 :(

Didn't have the time yet to look at Docker CLI source code to check how is CLI using it.

[mikeseese]

This is no longer an issue. dockerode inherently supports BuildKit by providing the version: "2" build option:

const docker = require("dockerode");
docker.buildImage({...}, { version: "2" })

The support for the version option was added in API v1.38: moby/moby@0c3b8cc

If I get a chance, I'll update the DefinitelyTyped package

[pimterry]

@mikeseese Have you tested that? I think there's quite a lot more to this than just sending a different version in the request unfortunately. The protocol to actually run the build is completely different, and as far as I know there's no kind of compatibility layer or similar - we'll need to fully implement the new protocol.

Under the hood, non-BuildKit builds just send a single HTTP request with all files and the build options, and then stream the raw build output back. BuildKit builds meanwhile just start a session with initial options, then uses gRPC over HTTP/2 to do two-way communication throughout the build. Files aren't sent up front: instead the server requests each file it needs from the client on demand, and similarly injects secrets and proxies remote SSH authentication from the client over the connection. It's not just a flag to enable.

For example, try building this minimal Buildkit Dockerfile:

# syntax=docker/dockerfile:1
FROM bash
RUN echo 'hello' > /tmp/hello

Running this:

let Dockerode = require('dockerode')
let d = new Dockerode()
d.buildImage({ context: process.cwd(), src: ['Dockerfile'] }, { version: '2' }, (e, res) => res.pipe(process.stdout))

Prints:

{"id":"moby.buildkit.trace","aux":"Cm8KR3NoYTI1Njo1NTNiMmZmZTZjOTM2ZDRlYjczNGY0ZjRkZTRlZWU4N2M0YWRiN2Y0NjlkYjk0YTQzZGM3NTkzNzhhYWVkNGU3GiRbaW50ZXJuYWxdIGxvYWQgcmVtb3RlIGJ1aWxkIGNvbnRleHQ="}
{"id":"moby.buildkit.trace","aux":"Cn0KR3NoYTI1Njo1NTNiMmZmZTZjOTM2ZDRlYjczNGY0ZjRkZTRlZWU4N2M0YWRiN2Y0NjlkYjk0YTQzZGM3NTkzNzhhYWVkNGU3GiRbaW50ZXJuYWxdIGxvYWQgcmVtb3RlIGJ1aWxkIGNvbnRleHQqDAjl5aCzBhDp2aulAQ=="}
{"id":"moby.buildkit.trace","aux":"CosBCkdzaGEyNTY6NTUzYjJmZmU2YzkzNmQ0ZWI3MzRmNGY0ZGU0ZWVlODdjNGFkYjdmNDY5ZGI5NGE0M2RjNzU5Mzc4YWFlZDRlNxokW2ludGVybmFsXSBsb2FkIHJlbW90ZSBidWlsZCBjb250ZXh0KgwI5eWgswYQ6dmrpQEyDAjl5aCzBhCBjeDFAQ=="}
{"id":"moby.buildkit.trace","aux":"CosBCkdzaGEyNTY6NTUzYjJmZmU2YzkzNmQ0ZWI3MzRmNGY0ZGU0ZWVlODdjNGFkYjdmNDY5ZGI5NGE0M2RjNzU5Mzc4YWFlZDRlNxokW2ludGVybmFsXSBsb2FkIHJlbW90ZSBidWlsZCBjb250ZXh0KgwI5eWgswYQzZP1xQEyDAjl5aCzBhDs1PjFAQ=="}
{"id":"moby.buildkit.trace","aux":"CqMBCkdzaGEyNTY6YWM2MjQ0YzMyMGNjNTI5NTRjMDAyYTM0Njg1YzUwNjlkM2RhNGMyNTYzZjUyNGQwNTY1ZWM0OTUxNTk3NDJmZRJHc2hhMjU2OjU1M2IyZmZlNmM5MzZkNGViNzM0ZjRmNGRlNGVlZTg3YzRhZGI3ZjQ2OWRiOTRhNDNkYzc1OTM3OGFhZWQ0ZTcaD2NvcHkgL2NvbnRleHQgLw=="}
{"id":"moby.buildkit.trace","aux":"CrEBCkdzaGEyNTY6YWM2MjQ0YzMyMGNjNTI5NTRjMDAyYTM0Njg1YzUwNjlkM2RhNGMyNTYzZjUyNGQwNTY1ZWM0OTUxNTk3NDJmZRJHc2hhMjU2OjU1M2IyZmZlNmM5MzZkNGViNzM0ZjRmNGRlNGVlZTg3YzRhZGI3ZjQ2OWRiOTRhNDNkYzc1OTM3OGFhZWQ0ZTcaD2NvcHkgL2NvbnRleHQgLyoMCOXloLMGELvFjsoB"}
{"id":"moby.buildkit.trace","aux":"Cr8BCkdzaGEyNTY6YWM2MjQ0YzMyMGNjNTI5NTRjMDAyYTM0Njg1YzUwNjlkM2RhNGMyNTYzZjUyNGQwNTY1ZWM0OTUxNTk3NDJmZRJHc2hhMjU2OjU1M2IyZmZlNmM5MzZkNGViNzM0ZjRmNGRlNGVlZTg3YzRhZGI3ZjQ2OWRiOTRhNDNkYzc1OTM3OGFhZWQ0ZTcaD2NvcHkgL2NvbnRleHQgLyoMCOXloLMGELvFjsoBMgwI5eWgswYQzPKy6gE="}
{"id":"moby.buildkit.trace","aux":"Co8BCkdzaGEyNTY6YWMwNzJkNTIxOTAxMjIyZWVlZjU1MGY1MjI4Mjg3N2YxOTZlMTZiMDI0Nzg0NGJlOWNlYjFjY2MxZWFjMzkxZBo2cmVzb2x2ZSBpbWFnZSBjb25maWcgZm9yIGRvY2tlci5pby9kb2NrZXIvZG9ja2VyZmlsZToxKgwI5eWgswYQ07bR7gE="}
{"id":"moby.buildkit.trace","aux":"CrEBCkdzaGEyNTY6YWMwNzJkNTIxOTAxMjIyZWVlZjU1MGY1MjI4Mjg3N2YxOTZlMTZiMDI0Nzg0NGJlOWNlYjFjY2MxZWFjMzkxZBo2cmVzb2x2ZSBpbWFnZSBjb25maWcgZm9yIGRvY2tlci5pby9kb2NrZXIvZG9ja2VyZmlsZToxKgwI5eWgswYQ07bR7gEyDAjl5aCzBhD1ytPGAzoSbm8gYWN0aXZlIHNlc3Npb25z"}
{"errorDetail":{"message":"no active sessions"},"error":"no active sessions"}

Sending the new version does switch to the buildkit protocol, but Dockerode doesn't know how to speak that protocol, so never successfully does anything. Even though this case isn't using any buildkit features, the build fails, and no image is created.

[mikeseese]

@apocas I was able to create a reproduction repo that will successfully build an image with COPY --chmod feature given the version flag: https://github.com/mikeseese/dockerode-buildkit

Removing the version flag will fail saying the chmod feature requires BuildKit but keeping the flag does indeed give me an image has

[mikeseese]

The no active sessions does mean that there isn't complete support from dockerode, but this specific scenario might allow you to build fine after doing a docker pull bash

[Hiekki4]

Is it possible to use a remote builder for the build?

[pimterry]

Is it possible to use a remote builder for the build?

You mean using a remote Docker engine, or using a remote Buildkitd daemon directly? The former will work (just as with any Dockerode command - local or remote is the same) but the latter almost certainly won't (https://github.com/moby/buildkit#expose-buildkit-as-a-tcp-service suggests the daemon's direct API only speaks its own gRPC API, while Dockerode is currently just uses plain HTTP with the Docker Engine API).

Or do you mean something else? If you have a concrete example command/environment that'd be useful.

[pimterry]

The no active sessions does mean that there isn't complete support from dockerode, but this specific scenario might allow you to build fine after doing a docker pull bash

@mikeseese Nice example! Yes, I can confirm that minimal case works, which is interesting. It seems that in that case you have managed to avoid everything that might require an active session, neat.

To list them explicitly, the limitations I can see that do require a session (and so are currently not supported by Dockerode, or any implementation that just changes the version without supporting the full protocol) are:

• Adding any syntax declaration (e.g. #syntax=docker/dockerfile:1 - part of the problem for the failing example above). Note that this is very common, e.g. Docker's current getting started example and every example in their best practices (https://docs.docker.com/build/building/best-practices/) includes this.
• Pulling any dependencies (i.e. FROM bash won't work, unless it's already pre-pulled) or fetch any files on-demand (e.g. if you don't explicitly include hello.txt in the src option, it fails, though a big part of the benefit of Buildkit is avoiding that to speed up builds dramatically)
• Using any other session-backed features (secrets, SSH auth, dynamic file loading, cache sharing, and probably various other features)

Also, the build output consists of JSON with base64'd content in the aux field, which then seems to be protobuf-encoded (best guess) with other metadata too, so that will need some processing to give you anything useful.

In cases where you can avoid the above features though and you only need the final image id (not the streamed build output) then just upgrading the version as above should work OK, so that might indeed be a workaround for some setups.

I think going further though, it looks quite achievable to fully support the core Buildkit featureset, so that most normal builds do work without many constraints and give us the main benefits of Buildkit, and it'd be good to somehow aim for that. AFAICT the main steps required are:

• Creating an initial Buildkit session when starting a build. That requires an HTTP/2 negotiation - the way this works is with an HTTP upgrade flow, where the connection changes from HTTP/1 to HTTP/2 and flips direction (!) so the client starts acting as a server (and so allowing dynamic requests for data) & vice-versa.
• Implement the minimal gRPC APIs for the BuildKit features we want to support. Imo, enough to do a normal build of any Dockerfile using the standard instruction set would be a good start (enough to build any typical image, but leaving secrets/delegated auth/advanced cache setups for later). I think the API definitions are all proto files in the Buildkit repo, e.g. https://github.com/moby/buildkit/blob/master/session/filesync/filesync.proto defines the API used to dynamically fetch files from the context. Most of these aren't too complicated, but it's not clear up front which ones are required without incremental testing & implementation.
• Post-process the output stream, to provide a more usable output with per-image streams (note that Buildkit can build images in parallel) plus keeping easy access to the other data, like the final image id.

I'm definitely interested in this, and I've got some outlines of the initial session negotiation logic working already, but I'm unlikely to have time to fully implement everything myself for the next couple of months at least. If anybody is keen to look at that in the meantime though then I'd be very happy to provide some pointers, just let me know.

[pipex]

Implement the minimal gRPC APIs for the BuildKit features we want to support.

@pimterry in my minimal exploration of this I found that the gprc-js package was missing a feature that exists in the Go API to bind a GPRC server to an existing socket https://github.com/grpc/grpc-node/blob/master/packages/grpc-js/src/server.ts#L806

However this might be changing with v1.11 of gprc-js according to this grpc/grpc-node#2675 so that might be the final piece needed to implement buildkit support into dockerode

[schummar]

I was just able to create a POC for this. Happy for any feedback and improvements: #766