feat: support local development using Buck2 #8504
Conversation
bolinfest
force-pushed
the
pr8502
branch
8 times, most recently
from
295b531 to
ec91a29
Compare
|
@bolinfest Hi! I was pointed to this PR by the bazel community, some folks were a bit surprised to see this as Openai is generally using Bazel internally for many projects :) I don't want to derail the experiments, but I was also curious why not use Bazel, given the feature gaps you mentioned in Buck?
We can do this in Bazel from day 1 using Buildbuddy free tier :)
Last time I checked, Buck2 did not support any sandboxing for local execution, only remote. Has that been fixed?
There's a slightly different approach with Bazel - it integrates with Cargo.lock/Cargo.toml and generates external repositories with BUILD files. I maintain a ruleset that does this. It sounds like this was the missing piece you may have been unaware of? The advantage of using Bazel is that it seems like the rust/cargo support may be more mature - rules_rs and rules_rust go through a lot of effort to match cargo's build script execution environment very closely, which should make the reindeer fixups in this PR unnecessary. If you're open to it, I'm happy to prototype a Bazel integration in parallel to this Buck one - it should support test caching and remote build/test out-of-the-box, and I'm hopeful it will also support remote cross-builds to simplify the release story (though it depends on the complexity of the dep tree, we'll see). WDYT? |
|
@dzbarsky Hello! Honestly, I decided to try Buck2 first because I have personal contacts with the Buck2 and Reindeer folks (incidentally, I also created Buck "1"). Also, Buck2 is written in Rust, so I would certainly expect its Rust build rules to be performant. I'm not keen to require folks to install a JVM to be able to run Bazel. As you can see in this PR, the DotSlash file for
I haven't looked, tbh. Though what sort of "sandboxing" does it leverage? Is it using micro-VMs/containers or something else?
External repositories? That sounds a bit complicated...
If you're willing to give it a shot, I'm certainly interested! For sure, distributed builds and caching [however we get it] should materially speed things up for both iterative development and CI jobs. |
I mean build rules are starlark, so it comes down to which starlark interpreter is faster :)
You don't need a JVM to run bazel, it ships as a self-contained executable. All you need is a
Depends on the platform. OSX uses
Sorry, that's a bazelism. It just means "code that is pulled from elsewhere instead of vendored", but it's completely transparent. The user experience is just
Cool, i'll give it a try, hopefully it shouldn't be too bad. It sounds from the above that you are (understandly) much more familiar with Buck than Bazel so hopefully a working demo should show what I mean a lot more clearly :) |
|
Abandoning in favor of Bazel support in #8875. |
This PR configures Codex CLI so it can be built with [Bazel](https://bazel.build) in addition to Cargo. The `.bazelrc` includes configuration so that remote builds can be done using [BuildBuddy](https://www.buildbuddy.io). If you are familiar with Bazel, things should work as you expect, e.g., run `bazel test //... --keep-going` to run all the tests in the repo, but we have also added some new aliases in the `justfile` for convenience: - `just bazel-test` to run tests locally - `just bazel-remote-test` to run tests remotely (currently, the remote build is for x86_64 Linux regardless of your host platform). Note we are currently seeing the following test failures in the remote build, so we still need to figure out what is happening here: ``` failures: suite::compact::manual_compact_twice_preserves_latest_user_messages suite::compact_resume_fork::compact_resume_after_second_compaction_preserves_history suite::compact_resume_fork::compact_resume_and_fork_preserve_model_history_view ``` - `just build-for-release` to build release binaries for all platforms/architectures remotely To setup remote execution: - [Create a buildbuddy account](https://app.buildbuddy.io/) (OpenAI employees should also request org access at https://openai.buildbuddy.io/join/ with their `@openai.com` email address.) - [Copy your API key](https://app.buildbuddy.io/docs/setup/) to `~/.bazelrc` (add the line `build --remote_header=x-buildbuddy-api-key=YOUR_KEY`) - Use `--config=remote` in your `bazel` invocations (or add `common --config=remote` to your `~/.bazelrc`, or use the `just` commands) ## CI In terms of CI, this PR introduces `.github/workflows/bazel.yml`, which uses Bazel to run the tests _locally_ on Mac and Linux GitHub runners (we are working on supporting Windows, but that is not ready yet). Note that the failures we are seeing in `just bazel-remote-test` do not occur on these GitHub CI jobs, so everything in `.github/workflows/bazel.yml` is green right now. The `bazel.yml` uses extra config in `.github/workflows/ci.bazelrc` so that macOS CI jobs build _remotely_ on Linux hosts (using the `docker://docker.io/mbolin491/codex-bazel` Docker image declared in the root `BUILD.bazel`) using cross-compilation to build the macOS artifacts. Then these artifacts are downloaded locally to GitHub's macOS runner so the tests can be executed natively. This is the relevant config that enables this: ``` common:macos --config=remote common:macos --strategy=remote common:macos --strategy=TestRunner=darwin-sandbox,local ``` Because of the remote caching benefits we get from BuildBuddy, these new CI jobs can be extremely fast! For example, consider these two jobs that ran all the tests on Linux x86_64: - Bazel 1m37s https://github.com/openai/codex/actions/runs/20861063212/job/59940545209?pr=8875 - Cargo 9m20s https://github.com/openai/codex/actions/runs/20861063192/job/59940559592?pr=8875 For now, we will continue to run both the Bazel and Cargo jobs for PRs, but once we add support for Windows and running Clippy, we should be able to cutover to using Bazel exclusively for PRs, which should still speed things up considerably. We will probably continue to run the Cargo jobs post-merge for commits that land on `main` as a sanity check. Release builds will also continue to be done by Cargo for now. Earlier attempt at this PR: #8832 Earlier attempt to add support for Buck2, now abandoned: #8504 --------- Co-authored-by: David Zbarsky <[email protected]> Co-authored-by: Michael Bolin <[email protected]>
This is an experiment to get the Rust portion of the Codex repo setup to build with Buck2.
The motivation is to get build time benefits, both for local/incremental builds as well as in CI, though of note:
BUCKbuild files for existing crates: https://github.com/facebookincubator/reindeer (I am not aware of a comparable offering for Bazel)Unfortunately,
buck2 testdoes not currently cache test results as Bazel does:facebook/buck2#183
This is pretty important, as we want incremental invocations of:
./scripts/buck2 test //codex-rs/ato be as fast as possible, so I'll follow up on this.