feat(build-analysis): JSONL-based logging infra #16150

weihanglo · 2025-10-24T21:27:49Z

What does this PR try to resolve?

This adds A new JSONL-based logging infrastructure for -Zbuild-analysis.

Some highlights:

JSONL logs stored in ~/.cargo/log/.
Unique log file for each cargo invocation,
with a run ID based on workspace hash and timestamp.
Uses background thread for non-blocking log writes.
Open to use some other battle-tested crates in the future.
A example build-started log message is added.

Note that this is completely different than the original SQLite based design.
I realized this is better as we do writes and zero read during the build. We shouldn't pay the cost if we don't read those metrics.

See the design doc https://hackmd.io/K5-sGEJeR5mLGsJLXqsHrw and #15844

How to test and review this PR?

CARGO_BUILD_ANALYSIS_ENABLED=true cargo -Zbuild-analysis check

rustbot · 2025-10-24T21:27:54Z

r? @epage

rustbot has assigned @epage.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

src/cargo/util/logger.rs

epage · 2025-10-24T21:38:22Z

src/cargo/util/logger.rs

+            for json_line in rx {
+                let _ = writer.write_all(json_line.as_bytes());
+                let _ = writer.write_all(b"\n");
+            }
+            let _ = writer.flush();


Do we need to flush after each json_line to avoid problems if this doesn't get cleaned up properly?

I think it depends on whether we want to flush every single jsonline. If yes, they we probably also want to remove the BufWriter. Otherwise I believe BufWriter will flush every 8KiB.

I don't think the log atm is critical to always flush though. If cargo crashes they might probably want to look at something else than this log. However, this log can expand to serve more purposes.

src/cargo/util/logger.rs

epage · 2025-10-24T21:41:15Z

src/cargo/util/log_message.rs

+use jiff::Timestamp;
+use serde::Serialize;
+
+pub trait LogMessage: Serialize {


The nice thing about an enum is we have a single place to look for and handle the schema for processing these

Sorry I don't understand. What change you're suggesting/looking for?

BTW this is modeled after machine_message.rs. If we're talking about make a giant enum for all kind of log message schema, I am totally okay with that.

Updated. Let me know if it is desired.

I'm not sure if there is a "right" answer here. We can try something and see how it works.

For example, if we decide to forward all content from machine_message.rs, then we'll run into issues with composing that with the enum.

I think we're fine duplicating them a bit.

Once we have this logging mechanism, the unstable timing JSON message may go away

We should be cautious not changing any stable message schema during the experiment of build-analysis

src/cargo/util/log_message.rs

Remove HTML timing output generation in favor of the upcoming JSONL logging system.

The logging infrastructure for `-Zbuild-analysis`. Some highlights: - JSONL logs stored in `~/.cargo/log/`. - Unique log file for each cargo invocation, with a run ID based on workspace hash and timestamp. - Uses background thread for log writes and serialization. Open to use other battle-tested crates in the future.

The build-started message ought to be pretty much the same as the header in timings HTML report, but some data is only available after the entire build or dependency resolution. Let's collect these data first.

Should produce one logfile per call

rustbot · 2025-10-28T14:34:49Z

This PR was rebased onto a different master commit. Here's a range-diff highlighting what actually changed.

Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers.

Update cargo submodule 11 commits in 344c4567c634a25837e3c3476aac08af84cf9203..6c1b6100343691341b9e76c5acc594e78220f963 2025-10-15 15:01:32 +0000 to 2025-10-28 16:27:52 +0000 - feat(build-analysis): JSONL-based logging infra (rust-lang/cargo#16150) - feat: support array of any types in Cargo config (rust-lang/cargo#16103) - test(git): add more fetch-index backend interop (rust-lang/cargo#16162) - feat(git): support shallow fetch for Git CLI backend (rust-lang/cargo#16156) - Fix mdman to not incorrectly strip `<p>` tags (rust-lang/cargo#16158) - chore(triagebot): enable range-diff and review-changes-since (rust-lang/cargo#16152) - Avoid specifying which version will change behavior (rust-lang/cargo#16153) - Make shell completion variables private. (rust-lang/cargo#16144) - More warning conversions (rust-lang/cargo#16143) - Bump openssl-src to 300.3.5.4+3.5.4 (rust-lang/cargo#16140) - build: remove duplicate dependency, consolidate over unicode-ident (rust-lang/cargo#16137)

rustbot added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Oct 24, 2025

rustbot assigned epage Oct 24, 2025

weihanglo mentioned this pull request Oct 24, 2025

Tracking Issue for -Zbuild-analysis #15844

Open

weihanglo changed the title ~~fix(build-analysis): JSONL-based logging infra~~ feat(build-analysis): JSONL-based logging infra Oct 24, 2025