fix(jsonl): abandon an over-long line as it streams instead of buffering it whole

[joaovictor91123]

Summary

Closes #930. max_line_bytes documents itself as:

/// Skip individual lines larger than this instead of allocating them whole.
pub const DEFAULT_MAX_JSONL_LINE_BYTES: usize = 256 * 1024;

tail_lines did the opposite. Bytes were pushed into current_line
unconditionally, and the size test only ran once the terminating newline was
reached — so an over-long line was read into memory in full and only then
discarded. The cap changed which lines were returned; it did not bound memory.

Measured against main, tailing a file whose middle line is 8 MiB with
max_line_bytes = 4096:

tail_lines allocated 16777327 bytes for an 8388608-byte line with
max_line_bytes=4096; budget is 1048576 bytes.

16 MiB of peak allocation (the buffer doubles past the 8 MiB it needs) to
produce two short lines — the long one is not even returned to the caller.

Why it matters

tail_lines exists precisely to stop this: "loading the entire file on every
poll grows memory with log age" (#223). One pathological line reintroduces it
on the same polling paths:

• genie-api::routes tails 50 lines on a dashboard request
• genie-core::tools::actuation tails the action log on the actuation path

A runaway tool result, an embedded blob, or a partially-written record therefore
costs a multi-megabyte transient allocation on every poll, on an 8 GB Jetson
whose governor is already making mode decisions off memory pressure.

Changes

• Abandon the line as it streams: once current_line passes max_line_bytes,
  clear the buffer and set a flag that discards the rest of that line's bytes
  until the newline that starts it. Peak buffering is then bounded by
  max_line_bytes regardless of the longest line in the file.
• The predicate stays inclusive — len > max abandons — so a line of exactly
  max_line_bytes is still returned, as before.
• The trailing-fragment path (a first line with no leading newline) honours the
  same flag, so an over-long first line is skipped rather than resurrected.

Real Behavior Proof

• I have built and run the affected code locally (or noted why I could not).
• I have verified the change end-to-end on Jetson hardware.
• I have NOT verified on Jetson hardware, and I explain the equivalent verification path or validation gap below.

Tested profile / hardware (check all that apply):

• jetson
• raspberry_pi
• portable_sbc
• laptop
• mac
• CI-only / docs-only
• Not run locally

What I ran

x86_64 Linux laptop (Ubuntu 22.04 LTS, kernel 6.8.0-136-generic,
rustc 1.96.0 / cargo 1.96.0). No Jetson available to me — see the validation
gap below.

cargo test --workspace
cargo clippy --workspace --all-targets
cargo fmt --all -- --check

Then, to confirm the new test actually catches the bug rather than merely
passing: checked out main's jsonl.rs underneath the new test file and ran it
again.

What I observed

Against main's implementation with the new test in place:

running 1 test
test tail_lines_peak_allocation_stays_bounded_by_max_line_bytes ... FAILED

tail_lines allocated 16777327 bytes for an 8388608-byte line with
max_line_bytes=4096; budget is 1048576 bytes. An over-long line must be
abandoned as it streams, not buffered and then discarded.

With the fix restored, the same test passes, and so does the rest of the
workspace: 141 genie-common tests + the new integration binary green,
whole-workspace cargo test green, clippy clean, cargo fmt --check clean.

Why an allocator-based test

The behavioural tests only observe which lines come back. They pass just as
well against an implementation that reads a 50 MB line into memory and then
throws it away — which is exactly the bug. So the contract needs a test that
watches the allocator:
crates/genie-common/tests/jsonl_tail_memory.rs installs a peak-tracking
#[global_allocator], opens its measurement window after the fixture is
written, and asserts peak growth during the call.

It lives in its own integration-test binary so the instrumentation cannot
perturb, or be perturbed by, any other test. The budget (1 MiB) is ~256x
max_line_bytes and ~8x below the line itself — wide enough that incidental
harness allocation cannot trip it, tight enough that buffering the line whole
cannot pass.

Two behavioural tests come with it, covering paths the existing ones do not:

• an oversize line spanning 16 chunks. The existing oversize test uses a
  512-byte line, which fits inside a single 4096-byte read — so the state that
  has to survive across loop iterations (flag set in one chunk, cleared by a
  newline several chunks later) was untested. The test also re-reads the same
  file with a larger cap and asserts the line comes back, proving the skip is
  driven by max_line_bytes and not by the chunking.
• the exact-length boundary, so an off-by-one here cannot silently start
  discarding lines that used to be returned.

Validation gap

I could not verify on Jetson hardware. The equivalent verification path: this is
pure buffer management inside one synchronous function — no async, no hardware
dependency, no model involvement. Its two observable properties are the lines it
returns and the memory it uses to return them, and both are asserted directly
above. The memory property is architecture-independent: the allocation being
removed is a Vec<u8> sized by the input line, which is the same on aarch64 as
on x86_64. Callers are unchanged.

Test plan

1. Check out this branch, then git checkout main -- crates/genie-common/src/jsonl.rs.
2. cargo test -p genie-common --test jsonl_tail_memory — fails with the 16 MB
   figure above.
3. git checkout HEAD -- crates/genie-common/src/jsonl.rs, rerun — passes.

Notes for reviewers

• The #[global_allocator] is scoped to that one test binary and does not
  affect any shipped code path or any other test.
• No prompt growth, no new dependencies. The 4096-token Jetson context contract
  is untouched.