Skip to content

Commit 5d6cdd8

Browse files
committed
Record structured performance layout and fresh regression
1 parent 7290172 commit 5d6cdd8

7 files changed

Lines changed: 181 additions & 3 deletions

‎docs/proofs/spend-turn-performance-details.json‎

Lines changed: 118 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -238,5 +238,123 @@
238238
"tests": 7,
239239
"command": "./Scripts/test_fast.sh --skip-build --filter OpenAISubscriptionMetadataTests",
240240
"note": "separate confirmation after full suite; no assertion changes"
241+
},
242+
"structured_layout": {
243+
"date": "2026-10-08",
244+
"source_commit": "7290172ec2fd8cc780269864b66e136560a24fac",
245+
"upstream_base": "844b0e19b",
246+
"parser_revision": 11,
247+
"parser_hash": "c0f8e9be04d824c2",
248+
"changes": [
249+
"title and secondary billing header; icon beside title; visible session rank removed",
250+
"three prominent values with smaller labels",
251+
"collapsed details with visible timed-turn count",
252+
"four aligned details with independent sample and cache coverage notes",
253+
"numeric model/effort table with narrow-width compact fallback",
254+
"explanations in help and accessibility text"
255+
],
256+
"component_render": {
257+
"source_commit": "70937cc624876642b5fcb91d9c56b139bb7c16a8",
258+
"production_ui_unchanged_after_upstream_merge": true,
259+
"synthetic": true,
260+
"languages": [
261+
"en",
262+
"zh-Hans"
263+
],
264+
"appearances": [
265+
"light",
266+
"dark"
267+
],
268+
"widths_points": [
269+
320,
270+
420,
271+
820
272+
],
273+
"visually_checked": true
274+
},
275+
"installed_window_before_upstream_merge": {
276+
"source_commit": "70937cc624876642b5fcb91d9c56b139bb7c16a8",
277+
"binary_sha256": "b2a08521ee7c3885316e35e1eae52b4d6e1f9ab96e6b6e5398d69fb7656b0a07",
278+
"synthetic": true,
279+
"window_size_points": [
280+
800,
281+
568
282+
],
283+
"completed_checks": [
284+
"range24",
285+
"collapse",
286+
"expand",
287+
"today12",
288+
"clear24",
289+
"yesterday12",
290+
"clear24"
291+
],
292+
"model_groups_and_values_match_reference": true,
293+
"continuous_interaction_completed": false,
294+
"interruption": "AX window became unavailable during repeated expansion; process remained running; no crash confirmed",
295+
"screenshots": "installed window with isolated synthetic data; no host desktop"
296+
},
297+
"fresh_installed_build": {
298+
"source_commit": "7290172ec2fd8cc780269864b66e136560a24fac",
299+
"binary_sha256": "12eb893c85c88b6c95a3c8b9a9662c5c6755c0e4141784785272c99c3cbad835",
300+
"build_timestamp": "2026-10-08T01:04:21Z",
301+
"build": "debug; ad-hoc; source unmodified for proof",
302+
"isolated_synthetic_rows": 24,
303+
"isolated_synthetic_files": 1,
304+
"credential_access_disabled": true,
305+
"native_interaction_passed": false,
306+
"native_interaction_state": "blocked by locked macOS session",
307+
"evidence": "CGSessionCopyCurrentDictionary CGSSessionScreenIsLocked = 1; AXWindows returned application elements and target-window screenshot failed",
308+
"process_exited": false,
309+
"watchdog_ms_before_lock_diagnosis": [
310+
540,
311+
840,
312+
384
313+
],
314+
"limits": "debug build on shared host; full regression was active during early attempts; no paired baseline, no release performance or leak guarantee",
315+
"watchdog_ms_after_restart_while_locked": [
316+
393,
317+
557
318+
]
319+
},
320+
"validation": {
321+
"focused_tests": {
322+
"passed": true,
323+
"tests": 46,
324+
"suites": 5,
325+
"cache_adoption_parameter_cases": 47
326+
},
327+
"static_checks": {
328+
"passed": true,
329+
"command": "make check",
330+
"swift_files": 2856,
331+
"violations": 0
332+
},
333+
"full_regression": {
334+
"passed": true,
335+
"exit_code": 0,
336+
"command": "./Scripts/test.sh --direct-workers 4",
337+
"workers": 4,
338+
"selected_selections": 1599,
339+
"selected_groups": 144,
340+
"verified_runtime_methods": 14098,
341+
"first_pass_successful_groups": 144,
342+
"first_pass_failed_groups": 0,
343+
"full_group_retries": 0,
344+
"recovered_groups": 0,
345+
"timed_out_groups": 0,
346+
"isolated_selection_retries": 0,
347+
"total_seconds": 592.1,
348+
"assertions": "no assertions relaxed or failures suppressed"
349+
},
350+
"earlier_failed_layout_run": {
351+
"source_commit": "70937cc624876642b5fcb91d9c56b139bb7c16a8",
352+
"exit_code": 1,
353+
"first_pass_successful_groups": 88,
354+
"first_pass_failed_groups": 1,
355+
"issue": "Italian exact unchanged-key allowlist omitted four shared unit/numeric formats",
356+
"fix": "register only seconds, rate, rate range and percent; exact set equality retained"
357+
}
358+
}
241359
}
242360
}
233 KB
Loading
118 KB
Loading
262 KB
Loading
113 KB
Loading
255 KB
Loading

‎docs/spend-turn-performance-validation.md‎

Lines changed: 63 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,8 @@ read_when:
88
# Spend session turn performance validation
99

1010
Validated on arm64 macOS 27 with Swift 6.4. Current installed-window code:
11-
`d66ac3e9cc37eb4a1e8f46b143aec35f8887ec31`. The historical release comparison
11+
`7290172ec2fd8cc780269864b66e136560a24fac` (structured layout; final native
12+
interaction remains blocked by a locked host session). The historical release comparison
1213
used upstream baseline `ab32496d3c1f861981aa1a19aee2f933204f9bb9` and feature
1314
production code `22fdcb88e8d77a03c9d8447f889012247d4bb483`.
1415
The original validation used parser revision 9. The advanced details use parser revision 11 after merging main at `03a51bdcf`; the current generated parser hash is checked by `make check`.
@@ -17,7 +18,7 @@ The original validation used parser revision 9. The advanced details use parser
1718

1819
Native Codex session rows show total valid output tokens divided by total valid
1920
turn duration, median available first-token latency, and median completed-turn
20-
duration. The number of timed turns remains in the tooltip and expanded details. Output includes reasoning tokens already contained in output. Duration
21+
duration. The number of timed turns remains visible beside the collapsed disclosure and in its tooltip. Output includes reasoning tokens already contained in output. Duration
2122
includes tools, retries, network and waiting; this is whole-turn throughput.
2223
Session summaries can combine models and tool-heavy turns and are not a model
2324
streaming benchmark. Samples belong to their completion day.
@@ -78,7 +79,8 @@ provider CLI stub. No credentials or account configuration were copied.
7879

7980
Those earlier artifacts did not establish installed-window behavior. The current
8081
synthetic native-window checks and their bounded runtime limits are recorded in
81-
[Installed-window verification](#installed-window-verification-2026-10-07) below.
82+
[Structured session layout](#structured-session-layout-2026-10-08) and the historical
83+
[installed-window verification](#installed-window-verification-2026-10-07) below.
8284

8385
## Follow-up date-filter review
8486

@@ -345,3 +347,61 @@ revisions. See the [exact receipt](proofs/spend-turn-performance-details.json).
345347
A separate post-suite confirmation of `OpenAISubscriptionMetadataTests` passed
346348
all seven XCTest cases without retry. This does not erase the full-run first-pass
347349
fixture timeout recorded above.
350+
351+
## Structured session layout (2026-10-08)
352+
353+
Session rows now separate the title/billing header from the performance metrics.
354+
The provider icon stays beside the title, with billing as secondary text. The
355+
visible session rank was removed. Three primary values show first-token latency,
356+
whole-turn output and completed-turn duration; labels remain smaller than values.
357+
358+
The initially collapsed disclosure keeps the timed-turn count visible. Expanded
359+
details use four aligned fields for P95 first token, P95 duration, the middle 50%
360+
of per-turn speeds and cached input, including its coverage. Missing values use
361+
an em dash with an independent insufficient-sample note. Model/effort observations
362+
use a comparison table with aligned numeric columns and partial TTFT coverage;
363+
narrow widths switch to compact model blocks. Explanation is available through
364+
help and accessibility text instead of repeated prose in each row.
365+
366+
The data contract above remains unchanged. These observations include workload,
367+
reasoning, tools and waits and are not a model streaming-speed leaderboard.
368+
369+
Validation and screenshots are recorded separately from the October 7 layout.
370+
371+
Production UI renders were checked in English and Simplified Chinese, light and
372+
dark, at 320, 420 and 820 points. The installed `70937cc62` window completed the
373+
seven-day/collapse/expand/today/clear/yesterday/clear checks, with 24/12/12 timed
374+
turns matching the independent synthetic reference. Continuous expansion was
375+
interrupted when its AX window became unavailable; the process remained alive.
376+
These screenshots belong to that UI revision, which is unchanged by the latest
377+
upstream merge.
378+
379+
![Structured details in the installed window, synthetic data](screenshots/spend-turn-performance-layout-expanded-synthetic.png)
380+
381+
[Collapsed](screenshots/spend-turn-performance-layout-collapsed-synthetic.png) ·
382+
[Selected day with insufficient P95 samples](screenshots/spend-turn-performance-layout-today-synthetic.png) ·
383+
[Narrow dark component](screenshots/spend-turn-performance-layout-narrow-dark-synthetic.png)
384+
385+
The latest source merges upstream `844b0e19b`. Parser revision 11 now has hash
386+
`c0f8e9be04d824c2`; value-preserving string-sharing and earlier timing caches
387+
remain compatible, including saved report pricing. Forty-six focused tests in
388+
five suites passed, including all 47 predecessor-adoption parameter cases.
389+
`make check` passed with 2,856 Swift files and zero violations. A fresh complete
390+
four-worker regression exited 0: all 144 groups and 1,599 selections passed on
391+
the first attempt, with no failures, timeouts or retries (592.1 seconds; runtime
392+
inventory verified 14,098 methods).
393+
394+
The first layout run stopped at the Italian unchanged-key allowlist. Four shared
395+
unit/numeric formats were registered explicitly; exact set equality remains.
396+
The failed run is retained in the receipt and is not counted as passed.
397+
398+
A fresh debug/ad-hoc bundle was built from the clean merged source. Its isolated
399+
cache contains exactly one synthetic file and 24 usage rows; no credentials were
400+
copied. Final native-window repetition is **not passed**: macOS reported
401+
`CGSSessionScreenIsLocked = 1`; the accessibility windows were returned with the
402+
wrong roles, and target-window capture was unavailable. The process stayed
403+
running. Before diagnosing the lock, the watchdog recorded 540/840/384 ms on a
404+
shared host with the regression concurrently active. This is not stall-free or
405+
release performance evidence. A restart while locked recorded 393/557 ms; it
406+
also does not establish interactive responsiveness. Unlocking the host is required to finish this
407+
runtime gate. Earlier October 7 runtime receipts remain historical evidence.

0 commit comments

Comments
 (0)