Skip to content

Commit fc0817e

Browse files
committed
docs: refine v0.2 results presentation
1 parent 7a02972 commit fc0817e

19 files changed

Lines changed: 7061 additions & 951 deletions

README.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -106,11 +106,11 @@ NetOpsBench reports detection, fault type, device/interface localization, runtim
106106

107107
**Diagnosis score** is the mean end-to-end case score: healthy cases require the correct verdict, while fault cases receive localization credit only after the fault is detected. **Fault detection F1** measures the fault-versus-healthy decision independently.
108108

109-
![NetOpsBench v0.2.0 quality across seven scales](docs/public/assets/benchmark/fig_deepseek_v02_overview.svg)
109+
![Diagnosis score and Fault detection F1 across all seven NetOpsBench v0.2.0 topology scales](docs/public/assets/benchmark/fig_deepseek_v02_overview.svg)
110110

111111
The largest validated Fat-tree profile provides a compact case-level view. Each square below is one K=12 case; detailed cross-topology observability analysis remains in the full results.
112112

113-
![Fat-tree K=12 case outcomes grouped by fault family](docs/public/assets/benchmark/fig_deepseek_v02_k12_cases.svg)
113+
![All 70 Fat-tree K=12 cases grouped by fault family and diagnosis outcome](docs/public/assets/benchmark/fig_deepseek_v02_k12_cases.svg)
114114

115115
Read the [v0.2.0 release notes](docs/content/docs/releases/v0.2.0.mdx), [Benchmark Methodology](docs/content/docs/run-benchmarks/methodology.mdx), and [Benchmark Results](docs/content/docs/run-benchmarks/results.mdx) for the full validation snapshot and scoring definitions.
116116

docs/content/docs/index.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -75,5 +75,5 @@ These docs are organized around the agent-development workflow.
7575

7676
- [System Overview](/docs/architecture/system-overview) explains the runtime loop, evidence path, worker isolation, and report aggregation.
7777
- [Benchmark Methodology](/docs/run-benchmarks/methodology) defines scenario coverage, scoring, negative samples, and optional semantic fault-type matching.
78-
- [Benchmark Results](/docs/run-benchmarks/results) records one completed cross-model run for comparison context.
78+
- [Benchmark Results](/docs/run-benchmarks/results) reports the current seven-scale v0.2 validation; [Legacy Results](/docs/run-benchmarks/legacy-results) preserves the earlier cross-model snapshot separately.
7979
- [NetOpsBench v0.2.0](/docs/releases/v0.2.0) summarizes the Xlarge, Fat-tree, native client-agent, and lifecycle release.
Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
---
2+
title: Legacy Cross-model Results
3+
description: Earlier four-model NetOpsBench comparison retained for within-snapshot analysis.
4+
---
5+
6+
This snapshot predates the NetOpsBench v0.2.0 release rerun. Its four models were evaluated under the same earlier contract, so it remains useful for within-snapshot comparison. Do not combine these values with the current [v0.2.0 Results](/docs/run-benchmarks/results) as if both snapshots used the same runtime and detector contract.
7+
8+
## Experiment scope
9+
10+
| Dimension | Values |
11+
|---|---|
12+
| Models | Kimi K2.6, DeepSeek V4 Pro, OpenAI GPT-5.5, MiniMax M3 |
13+
| Topology scales | XS, Small, Medium, Large |
14+
| Fault types | 12 canonical types across link, routing, impairment, system, and ACL categories |
15+
| Quality metrics | Verdict F1-score, device localization, interface localization, composite score |
16+
| Efficiency metrics | Diagnosis time, tool calls, input/output tokens |
17+
18+
## Cross-scale snapshot
19+
20+
Each cell is `XS → Large`.
21+
22+
| Model | Verdict F1 (%) | Device Loc (%) | Interface Loc (%) | Composite (%) | Avg Time (s) | Avg Tools | Input Tokens (K) |
23+
|---|---:|---:|---:|---:|---:|---:|---:|
24+
| Kimi K2.6 | 76.2 → 84.7 | 66.7 → 75.0 | 71.4 → 71.4 | 64.3 → 74.0 | 272.0 → 399.5 | 36.5 → 38.8 | 367.6 → 741.1 |
25+
| DeepSeek V4 Pro | 100.0 → 97.9 | 83.3 → 91.7 | 57.1 → 57.1 | 78.6 → 83.7 | 83.1 → 83.1 | 24.9 → 20.7 | 247.6 → 477.3 |
26+
| OpenAI GPT-5.5 | 95.7 → 95.8 | 83.3 → 79.2 | 57.1 → 21.4 | 75.0 → 60.6 | 71.9 → 85.0 | 17.9 → 15.9 | 85.4 → 153.7 |
27+
| MiniMax M3 | 80.0 → 82.9 | 50.0 → 62.5 | 71.4 → 42.9 | 60.7 → 58.7 | 214.3 → 230.3 | 26.4 → 24.0 | 133.3 → 316.1 |
28+
29+
Verdict classification was generally easier than precise localization. Interface localization was the weakest quality metric on larger topologies.
30+
31+
## Quality metrics
32+
33+
![Legacy Verdict F1-score across XS through Large](/assets/benchmark/fig_verdict_f1.png)
34+
35+
![Legacy device localization across XS through Large](/assets/benchmark/fig_device_loc.png)
36+
37+
![Legacy interface localization across XS through Large](/assets/benchmark/fig_intf_loc.png)
38+
39+
![Legacy composite score across XS through Large](/assets/benchmark/fig_avg_score.png)
40+
41+
## Runtime cost
42+
43+
![Legacy average diagnosis time across XS through Large](/assets/benchmark/fig_avg_time.png)
44+
45+
![Legacy tool calls across XS through Large](/assets/benchmark/fig_tool_calls.png)
46+
47+
![Legacy input tokens across XS through Large](/assets/benchmark/fig_input_tokens.png)
48+
49+
![Legacy output tokens across XS through Large](/assets/benchmark/fig_output_tokens.png)
50+
51+
Compare cost with localization quality, not verdict quality alone. Higher tool or token use does not automatically improve device or interface precision.
Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
11
{
22
"title": "Benchmark Runs",
3-
"pages": ["run-scenario-vs-suite", "interactive-simulator", "methodology", "results"]
3+
"pages": ["run-scenario-vs-suite", "interactive-simulator", "methodology", "results", "legacy-results"]
44
}

0 commit comments

Comments
 (0)