You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
docs(studio): add Experiments, Insights, Traces, and Model Evaluation… (#1828)
* docs(studio): add Experiments, Insights, Traces, and Model Evaluations pages
Covers experiment create/edit, evaluation drill-down, and side-by-side
run comparison; Optimizer insight list/detail; Traces/Spans tables and
session drill-down with per-span-kind views; and the Model Evaluations
results list/detail. These previously had only a one-line mention (or,
for Model Evaluations, no page) in docs/studio/index.mdx.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: Henrique Tolentino <htolentino@nvidia.com>
* docs(studio): fix stale CLI, Pareto axes, and insight-action claims in Experiments/Insights docs
- Experiments: CLI command tab runs nemo agents experimentalist run
(--train-dataset/--validation-dataset), not the nonexistent nemo exp
run; confirmed against nemo-experimentalist's CLI and the matching
command builder in InsightOpenModal.
- Experiments: Pareto view defaults to cost vs. latency, not accuracy,
and axes are configurable to cost_usd, latency_ms, or
evaluators.<name>, per ExperimentParetoChart's deriveParetoMetrics.
- Insights: Run experiment is shown only for resolved or deleted
insights, per insightStatus.ts's action table.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: Henrique Tolentino <htolentino@nvidia.com>
---------
Signed-off-by: Henrique Tolentino <htolentino@nvidia.com>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
3
+
# SPDX-License-Identifier: Apache-2.0
4
+
5
+
title: "NeMo Studio Experiments"
6
+
description: "Group evaluations into experiments, drill into results, and compare runs side by side."
7
+
---
8
+
9
+
Use **Evaluations > Experiments** in the NeMo Studio workspace sidebar to group evaluations for comparison, and to drill into evaluation results and individual runs.
10
+
11
+
<Note>
12
+
13
+
Experiments is enabled by default. An administrator can disable it by setting `studio.feature_flags.experiment: false` in the platform configuration and restarting Studio.
14
+
15
+
</Note>
16
+
17
+
## Create an Experiment
18
+
19
+
Select **New experiment** from the Experiments list. In the **Create experiment** tab, provide a name, an optional description, and a default sort order for the experiment's evaluations, then select **Create**.
20
+
21
+
<Note>
22
+
23
+
The modal also has **NeMo Assistant** and **CLI command** tabs. **CLI command** shows an equivalent `nemo agents experimentalist run` invocation, with `--train-dataset` and `--validation-dataset` selecting the training and validation data; **NeMo Assistant** is a placeholder for a future guided flow.
24
+
25
+
</Note>
26
+
27
+
## Experiment Detail
28
+
29
+
Select an experiment to open its detail page, which shows aggregate metrics across its evaluations and a table of the evaluations in the group. Toggle **Pareto view** to plot evaluations on a chart instead of the table, by default cost against latency. Change either axis to cost, latency, or any evaluator's score (`cost_usd`, `latency_ms`, or `evaluators.<name>`); the selection is saved per experiment.
30
+
31
+
Select **Edit** to update the experiment's name, description, or default sort order.
32
+
33
+
If the experiment was created from an Optimizer insight, an **Insight description** card appears with a link back to that insight. See [NeMo Studio Insights](/documentation/studio/insights).
34
+
35
+
## Evaluation Drill-Down
36
+
37
+
Select an evaluation from an experiment's table to open its detail page, which shows aggregate metrics for that evaluation and a table of its test-case runs (sessions).
38
+
39
+
Select a session to open its detail page — the same trace and span view used by **Traces**, scoped to that test case. See [NeMo Studio Traces](/documentation/studio/traces).
40
+
41
+
## Compare Runs Side by Side
42
+
43
+
From a session's detail page, select **Compare against…** and choose another run of the same test case. Studio splits the view into two columns — the current run on the left, the selected run on the right — so you can compare trace output side by side. Select **Clear** to return to the single-run view.
Copy file name to clipboardExpand all lines: docs/studio/index.mdx
+18-6Lines changed: 18 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -32,11 +32,11 @@ The current sidebar groups related pages under expandable parents. Selecting a p
32
32
33
33
| Sidebar group | Current entry point | Purpose and availability |
34
34
| --- | --- | --- |
35
-
|**Observability**|**Insights**| Review recurring agent failure patterns and optimization insights. Shown when optimization is enabled. |
36
-
|**Observability**|**Traces**| Inspect ingested traces, spans, and session details. |
35
+
|**Observability**|**Insights**| Review optimization insights generated from observed agent sessions. Shown when optimization is enabled; see [NeMo Studio Insights](/documentation/studio/insights). |
36
+
|**Observability**|**Traces**| Inspect ingested traces, spans, and session details; see [NeMo Studio Traces](/documentation/studio/traces). |
37
37
|**Components**|**Agents**| Opens the agent list. Select an agent to access Deployments, Logs, Chat, Evaluations, and Details. |
38
-
|**Components**|**Models**| Opens the base-model list directly. Expand **Models** for **Fine-tune**, **Model Evaluations**, **Playground**, and **Virtual Models**, when those capabilities are enabled. **Fine-tune** opens the Custom Models list; see [NeMo Studio Custom Models](/documentation/studio/customization). |
39
-
|**Evaluations**|**Experiments**|Review experiments, evaluations, sessions, and published results across the workspace. |
38
+
|**Components**|**Models**| Opens the base-model list directly. Expand **Models** for **Fine-tune**, **Model Evaluations**, **Playground**, and **Virtual Models**, when those capabilities are enabled. **Fine-tune** opens the Custom Models list; see [NeMo Studio Custom Models](/documentation/studio/customization). **Model Evaluations** opens evaluation job results; see [NeMo Studio Model Evaluations](/documentation/studio/model-evaluations). |
39
+
|**Evaluations**|**Experiments**|Group evaluations for comparison, drill into results, and compare runs side by side; see [NeMo Studio Experiments](/documentation/studio/experiments). |
40
40
|**Data**|**Datasets > Data Designer**| Build and monitor synthetic-data jobs. Other installed data plugins can appear below **Datasets**. |
41
41
|**Governance**|**Guardrails**| Manage guardrail configurations. Disabled by default; see [Studio Guardrail Configs](/documentation/studio/guardrail-configs). |
42
42
|**Governance**|**Iron Swarm**| Plugin-provided agent hardening UI. Appears only when the Iron Swarm plugin is installed; see [Studio Plugin UIs](/documentation/studio/plugins). |
@@ -51,6 +51,18 @@ Use the Studio Agents area to review Platform-managed agents, including the curr
51
51
52
52
For the full workflow, see [Studio Agents](/documentation/studio/agents).
53
53
54
+
### Insights
55
+
56
+
Use **Observability > Insights** to review optimization insights the analyst agent generates from observed agent sessions and traces, and drill into an insight's linked experiments and source traces. For a resolved or deleted insight, **Run experiment** shows the CLI command to start experiments for it.
57
+
58
+
For the full workflow, see [Studio Insights](/documentation/studio/insights).
59
+
60
+
### Traces
61
+
62
+
Use **Observability > Traces** to inspect ingested telemetry in the Traces and Spans tables, and drill into a session's trace and span detail, with specialized views for LLM, tool, agent, chain, embedding, reranker, retriever, guardrail, and evaluator spans.
63
+
64
+
For the full workflow, see [Studio Traces](/documentation/studio/traces).
65
+
54
66
### Monitor
55
67
56
68
Use **Agents \> Monitor** to inspect agent telemetry stored by the platform, including recent inference logs and token usage summaries.
@@ -120,8 +132,8 @@ Studio provides separate agent, model, and workspace evaluation views:
120
132
| Location | What it shows |
121
133
| -------- | ------------- |
122
134
|**Agents > select an agent > Evaluations**| Active evaluation jobs, completed evaluations, and experiments scoped to the selected agent. |
123
-
|**Models > Model Evaluations**| Model evaluation results. |
124
-
|**Evaluations > Experiments**| Experiments and their evaluations across the workspace. |
135
+
|**Models > Model Evaluations**| Model evaluation job results and per-row scores. See [Studio Model Evaluations](/documentation/studio/model-evaluations). |
136
+
|**Evaluations > Experiments**| Experiments and their evaluations across the workspace, evaluation drill-down, and side-by-side run comparison. See [Studio Experiments](/documentation/studio/experiments). |
125
137
126
138
Use the agent's **Evaluations** tab when reviewing how a specific agent performed. Use **Experiments** for the broader workspace history.
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
3
+
# SPDX-License-Identifier: Apache-2.0
4
+
5
+
title: "NeMo Studio Insights"
6
+
description: "Review optimization insights generated from observed agent sessions."
7
+
---
8
+
9
+
Use **Observability > Insights** in the NeMo Studio workspace sidebar to review insights the analyst agent generates from observed agent sessions and traces.
10
+
11
+
<Note>
12
+
13
+
Insights is enabled by default. An administrator can disable it by setting `studio.feature_flags.optimizer_enabled: false` in the platform configuration and restarting Studio.
14
+
15
+
</Note>
16
+
17
+
## Insight List
18
+
19
+
The Insights table shows each insight's status, title, source agent, number of observed traces, number of linked experiments, and creation and last-seen times. Select a row to open the insight's detail page.
20
+
21
+
## Insight Detail
22
+
23
+
The detail page shows the insight's status, source agent, description, and:
24
+
25
+
| Section | Shows |
26
+
| ------- | ----- |
27
+
| Experiments | Experiments linked to this insight, with evaluation counts and last-updated times. Select a row to open that experiment. |
28
+
| Traces | The observed sessions and traces that produced this insight. |
29
+
30
+
Available actions depend on the insight's status:
31
+
32
+
| Status | Available actions |
33
+
| ------ | ------------------ |
34
+
| Open |**Delete**, or **Resolve**. |
35
+
| Resolved or Deleted |**Run experiment** — reopens the insight and shows a modal with the `nemo exp run` CLI command to start experiments for it. Studio doesn't launch the experiment for you; run the command yourself. |
36
+
37
+
If the insight originated an experiment, that experiment's detail page links back here. See [NeMo Studio Experiments](/documentation/studio/experiments).
38
+
39
+
## Related Topics
40
+
41
+
-[NeMo Studio Experiments](/documentation/studio/experiments)
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
3
+
# SPDX-License-Identifier: Apache-2.0
4
+
5
+
title: "NeMo Studio Model Evaluations"
6
+
description: "Review model evaluation job results and per-row scores."
7
+
---
8
+
9
+
Use **Models > Model Evaluations** in the NeMo Studio workspace sidebar to review evaluation jobs run against models in the workspace.
10
+
11
+
<Note>
12
+
13
+
Model Evaluations is enabled by default. An administrator can disable it by setting `studio.feature_flags.evaluator_enabled: false` in the platform configuration and restarting Studio, which also removes agent- and customization-scoped evaluation entry points.
14
+
15
+
</Note>
16
+
17
+
This page is read-only: it lists evaluation jobs and their results. Start an evaluation from an agent's **Evaluations** tab, a Custom Model's **Evaluate this Model** action, or the `nemo evaluation` CLI or API.
18
+
19
+
## Evaluation Results List
20
+
21
+
The list shows each evaluation job's name, status, and creation time. Search by name or filter by status. Select a row to open the job's detail page.
22
+
23
+
## Evaluation Result Detail
24
+
25
+
The detail page shows the job's details, a **Scores** panel with aggregate metrics, per-row results for the evaluated dataset, and a collapsible **Logs** panel. Scores and row results are computed once the job reaches a terminal state; if the job failed, results aren't available.
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
3
+
# SPDX-License-Identifier: Apache-2.0
4
+
5
+
title: "NeMo Studio Traces"
6
+
description: "Inspect ingested traces, spans, and session detail from agent telemetry."
7
+
---
8
+
9
+
Use **Observability > Traces** in the NeMo Studio workspace sidebar to inspect telemetry ingested from agent runs.
10
+
11
+
<Note>
12
+
13
+
Traces is enabled by default. An administrator can disable it by setting `studio.feature_flags.intake_enabled: false` in the platform configuration and restarting Studio.
14
+
15
+
</Note>
16
+
17
+
## Traces and Spans Tables
18
+
19
+
The **Traces** and **Spans** tabs list ingested telemetry at different granularity:
Select a row in either table to open the session it belongs to.
27
+
28
+
## Session Detail
29
+
30
+
The session detail page shows a summary header, then the selected trace's spans. Use the **Tree** / **List** toggle to switch between a nested span tree and a flat list, and **Collapse all** / **Expand all** to manage the tree view.
31
+
32
+
Selecting a span opens its detail body. Studio renders a specialized view for known span kinds — LLM, tool, agent, chain, embedding, reranker, retriever, guardrail, and evaluator — and falls back to a generic view for unrecognized kinds. Each span body includes metadata, an annotations panel, and a raw JSON view for debugging.
33
+
34
+
Evaluation test-case runs reuse this same session detail view; see [Compare Runs Side by Side](/documentation/studio/experiments#compare-runs-side-by-side) in NeMo Studio Experiments.
35
+
36
+
## Related Topics
37
+
38
+
-[NeMo Studio Experiments](/documentation/studio/experiments)
39
+
-[NeMo Studio Insights](/documentation/studio/insights)
0 commit comments