docs(evaluator): fix submit() packager, FilesetRef import, and leaked titles (#406)

* docs(evaluator): fix SDK submit() examples, FilesetRef import, ModelRef, and leaked titles

The Evaluator docs drifted from the current plugin SDK contract. This brings
the runnable snippets back in line and fixes two broken tutorial titles.

- submit(): add the now-required `metric_bundle_packager=CloudpickleMetricBundlePackager()`
  (and its import) to every runnable `evaluator.submit(...)` example across
  index, sdk-resources, and the metrics/* and tutorials/* pages. `run()`
  examples are unchanged (they do not take the packager).
- LLM Judge tutorial: import `FilesetRef` from `nemo_evaluator.sdk` instead of
  the non-existent `nemo_evaluator_sdk.values.FilesetRef`.
- model-configuration: document that local `run()` requires an inline `Model`
  while remote `submit()` also accepts a `ModelRef`.
- test_doc_examples.py: replace the stale `/v2/...` REST script with offline
  contract checks for the import paths and the submit() packager requirement.
- Fix two tutorial pages that rendered `@nemo-nb: hide` as the page title
  (leaked cell marker in frontmatter `title`); set real titles and drop the
  duplicate body H1, matching every other evaluator page.

Verified: `make docs-check` and `make docs-broken-links` pass; ruff/ty clean;
the new contract test passes (7/7); both tutorial titles confirmed via the Fern
dev-server render.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Sandy Chapman <schapman@nvidia.com>

* docs(evaluator): fix ModelRef submit example config + shorten test name

Address review feedback on #406:
- ModelRef submit example now passes config=RunConfigOnlineModel(). A ModelRef
  target generates outputs online, and _submit_params raises
  TypeError("ModelRef target requires RunConfigOnlineModel") otherwise — so the
  example was not runnable. (CodeRabbit flagged this; its suggested fix used
  params=/the plugin import path — the public submit() keyword is config= and
  the value type is imported from nemo_evaluator_sdk like the other docs.)
- Rename test_submit_exposes_metric_bundle_packager_but_run_does_not to
  test_packager_param_is_submit_only (reviewer: name too long).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Sandy Chapman <schapman@nvidia.com>

---------

Signed-off-by: Sandy Chapman <schapman@nvidia.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

Author: SandyChapman

docs/evaluator/index.mdx

@@ -59,6 +59,7 @@ Submit your evaluation to the Evaluator service using the NeMo Platform SDK:
 
 ```python
 from nemo_evaluator.sdk import Evaluator
+from nemo_evaluator.shared.metric_bundles.cloudpickle import CloudpickleMetricBundlePackager
 from nemo_platform import NeMoPlatform
 
 
@@ -69,7 +70,12 @@ evaluator: Evaluator = sdk.evaluator
 local_result = evaluator.run(metric=metric, dataset=dataset, config=config)
 
 # Production evaluation as a durable platform job
-job = evaluator.submit(metric=metric, dataset=dataset, config=config)
+job = evaluator.submit(
+    metric=metric,
+    dataset=dataset,
+    config=config,
+    metric_bundle_packager=CloudpickleMetricBundlePackager(),
+)
 job.wait_until_done()
 result = job.get_result()
 ```

docs/evaluator/metrics/agent-configuration.mdx

@@ -164,6 +164,7 @@ from nemo_evaluator_sdk import Agent, RunConfigOnline
 
 
 from nemo_evaluator_sdk import ExactMatchMetric
+from nemo_evaluator.shared.metric_bundles.cloudpickle import CloudpickleMetricBundlePackager
 metric = ExactMatchMetric(reference="{{item.expected_answer}}")
 agent = Agent(
     url="https://my-nat-agent.example.com",
@@ -184,6 +185,7 @@ job = evaluator.submit(
             {"role": "user", "content": "{{item.question}}"},
         ],
     },
+    metric_bundle_packager=CloudpickleMetricBundlePackager(),
 )
 job.wait_until_done()
 result = job.get_result()

docs/evaluator/metrics/agentic.mdx

@@ -228,6 +228,7 @@ print(result.aggregate_scores)
 ```python
 from nemo_evaluator_sdk import RunConfig
 from nemo_evaluator_sdk.metrics.ragas import ToolCallAccuracyMetric
+from nemo_evaluator.shared.metric_bundles.cloudpickle import CloudpickleMetricBundlePackager
 metric = ToolCallAccuracyMetric()
 
 job = evaluator.submit(
@@ -250,6 +251,7 @@ job = evaluator.submit(
         }
     ],
     config=RunConfig(parallelism=4),
+    metric_bundle_packager=CloudpickleMetricBundlePackager(),
 )
 job.wait_until_done()
 result = job.get_result()
@@ -428,6 +430,7 @@ print(result.aggregate_scores)
 
 ```python
 from nemo_evaluator_sdk import RunConfig, ToolCallingMetric
+from nemo_evaluator.shared.metric_bundles.cloudpickle import CloudpickleMetricBundlePackager
 
 metric = ToolCallingMetric(reference="{{item.tool_calls}}")
 
@@ -462,6 +465,7 @@ job = evaluator.submit(
         }
     ],
     config=RunConfig(parallelism=4),
+    metric_bundle_packager=CloudpickleMetricBundlePackager(),
 )
 job.wait_until_done()
 result = job.get_result()
@@ -568,6 +572,7 @@ print(result.aggregate_scores)
 ```python
 from nemo_evaluator_sdk import RunConfig, Model
 from nemo_evaluator_sdk.metrics.ragas import TopicAdherenceMetric
+from nemo_evaluator.shared.metric_bundles.cloudpickle import CloudpickleMetricBundlePackager
 
 judge_model = Model(
     url="https://integrate.api.nvidia.com/v1/chat/completions",
@@ -591,6 +596,7 @@ job = evaluator.submit(
         }
     ],
     config=RunConfig(parallelism=4),
+    metric_bundle_packager=CloudpickleMetricBundlePackager(),
 )
 job.wait_until_done()
 result = job.get_result()
@@ -762,6 +768,7 @@ print(result.aggregate_scores)
 ```python
 from nemo_evaluator_sdk import RunConfig, Model
 from nemo_evaluator_sdk.metrics.ragas import AgentGoalAccuracyMetric
+from nemo_evaluator.shared.metric_bundles.cloudpickle import CloudpickleMetricBundlePackager
 
 judge_model = Model(
     url="https://integrate.api.nvidia.com/v1/chat/completions",
@@ -791,6 +798,7 @@ job = evaluator.submit(
         }
     ],
     config=RunConfig(parallelism=4),
+    metric_bundle_packager=CloudpickleMetricBundlePackager(),
 )
 job.wait_until_done()
 result = job.get_result()
@@ -917,6 +925,7 @@ print(result.aggregate_scores)
 ```python
 from nemo_evaluator_sdk import RunConfig, Model
 from nemo_evaluator_sdk.metrics.ragas import AgentGoalAccuracyMetric
+from nemo_evaluator.shared.metric_bundles.cloudpickle import CloudpickleMetricBundlePackager
 
 judge_model = Model(
     url="https://integrate.api.nvidia.com/v1/chat/completions",
@@ -954,6 +963,7 @@ job = evaluator.submit(
         }
     ],
     config=RunConfig(parallelism=4),
+    metric_bundle_packager=CloudpickleMetricBundlePackager(),
 )
 job.wait_until_done()
 result = job.get_result()
@@ -1013,6 +1023,7 @@ print(result.aggregate_scores)
 ```python
 from nemo_evaluator_sdk import RunConfig, Model
 from nemo_evaluator_sdk.metrics.ragas import AnswerAccuracyMetric
+from nemo_evaluator.shared.metric_bundles.cloudpickle import CloudpickleMetricBundlePackager
 
 judge_model = Model(
     url="https://integrate.api.nvidia.com/v1/chat/completions",
@@ -1031,6 +1042,7 @@ job = evaluator.submit(
         }
     ],
     config=RunConfig(parallelism=4),
+    metric_bundle_packager=CloudpickleMetricBundlePackager(),
 )
 job.wait_until_done()
 result = job.get_result()
@@ -1043,6 +1055,7 @@ print(result.aggregate_scores)
 ```python
 from nemo_evaluator_sdk import RunConfigOnlineModel, InferenceParams, Model
 from nemo_evaluator_sdk.metrics.ragas import AnswerAccuracyMetric
+from nemo_evaluator.shared.metric_bundles.cloudpickle import CloudpickleMetricBundlePackager
 
 judge_model = Model(
     url="https://integrate.api.nvidia.com/v1/chat/completions",
@@ -1077,6 +1090,7 @@ job = evaluator.submit(
             }
         ]
     },
+    metric_bundle_packager=CloudpickleMetricBundlePackager(),
 )
 
 job.wait_until_done()

docs/evaluator/metrics/job-management.mdx

@@ -21,6 +21,7 @@ from nemo_evaluator.sdk import Evaluator
 from nemo_platform import NeMoPlatform
 from nemo_evaluator_sdk import RunConfig
 from nemo_evaluator_sdk import ExactMatchMetric
+from nemo_evaluator.shared.metric_bundles.cloudpickle import CloudpickleMetricBundlePackager
 
 sdk = NeMoPlatform(
     base_url=os.environ.get("NMP_BASE_URL", "http://localhost:8080"),
@@ -37,6 +38,7 @@ job = evaluator.submit(
         {"expected": "Berlin", "output": "Munich"},
     ],
     config=RunConfig(parallelism=4),
+    metric_bundle_packager=CloudpickleMetricBundlePackager(),
 )
 print("Submitted job:", job.name)
 

docs/evaluator/metrics/llm-as-a-judge.mdx

@@ -298,6 +298,7 @@ For production workloads, submit the same metric and dataset as a durable platfo
 
 ```python
 from nemo_evaluator_sdk import RunConfig, JSONScoreParser, Model, RubricScore, LLMJudgeMetric
+from nemo_evaluator.shared.metric_bundles.cloudpickle import CloudpickleMetricBundlePackager
 
 metric = LLMJudgeMetric(
     model=Model(
@@ -346,6 +347,7 @@ job = evaluator.submit(
         {"input": "What is 2 + 2?", "output": "4"},
     ],
     config=RunConfig(parallelism=8, limit_samples=100),
+    metric_bundle_packager=CloudpickleMetricBundlePackager(),
 )
 print("Submitted job:", job.name)
 

docs/evaluator/metrics/manage-metrics.mdx

@@ -98,6 +98,7 @@ For online evaluations, provide a model or agent target and use the online param
 
 ```python
 from nemo_evaluator_sdk import RunConfig, ExactMatchMetric
+from nemo_evaluator.shared.metric_bundles.cloudpickle import CloudpickleMetricBundlePackager
 
 metric = ExactMatchMetric(reference="{{item.expected}}", candidate="{{item.output}}")
 
@@ -108,6 +109,7 @@ job = evaluator.submit(
         {"expected": "Berlin", "output": "Munich"},
     ],
     config=RunConfig(parallelism=4),
+    metric_bundle_packager=CloudpickleMetricBundlePackager(),
 )
 
 job.wait_until_done()

docs/evaluator/metrics/model-configuration.mdx

@@ -183,7 +183,43 @@ Use plain `RunConfig` for offline evaluations where the dataset already contains
 
 ## Model References
 
-The plugin SDK examples on this page use inline `Model` objects. If your deployment resolves platform model entities into model endpoint details, perform that lookup before constructing the `Model`, then pass the resulting inline model to the metric or request.
+You can supply the evaluation target two ways. Which one is valid depends on whether you run the evaluation locally or submit it as a durable platform job.
+
+### Inline `Model` (required for `evaluator.run(...)`)
+
+`evaluator.run(...)` executes in your local Python process, so it needs the resolved endpoint details inline. Always pass an inline `Model` as the `target` (or as a judge/embeddings field on the metric). If your deployment stores platform model entities, resolve the entity into endpoint details before constructing the `Model`:
+
+```python
+from nemo_evaluator_sdk import Model
+
+model_entity = client.models.retrieve("my-model", workspace="default")
+model = Model(
+    url=client.models.get_model_entity_route_openai_url(model_entity),
+    name="my-model",
+    api_key_secret="NVIDIA_API_KEY",
+)
+
+result = evaluator.run(metric=metric, dataset=dataset, target=model)
+```
+
+### `ModelRef` (supported by `evaluator.submit(...)`)
+
+Durable remote `evaluator.submit(...)` jobs additionally accept a `ModelRef` target. A `ModelRef` names a platform model entity (`workspace/model-name`) and is resolved by the evaluator backend when the job runs, so you do not have to resolve the endpoint yourself. Use this for platform-managed model routing. A `ModelRef` target generates outputs online, so it requires an online run config (`RunConfigOnlineModel`):
+
+```python
+from nemo_evaluator_sdk import ModelRef, RunConfigOnlineModel
+from nemo_evaluator.shared.metric_bundles.cloudpickle import CloudpickleMetricBundlePackager
+
+job = evaluator.submit(
+    metric=metric,
+    dataset=dataset,
+    config=RunConfigOnlineModel(),
+    target=ModelRef(root="default/my-model"),
+    metric_bundle_packager=CloudpickleMetricBundlePackager(),
+)
+```
+
+`ModelRef` is **not** valid for `evaluator.run(...)`; the local runtime cannot resolve a platform entity. Pass an inline `Model` for local runs and either a `Model` or a `ModelRef` for remote submits. See the [Define and Run Custom Python Metrics](/documentation/evaluate-models/tutorials/define-and-run-custom-python-metrics) tutorial for an end-to-end `ModelRef` + `FilesetRef` submit example.
 
 <Note>