fix(tier3): use one overall-score formula

CHANGELOG.md

@@ -12,6 +12,14 @@ All notable changes to SkillEvaluator are documented in this file.
 
 ### Fixed
 
+- Tier 3 now uses one versioned, equal-weight five-dimension overall-score
+  policy across Harbor rewards, pass@k, Skill Lift, reports, comparisons, and
+  best-agent selection. Previously, some paths averaged all six evaluators
+  directly while reports averaged five dimensions, which gave Effectiveness
+  two votes and could reverse the reported lift direction. Current artifacts
+  persist `skill-evaluator-dimension-mean-v1`; legacy and partial artifacts
+  retain their historical semantics
+  (Relates to [#61](https://github.com/NVIDIA/SkillEvaluator/issues/61)).
 - `--llm-verify` now refuses to send file context from paths outside the
   skill root, including `..`, absolute paths, and outbound file symlinks.
 - Gitleaks path allowlist now skips test/example/fixture/mock directories

docs/reports.mdx

@@ -192,25 +192,49 @@ command.
 With `default` or `default_plus_custom` grading, reports lead with five
 dimensions, each answering one question:
 
-| Dimension | Question answered |
-| --- | --- |
-| Security | Is it safe to use? |
-| Correctness | Is the answer correct? |
-| Discoverability | Was the right skill loaded when needed? |
-| Effectiveness | Did the skill help complete the task? |
-| Efficiency | Did it avoid wasted tool or skill usage? |
-
-Dimension scores are 0.0–1.0 rollups of the underlying evaluation signals. The
-verdict bands are fixed: a dimension **passes** at 0.50 or above, is
-**neutral** from 0.40 to below 0.50, and **fails** below 0.40. The judge runs on
-the configured provider model; override it per run as described in
-[Tier 3: Live Evaluation](tier3-live-evaluation.mdx).
+| Dimension | Question answered | Signals and source |
+| --- | --- | --- |
+| Security | Is it safe to use? | `security`, from deterministic trace checks |
+| Correctness | Is the answer correct? | `accuracy`, from the `judge_accuracy` LLM judge |
+| Discoverability | Was the right skill loaded when needed? | `skill_execution`, from deterministic trajectory checks |
+| Effectiveness | Did the skill help complete the task? | `goal_accuracy` and `behavior_check`, from LLM judges |
+| Efficiency | Did it avoid wasted tool or skill usage? | `skill_efficiency`, from deterministic trajectory checks |
+
+Every evaluator and dimension score is on a 0.0 to 1.0 scale. The generated
+standard grader rejects non-finite values and clamps numeric scores with
+`max(0.0, min(1.0, score))`. A failed required judge produces no score instead
+of a numeric zero. The fixed dimension bands are **pass** at 0.50 or above,
+**neutral** from 0.40 to below 0.50, and **fail** below 0.40.
+
+`run_config.json` records the configured judge provider, model, source, and
+catalog-verification status separately from the agent model. `result.json`
+embeds that run configuration alongside the scores. This provenance makes the
+LLM-judged values traceable, but it is not evidence that the judges are
+validated. The repository does not publish inter-rater agreement, calibration
+results, or a comparison against human-labeled ground truth.
+
+The **Quality Score** displayed out of 100 in the HTML report is the Tier 1
+static-analysis score. It is not an input to the Tier 3 dimensions, overall
+score, or Skill Lift.
+
+The standard overall score gives each dimension one equal vote:
+
+```text
+overall = (Security + Correctness + Discoverability + Effectiveness + Efficiency) / 5
+Effectiveness = (goal_accuracy + behavior_check) / 2
+```
+
+This formula is shared by the report, per-attempt pass@k thresholding, Skill
+Lift, comparisons, and best-agent selection. Standard artifacts identify it as
+`skill-evaluator-dimension-mean-v1` in `score_policy` and in the persisted
+attempt policy. Older metric sets and partial historical artifacts keep their
+legacy policy rather than being rescored as current data.
 
 ### Skill Lift
 
-Skill Lift is the with-skill score minus the without-skill baseline — the
-direct measurement of what your skill contributes. Because live agent runs are
-noisy (especially at low attempt counts), small deltas are deliberately kept
+Skill Lift is `with-skill overall score - without-skill overall score`. It uses
+the same score units as both arms and ranges from -1.0 to +1.0. Because live
+agent runs are noisy, especially at low attempt counts, small deltas are kept
 neutral. The verdict bands:
 
 | Lift | Verdict | Read it as |
@@ -236,7 +260,8 @@ see [Custom Graders & Tasks](custom-graders.mdx).
 pass@k is the reliability signal, reported separately from the dimension
 scores. With `--n-attempts k`, each eval case runs k times per arm, and a case
 counts as passed when at least one attempt clears the `--pass-threshold`
-score. Comparing pass@k across arms (in `pass_at_k_lift.json`) tells you
+score using the same overall formula above. Comparing pass@k across arms (in
+`pass_at_k_lift.json`) tells you
 whether the skill makes success more *repeatable*, not just whether the average
 score moved. Each arm also records a case-level 95% Wilson score interval for
 its pass rate. When both arms contain the same identified cases,

docs/tier3-live-evaluation.mdx

@@ -414,13 +414,48 @@ rollups, followed by a compact Artifacts panel pointing at the report and
 output directory. A failed run renders the same structured display instead of
 a bare exception.
 
-| Dimension | Key question | Maps to | Weights |
-| --- | --- | --- | --- |
-| Security | Is it safe to use? | `security` | 1.0 |
-| Correctness | Is the answer correct? | `accuracy` | 1.0 |
-| Discoverability | Was the right skill loaded when needed? | `skill_execution` | 1.0 |
-| Effectiveness | Did the skill help complete the task? | `goal_accuracy` + `behavior_check` | 0.5 + 0.5 |
-| Efficiency | Did it avoid wasted tool or skill usage? | `skill_efficiency` | 1.0 |
+Every evaluator score and dimension score is on a 0.0 to 1.0 scale. The
+generated standard grader rejects non-finite values and clamps numeric scores
+with `max(0.0, min(1.0, score))`. If a required judge fails, the result is
+unscored rather than converted to `0.0`.
+
+| Dimension | Key question | Maps to | Weights | Source |
+| --- | --- | --- | --- | --- |
+| Security | Is it safe to use? | `security` | 1.0 | Deterministic checks of the trace and agent actions |
+| Correctness | Is the answer correct? | `accuracy` | 1.0 | LLM judge (`judge_accuracy`) |
+| Discoverability | Was the right skill loaded when needed? | `skill_execution` | 1.0 | Deterministic checks of the trajectory and tool calls |
+| Effectiveness | Did the skill help complete the task? | `goal_accuracy` + `behavior_check` | 0.5 + 0.5 | LLM judges (`judge_goal_accuracy`, `judge_behavior_check`) |
+| Efficiency | Did it avoid wasted tool or skill usage? | `skill_efficiency` | 1.0 | Deterministic checks of the trajectory and tool calls |
+
+The run's configured judge provider and model are stored under `judge` in
+`run_config.json`, separately from the agent's provider and model. The same run
+configuration is embedded in `result.json`, so a judged score can be traced to
+the configuration that produced it. An allowed provider fallback may use a
+different model for an individual call.
+
+This provenance does not establish that the LLM judgments are correct. The
+repository does not publish inter-rater agreement, calibration results, or a
+comparison against human-labeled ground truth for these judges.
+
+The **Quality Score** shown as a value out of 100 in the HTML report comes from
+Tier 1 static analysis. It is separate from the Tier 3 evaluator scores,
+dimensions, overall score, and Skill Lift.
+
+For standard grading, the overall score is the equal-weight mean of these five
+dimensions:
+
+```text
+overall = (Security + Correctness + Discoverability + Effectiveness + Efficiency) / 5
+Effectiveness = (goal_accuracy + behavior_check) / 2
+```
+
+Effectiveness therefore enters the overall score once; its two source
+evaluators do not receive an extra combined vote. The same aggregate drives
+per-attempt pass@k, Skill Lift, comparison output, and best-agent selection.
+Artifacts record this contract as
+`score_policy: skill-evaluator-dimension-mean-v1`. Legacy metric sets and
+historical partial reports retain their recorded scoring semantics instead of
+being silently reinterpreted.
 
 `token_efficiency` is a standalone report signal, not a dimension source. It
 never changes a dimension score or the overall verdict.
@@ -435,8 +470,9 @@ agent passes only when every dimension passes. The top-level Tier 3 verdict is:
 | Neutral | No agent passes, but at least one has no dimension below 0.40 |
 | Fail | Every successful agent has a dimension below 0.40 |
 
-Skill Lift is the signed difference between the with-skill result and the
-without-skill baseline. Its independent band is **PASS** at +0.05 or above,
+Skill Lift is `with-skill overall score - without-skill overall score`. It uses
+the same score units as the two arms; because it is a difference, it ranges
+from -1.0 to +1.0. Its independent band is **PASS** at +0.05 or above,
 **NEUTRAL** between -0.10 and +0.05, and **FAIL** at -0.10 or below. Lift is
 diagnostic evidence and does not override the every-dimension gate.
 
@@ -455,7 +491,7 @@ The canonical Tier 3 payload embedded by both standalone `evaluate` and
 `validate --agent-eval` reports uses schema version 2.0. A completed
 standard-grading payload includes the summary, five dimensions, per-agent
 results, trials, pass@k, attempt policy, run-owned dataset summary and digest,
-evaluation timestamp, evaluator version, and verdict policy; advisory or
+evaluation timestamp, evaluator version, score policy, and verdict policy; advisory or
 skipped payloads may leave live-evidence sections empty.
 
 Generated `BENCHMARK.md` cards put the verdict first, label each result column

src/skillevaluator/constants.py

@@ -481,6 +481,7 @@
     "overall = mean(Security, Correctness, Discoverability, Effectiveness, Efficiency) "
     "dimensions; Security maps to security evaluator (fallback: behavior_check)"
 )
+DEFAULT_SCORE_POLICY = "skill-evaluator-dimension-mean-v1"
 
 AGENT_EVAL_VERDICT_PASS = "pass"
 AGENT_EVAL_VERDICT_NEUTRAL = "neutral"

src/skillevaluator/evaluation/tier3_report.py

@@ -30,6 +30,7 @@
 from skillevaluator.constants import (
     AGENT_EVAL_EVALUATORS,
     AGENT_EVAL_SCORE_DEFINITION,
+    DEFAULT_SCORE_POLICY,
     DIMENSION_HINTS,
     DIMENSION_MAPPING,
     DIMENSION_VERDICT_NEUTRAL_THRESHOLD,
@@ -218,6 +219,7 @@ def _advisory_agent_eval_payload(
         "dataset_summary": dataset_summary,
         "dataset_digest": None,
         "dataset_digest_algorithm": None,
+        "score_policy": attempt_policy["score_policy"],
         "verdict_policy": verdict_policy,
         "execution_status": "skipped",
         "execution_errors": [message],
@@ -257,6 +259,7 @@ def _advisory_agent_eval_payload(
         "dataset_summary": dataset_summary,
         "dataset_digest": None,
         "dataset_digest_algorithm": None,
+        "score_policy": attempt_policy["score_policy"],
         "verdict_policy": verdict_policy,
         "provenance": {
             "source": "advisory",
@@ -613,8 +616,18 @@ def build_agent_eval_payload(
 
     metric_ids = list(best.get("evaluators", {}).keys())
     metric_labels = _metric_labels(metric_ids)
+    policy_metrics = tuple(metric_ids) if metric_ids else tuple(metrics)
 
-    policy = attempt_policy or _default_attempt_policy()
+    from skillevaluator.tier3.harbor.metrics import score_definition, score_policy_for_metrics
+
+    if attempt_policy:
+        policy = dict(attempt_policy)
+        policy.setdefault("score_definition", score_definition(policy_metrics))
+        policy.setdefault("score_policy", score_policy_for_metrics(policy_metrics))
+    else:
+        policy = _default_attempt_policy()
+        policy["score_definition"] = score_definition(policy_metrics)
+        policy["score_policy"] = score_policy_for_metrics(policy_metrics)
     canonical_trials = _flatten_trials(agent_payloads)
     public_dataset = deduplicate_dataset_entries([entry for entry in (dataset or []) if isinstance(entry, dict)])
     computed_dataset_truth = (
@@ -654,6 +667,7 @@ def build_agent_eval_payload(
         "dataset_summary": dataset_summary,
         "dataset_digest": effective_dataset_digest,
         "dataset_digest_algorithm": effective_dataset_digest_algorithm,
+        "score_policy": policy["score_policy"],
         "verdict_policy": verdict_policy,
         "execution_status": execution_status,
         "execution_errors": execution_errors,
@@ -713,6 +727,7 @@ def build_agent_eval_payload(
         "dataset_summary": dataset_summary,
         "dataset_digest": effective_dataset_digest,
         "dataset_digest_algorithm": effective_dataset_digest_algorithm,
+        "score_policy": policy["score_policy"],
         "verdict_policy": verdict_policy,
         "agents": agent_payloads,
         "dimensions": best_dimensions,
@@ -1071,6 +1086,14 @@ def _serialized_payload_size(payload: dict[str, Any]) -> int:
 def _replace_with_minimal_payload(payload: dict[str, Any], report_budget: _ReportBudget) -> None:
     """Last-resort bounded shape for pathological single-field payloads."""
     summary = payload.get("summary") if isinstance(payload.get("summary"), dict) else {}
+    attempt_policy = payload.get("attempt_policy") if isinstance(payload.get("attempt_policy"), dict) else {}
+    compact_attempt_policy = {
+        key: attempt_policy[key] for key in ("max_attempts", "pass_threshold", "stop_on_pass") if key in attempt_policy
+    }
+    for key, limit in (("score_definition", 1024), ("score_policy", 256)):
+        value = attempt_policy.get(key)
+        if isinstance(value, str):
+            compact_attempt_policy[key] = value[:limit]
     compact_summary = {
         key: value
         for key, value in summary.items()
@@ -1082,13 +1105,16 @@ def _replace_with_minimal_payload(payload: dict[str, Any], report_budget: _Repor
             "overall_lift",
             "environment",
             "runtime_seconds",
+            "score_policy",
             "execution_status",
             "expected_attempts",
             "scored_attempts",
         }
     }
     compact_summary["skill_name"] = str(summary.get("skill_name") or payload.get("skill_name") or "")[:256]
     compact_summary["best_agent"] = str(summary.get("best_agent") or payload.get("best_agent") or "")[:256]
+    raw_score_policy = payload.get("score_policy", summary.get("score_policy"))
+    compact_summary["score_policy"] = raw_score_policy[:256] if isinstance(raw_score_policy, str) else None
     compact_summary["agents_run"] = [str(name)[:256] for name in (summary.get("agents_run") or [])[:64]]
     compact_summary["execution_errors"] = [str(error)[:1024] for error in (summary.get("execution_errors") or [])[:16]]
 
@@ -1109,6 +1135,8 @@ def _replace_with_minimal_payload(payload: dict[str, Any], report_budget: _Repor
         "expected_attempts": payload.get("expected_attempts", 0),
         "scored_attempts": payload.get("scored_attempts", 0),
         "runtime_seconds": payload.get("runtime_seconds", 0.0),
+        "score_policy": compact_summary["score_policy"],
+        "attempt_policy": compact_attempt_policy,
         "agents": {},
         "dimensions": [],
         "evaluators": {},
@@ -1157,6 +1185,12 @@ def _build_agent(
     metrics: list[str],
     model: str | None,
 ) -> dict[str, Any]:
+    from skillevaluator.tier3.harbor.metrics import (
+        LEGACY_METRICS,
+        canonical_dimension_mean,
+        overall_score_from_metrics,
+    )
+
     with_scores = info.get("with_skill") or {}
     without_scores = info.get("without_skill") or {}
     lift_data = info.get("lift") or {}
@@ -1174,8 +1208,18 @@ def _build_agent(
         info.get("dimensions_with_skill") or {},
         info.get("dimensions_without_skill") or {},
     )
-    overall_ws = _mean([d["with_skill"] for d in dimensions])
-    overall_bl = _mean([d["baseline"] for d in dimensions])
+    with_dimension_values = [d["with_skill"] for d in dimensions]
+    baseline_dimension_values = [d["baseline"] for d in dimensions]
+    if tuple(metrics) == LEGACY_METRICS:
+        overall_ws = overall_score_from_metrics(with_scores, LEGACY_METRICS)
+        overall_bl = overall_score_from_metrics(without_scores, LEGACY_METRICS)
+    else:
+        overall_ws = canonical_dimension_mean(with_dimension_values)
+        overall_bl = canonical_dimension_mean(baseline_dimension_values)
+    if overall_ws is None:
+        overall_ws = _mean(with_dimension_values)
+    if overall_bl is None:
+        overall_bl = _mean(baseline_dimension_values)
     if overall_ws is None and not metrics and with_quality_available:
         overall_ws = _finite_float(info.get("overall_with_skill"))
         if overall_ws is None and info.get("rewards_complete") is not False:
@@ -2544,6 +2588,11 @@ def _read_attempt_policy(run_dir: Path) -> dict[str, Any]:
             loaded = json.loads(policy_file.read_text(encoding="utf-8"))
             if isinstance(loaded, dict):
                 policy.update(loaded)
+                if "score_policy" not in loaded:
+                    policy.pop("score_policy", None)
+                return policy
+    policy.pop("score_definition", None)
+    policy.pop("score_policy", None)
     return policy
 
 
@@ -2668,6 +2717,7 @@ def _default_attempt_policy() -> dict[str, Any]:
         "pass_threshold": 0.50,
         "stop_on_pass": False,
         "score_definition": AGENT_EVAL_SCORE_DEFINITION,
+        "score_policy": DEFAULT_SCORE_POLICY,
     }
 
 

src/skillevaluator/tier3/commands.py

@@ -36,7 +36,7 @@
     HARBOR_AGENTS_SUPPORTED,
     canonical_agent_name,
 )
-from skillevaluator.tier3.harbor.metrics import DEFAULT_METRICS, LEGACY_METRICS
+from skillevaluator.tier3.harbor.metrics import DEFAULT_METRICS, LEGACY_METRICS, overall_score_from_metrics
 from skillevaluator.tier3.harbor.progress import (
     NullProgressReporter,
     ProgressEvent,
@@ -1045,12 +1045,10 @@ def compare_results(skill_path: Path, *, results_dir: Path | None = None) -> int
     table.add_row(*[""] * (1 + sum(2 if agent in agent_without else 1 for agent in agents)))
     overall_row: list[str | Text] = [Text("Overall", style="bold")]
     for agent in agents:
-        with_avg = sum(_safe_score(agent_with[agent], metric) for metric in overall_metrics) / len(overall_metrics)
+        with_avg = _overall_score_for_display(agent_with[agent], overall_metrics)
         overall_row.append(Text(f"{with_avg:.2f}", style=f"bold {_score_style(with_avg)}"))
         if agent in agent_without:
-            without_avg = sum(_safe_score(agent_without[agent], metric) for metric in overall_metrics) / len(
-                overall_metrics
-            )
+            without_avg = _overall_score_for_display(agent_without[agent], overall_metrics)
             delta = with_avg - without_avg
             delta_text = f"+{delta:.2f}" if delta > 0 else f"{delta:.2f}"
             delta_style = "bold green" if delta > 0 else ("bold red" if delta < 0 else "bold dim")
@@ -1108,6 +1106,14 @@ def _safe_score(scores: dict[str, float], metric: str) -> float:
     return float(value) if isinstance(value, int | float) else 0.0
 
 
+def _overall_score_for_display(scores: dict[str, float], metrics: tuple[str, ...]) -> float:
+    """Use the canonical policy while preserving partial historical displays."""
+    score = overall_score_from_metrics(scores, metrics)
+    if score is not None:
+        return score
+    return sum(_safe_score(scores, metric) for metric in metrics) / len(metrics)
+
+
 def _score_style(score: float) -> str:
     if score >= 0.8:
         return "green"