Fix HTML report template guidance, numeric metric aggregation, and output path handling

[Copilot]

This update addresses three report-generation edge cases in HTMLReport: misleading guidance when the built-in template is missing, unsafe overall-score aggregation over mixed metric types, and ambiguous handling of output paths without/with non-.html suffixes. The changes make error messaging and file-path behavior explicit while preventing TypeError from non-numeric metric values.

• Template resolution error messaging

  • Updated the missing built-in-template exception text to treat it as a packaging/install integrity issue (or custom template path issue), not a missing jinja2 dependency.

• Overall score computation hardening

  • overall_score now averages only numeric metric values (int/float, excluding bool) from metrics_summary.
  • Non-numeric entries (e.g., strings/dicts) are ignored instead of breaking report generation.

• Output path normalization

  • generate_to_file() now:
    • appends report.html only when no suffix is present (directory-like input),
    • rewrites non-.html suffixes to .html for file-like inputs.

• Focused tests added/updated

  • Added coverage for:
    • non-.html suffix replacement,
    • numeric-only overall score behavior with mixed metric types,
    • updated template-missing guidance message.

path = Path(output_path)
if path.suffix == "":
    path = path / "report.html"
elif path.suffix.lower() != ".html":
    path = path.with_suffix(".html")

numeric_metric_values = [
    value for value in metrics.values()
    if isinstance(value, (int, float)) and not isinstance(value, bool)
]
overall_score = (
    sum(numeric_metric_values) / len(numeric_metric_values)
    if numeric_metric_values else None
)

Original prompt

Please apply the following diffs and create a pull request.
Once the PR is ready, give it a title based on the messages of the fixes being applied.

[{"message":"The error message suggests installing jinja2 as a fix for a missing template file, but a missing built-in template is a packaging/installation issue, not a missing jinja2 dependency. The message should distinguish between the two causes and suggest checking the package installation instead.","fixFiles":[{"filePath":"openagent_eval/reports/html.py","diff":"diff --git a/openagent_eval/reports/html.py b/openagent_eval/reports/html.py\n--- a/openagent_eval/reports/html.py\n+++ b/openagent_eval/reports/html.py\n@@ -58,7 +58,8 @@\n \n         raise FileNotFoundError(\n             f\"HTML template not found at {builtin_path}. \"\n-            \"Install jinja2 and ensure the template file exists.\"\n+            \"Ensure the package is installed correctly and includes \"\n+            \"openagent_eval/reports/templates/report.html, or provide a valid custom template path.\"\n         )\n \n     def _render_template(self, context: dict[str, Any]) -> str:\n"}]},{"message":"`sum(metrics.values())` assumes all metric values are numeric. If any value in `metrics_summary` is non-numeric (e.g., a string or nested dict), this will raise a `TypeError`. Consider filtering or validating that values are numeric before summing.","fixFiles":[{"filePath":"openagent_eval/reports/html.py","diff":"diff --git a/openagent_eval/reports/html.py b/openagent_eval/reports/html.py\n--- a/openagent_eval/reports/html.py\n+++ b/openagent_eval/reports/html.py\n@@ -96,8 +96,13 @@\n \n         # Compute overall score\n         metrics = summary.get(\"metrics_summary\", {})\n+        numeric_metric_values = [\n+            value for value in metrics.values()\n+            if isinstance(value, (int, float)) and not isinstance(value, bool)\n+        ]\n         overall_score = (\n-            sum(metrics.values()) / len(metrics) if metrics else None\n+            sum(numeric_metric_values) / len(numeric_metric_values)\n+            if numeric_metric_values else None\n         )\n \n         # Format results for template\n"}]},{"message":"If `output_path` is a file path without an `.html` extension (e.g., `report.txt` or `report`), the code treats it as a directory and appends `report.html` to it, which is likely not the intended behavior. Consider checking whether the path is meant to be a file or directory more explicitly, such as checking if the path has a suffix or using `path.is_dir()`.","fixFiles":[{"filePath":"openagent_eval/reports/html.py","diff":"diff --git a/openagent_eval/reports/html.py b/openagent_eval/reports/html.py\n--- a/openagent_eval/reports/html.py\n+++ b/openagent_eval/reports/html.py\n@@ -142,8 +142,10 @@\n             Path to the written file.\n         \"\"\"\n         path = Path(output_path)\n-        if not str(path).endswith(\".html\"):\n+        if path.suffix == \"\":\n             path = path / \"report.html\"\n+        elif path.suffix.lower() != \".html\":\n+            path = path.with_suffix(\".html\")\n         path = self._ensure_output_dir(path)\n         content = self.generate(report)\n         path.write_text(content, encoding=\"utf-8\")\n"}]}]

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[himanshu231204]

done