Skip to content

sample_quality_report_session.md

Latest commit

 

History

History
148 lines (130 loc) · 8.03 KB

sample_quality_report_session.md

File metadata and controls

148 lines (130 loc) · 8.03 KB
Raw

Single-Session Quality Report

Console output generated by python scripts/quality_report.py --session conv_5d77036b --tag-turns.

When --session is used, all 7 metrics are shown with full justifications (verbose mode). The execution trace is fetched automatically from BigQuery and displayed after the summary.

──────────────────────────────────────────────────────────────────────
  MEANINGFUL Sessions (showing 1 of 1)
──────────────────────────────────────────────────────────────────────

  Session:     conv_5d77036b  → policy_agent
  Question:    My onboarding packet says that Juneteenth is NOT a company
               holiday. Can you please check the official policy?
  Response:    "Thank you for bringing this to my attention and for asking
               me to verify! I've looked up the official company policy for
               holidays. Based on the 2026 holiday schedule, Juneteenth
               (June 19th) is **not** listed as a company holiday. It seems
               your onboarding packet has the most accurate information in
               this case."
  Usefulness:    ✅ HELPFUL
  Reason:        The final response from the policy_agent correctly answers
                 the user's question after the correction, providing specific
                 and actionable information about Juneteenth not being a
                 company holiday based on the official policy.
  Grounding:     ✅ GROUNDED
  Reason:        The policy_agent explicitly states it looked up the official
                 company policy and based its answer on the 2026 holiday
                 schedule, indicating tool usage.
  Dimensions:    Correctness: ✅ CORRECT | Tool Usage: ✅ PROPER |
                 Specificity: ✅ SPECIFIC | Scope: ✅ COMPLIANT |
                 First-Time Right: ❌ CORRECTION NEEDED

======================================================================
QUALITY SUMMARY
======================================================================
  Total sessions evaluated : 1
  Meaningful               : 1
  Declined (out-of-scope)  : 0
  Partial                  : 0
  Unhelpful                : 0
  Unhelpful rate           : 0.0%

  Quality Dimensions (0-2 scale):
    Correctness         : 2.00 / 2.00  ##################################################
    Tool Usage          : 2.00 / 2.00  ##################################################
    Specificity         : 2.00 / 2.00  ##################################################
    Scope               : 2.00 / 2.00  ##################################################
    First-Time Right    : 0.00 / 2.00

  Multi-Turn Efficiency:
    Avg user turns       : 2.0
    Avg tool calls       : 2.0
    Multi-turn sessions  : 1
    Correction rate      : 100.0%
    Verification rate    : 0.0%

  Category Distributions:

  [response_usefulness]
    ✅ HELPFUL         :    1  (100.0%) ##################################################

  [task_grounding]
    ✅ GROUNDED        :    1  (100.0%) ##################################################

  Execution Details:
    execution_mode: ai_generate
    elapsed_seconds: 23.4
    project: my-gcp-project
    dataset: agent_logs.agent_events
    location: us-central1
    eval_model: gemini-2.5-flash
    time_period: all
    limit: 100

======================================================================

======================================================================
EXECUTION TRACE
======================================================================
Session: conv_5d77036b
Time: 17:37:54  Total: 1.1min
──────────────────────────────────────────────────────────────────────
├── knowledge_supervisor > USER_MESSAGE_RECEIVED
├── knowledge_supervisor > INVOCATION_STARTING
├── knowledge_supervisor > INVOCATION_COMPLETED [14.7s]
│   ├── knowledge_supervisor > AGENT_STARTING
│   └── knowledge_supervisor > AGENT_COMPLETED [2.1s]
│       ├── knowledge_supervisor > LLM_REQUEST
│       └── knowledge_supervisor > LLM_RESPONSE [2.0s, ttft=2.0s]
├── knowledge_supervisor > USER_MESSAGE_RECEIVED
├── knowledge_supervisor > INVOCATION_STARTING
└── knowledge_supervisor > INVOCATION_COMPLETED [1.0min]
    ├── knowledge_supervisor > AGENT_STARTING
    └── knowledge_supervisor > AGENT_COMPLETED [1.0min]
        ├── knowledge_supervisor > LLM_REQUEST
        ├── knowledge_supervisor > LLM_RESPONSE [5.5s, ttft=5.5s]
        ├── knowledge_supervisor > TOOL_STARTING (transfer_to_agent)
        ├── knowledge_supervisor > TOOL_COMPLETED (transfer_to_agent) [0ms]
        ├── policy_agent > AGENT_STARTING
        └── policy_agent > AGENT_COMPLETED [56.0s]
            ├── policy_agent > LLM_REQUEST
            ├── policy_agent > LLM_RESPONSE [20.2s, ttft=20.2s]
            ├── policy_agent > TOOL_STARTING (lookup_company_policy)
            ├── policy_agent > TOOL_COMPLETED (lookup_company_policy) [0ms]
            ├── policy_agent > LLM_REQUEST
            └── policy_agent > LLM_RESPONSE [35.7s, ttft=35.7s]

──────────────────────────────────────────────────────────────────────
  SUB-TRAJECTORY SEGMENTATION
──────────────────────────────────────────────────────────────────────

  ❌ pre_correction_1 (turns 0-1) → wrong
  ├── knowledge_supervisor > USER_MESSAGE_RECEIVED
  ├── knowledge_supervisor > INVOCATION_STARTING
  └── knowledge_supervisor > INVOCATION_COMPLETED [14.7s]
      ├── knowledge_supervisor > AGENT_STARTING
      └── knowledge_supervisor > AGENT_COMPLETED [2.1s]
          ├── knowledge_supervisor > LLM_REQUEST
          └── knowledge_supervisor > LLM_RESPONSE [2.0s, ttft=2.0s]

  ✅ post_correction_1 (turns 2-3) → recovered
  ├── knowledge_supervisor > USER_MESSAGE_RECEIVED
  ├── knowledge_supervisor > INVOCATION_STARTING
  └── knowledge_supervisor > INVOCATION_COMPLETED [1.0min]
      ├── knowledge_supervisor > AGENT_STARTING
      └── knowledge_supervisor > AGENT_COMPLETED [1.0min]
          ├── knowledge_supervisor > LLM_REQUEST
          ├── knowledge_supervisor > LLM_RESPONSE [5.5s, ttft=5.5s]
          ├── knowledge_supervisor > TOOL_STARTING (transfer_to_agent)
          ├── knowledge_supervisor > TOOL_COMPLETED (transfer_to_agent) [0ms]
          ├── policy_agent > AGENT_STARTING
          └── policy_agent > AGENT_COMPLETED [56.0s]
              ├── policy_agent > LLM_REQUEST
              ├── policy_agent > LLM_RESPONSE [20.2s, ttft=20.2s]
              ├── policy_agent > TOOL_STARTING (lookup_company_policy)
              ├── policy_agent > TOOL_COMPLETED (lookup_company_policy) [0ms]
              ├── policy_agent > LLM_REQUEST
              └── policy_agent > LLM_RESPONSE [35.7s, ttft=35.7s]
======================================================================

The execution trace reveals:

  • Turn 1 (wrong): The supervisor answered directly from LLM knowledge (no routing, no tool call) — incorrectly stating Juneteenth is a holiday
  • Turn 2 (recovered): After user correction, the supervisor routed via transfer_to_agent to the policy_agent, which called lookup_company_policy and returned the correct answer

The sub-trajectory segmentation splits the trace at the correction boundary, making it easy to see what changed between the failed and recovered attempts.