Skip to content

examples: simplify agent_improvement_cycle and self_evolving_agent_demo now that rubrics, golden matching, and evolution hooks are in core (#431/#432/#395) #482

Description

@evekhm

Now that the canonical evaluation rubrics (#431), golden Q&A matching (#432), and the skill-evolution host hooks (#395) live in core, two older examples still carry hand-rolled copies of that functionality. examples/skill_evolution_lab/ already imports the engine and scores via quality_report.py — no changes needed there.

examples/agent_improvement_cycle (~130 removable lines)

  • agent_improvement/prompts.py JUDGE_PROMPT and the {tool_check}/{tool_fail_rule} slots in eval_runner.py are lossy paraphrases of the canonical response_usefulness and tool_usage rubrics in src/bigquery_agent_analytics/evaluation_rubrics.py. Replace the bespoke judge_case() judge call with build_metrics() + Client.evaluate_categorical.
  • improver_agent.py build_report_from_eval_results() exists only to fake canonical categories back out of a binary pass flag — delete once the judge emits canonical categories.
  • improver_agent.py hand-partitions sessions and hand-formats trajectories; replace with skill_evolution.partition_trajectories() / format_trajectory() (the latter now renders tool calls and correction evidence).

Open decision: the canonical response_usefulness has three categories; the LoopAgent exit condition keys off a binary pass. A partial → pass/fail mapping policy must be chosen (mapping partial to fail makes some currently-passing cycles iterate longer). Also: canonical judging expects traces in BigQuery, but run_eval.py --golden is explicitly a no-BQ path — either keep a local-conversation judging path or accept a BQ dependency on that flag.

examples/self_evolving_agent_demo (~100 lines net)

  • analyze_and_evolve.py is a second, weaker evolution engine: _generate_candidate_prompt() (one-shot rewriter), the hand-rolled promotion gate, and _write_prompt_diff() are all superseded by evolve_skill() / select_candidate(..., incumbent_score=) / evolve_skill(artifacts_dir=...).
  • analytics/session_metrics.py load_quality_summary() aggregates a deterministic quality_passed stand-in for response_usefulness+tool_usage, superseded by the canonical rubric. (run_sdk_evaluators() is already correct SDK usage; keep.)

Open decision: this demo's narrative is operational-waste-driven evolution (token waste, tool-call overuse), not quality-failure analysis. Routing it through evolve_skill() risks flattening that distinct story — confirm intent before refactoring. The report-shape adapter it needs (quality_passedsessions[].metrics.response_usefulness.category) is the same bridge agent_improvement_cycle has; write it once, share it.

examples/README.md

Three demos now tell overlapping "agent improves its own prompt" stories with no stated differentiator. Add one line each naming what is distinct: LoopAgent tool-driven loop / operational-waste signal / golden-graded skill evolution.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions