Skip to content

feat(dea): parameterize agent_name, env, domain to use different agent types support - #497

Merged
graceqi-g merged 1 commit into
GoogleCloudPlatform:mainfrom
minzznguyen:parameterize_agent_types
Jul 17, 2026
Merged

graceqi-g merged 1 commit into
GoogleCloudPlatform:mainfrom
minzznguyen:parameterize_agent_types

Conversation

@minzznguyen

@minzznguyen minzznguyen commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Description

  • Dynamic Endpoint & Persona Parameterization: Supports env (staging vs prod), agent_name (sparkagent vs dataengineeringagent), and optional domain host override in DataEngineeringAgentGenerator.
  • A2A Header Extension Support: Registers AGENT_TYPE_URI extension (https://geminidataanalytics.googleapis.com/a2a/extensions/agenttype/v1) and maps sparkagent to string enum "SPARK_AGENT" in request metadata.
  • Configurations & Docs: Added datasets/dea-tools/example_model_config.yaml and updated example_run_config.yaml and README.md.
  • Verification: Verified end-to-end against live staging Spark Agent endpoints, passed 100% integration tests (pytest), and exported outputs to BigQuery (bq-dataworkeragent-test.evalbench).

Testing

  • Ran a local boq server and test the endpoints - able to make the connection - video here
  • Change agent type to spark_agent and was able to make connection to this agent type - video here

IsmailMehdi
IsmailMehdi previously approved these changes Jul 16, 2026
@@ -1,6 +1,6 @@
orchestrator: dea
dataset_format: dea-format
model_config: datasets/model_configs/gcp_data_engineering_agent_model.yaml

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it looks like the other datasets keep the model config in the datasets/model_configs folder, should we instead update the existing config?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

addressed

"gcpResourceId": target_workspace
}
# Configure Agent Type extension
if self.agent_name.lower() in ("sparkagent", "spark_agent", "spark"):

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since we are only accepting sparkagent or dataengieneringagent as specified in the readme, let's not allow/check other options

we can also refactor this into a match-case statement to improve readability slightly

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

addressed

@IsmailMehdi

Copy link
Copy Markdown
Collaborator

/gcbrun

@minzznguyen
minzznguyen force-pushed the parameterize_agent_types branch 2 times, most recently from 0801348 to bff4707 Compare July 16, 2026 20:45
IsmailMehdi
IsmailMehdi previously approved these changes Jul 16, 2026
@minzznguyen
minzznguyen force-pushed the parameterize_agent_types branch 2 times, most recently from 28b5ed0 to 6a22fef Compare July 16, 2026 22:58
Comment thread datasets/dea-tools/README.md Outdated
env: local # Target environment: "local", "staging", or "prod" (defaults to "prod")
port: 9876 # Required port for local Boq servers when env="local"
url_agent_name: sparkagent # URL path parameter appended to /v1/a2a/.../agents/{url_agent_name}
agent_type_uri: SPARK_AGENT # Persona string attached to AGENT_TYPE_URI metadata extension

@graceqi-g graceqi-g Jul 17, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

instead of making agent_type_uri a separate config, let's keep a dictionary which takes the agent name and outputs what agent type uri value to specify in the extension, since it is 1 to 1

we can change the url_agent_name config to agent_type to make the association clear

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

addressed

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also changed the name env to mode_env since it is making collision with the env config in run config since the two configs get merged in evaluator

)

if env == "local":
port = querygenerator_config.get("port")

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can have port default to the default port boq uses, i think it's 9876

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

addressed

@minzznguyen
minzznguyen force-pushed the parameterize_agent_types branch 4 times, most recently from 59cd848 to 0222eb1 Compare July 17, 2026 15:47
Comment thread evalbench/generators/models/gcp_data_engineering_agent.py
Comment thread evalbench/test/gcp_data_engineering_agent_test.py

if env == "local":
port = querygenerator_config.get("port", 9876)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: remove this empty line

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

addressed

@minzznguyen
minzznguyen force-pushed the parameterize_agent_types branch from 0222eb1 to 3cb2b42 Compare July 17, 2026 18:29
@graceqi-g

Copy link
Copy Markdown
Collaborator

/gcbrun

@graceqi-g
graceqi-g merged commit 30a065f into GoogleCloudPlatform:main Jul 17, 2026
9 checks passed
g-lynnzee pushed a commit to g-lynnzee/evalbench that referenced this pull request Jul 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants