Add BigQuery support to evaluation config generation - #194
Conversation
Implements the BigQuery portion of the Wave2 FR: - Add BigQueryConfigGenerator mapping tools.yaml bigquery sources to Evalbench db_config and GDA BigQueryTableReferences model config - Register the generator in the evaluate_generator factory map - Add bigquery.md connection reference template for the init skill - Attach agent_context_reference dynamically since the field is not yet in the public GDA SDK (restricted-visibility rollout)
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a new BigQueryConfigGenerator to support BigQuery configuration topologies, along with corresponding reference documentation and unit tests. The review feedback highlights a potential runtime AttributeError when dynamically attaching agent_context_reference if the AgentContextReference class itself is missing from the public SDK, and provides a robust code suggestion to handle this gracefully.
| # The agent_context_reference field on BigQueryTableReferences is not | ||
| # yet available in the public google-cloud-geminidataanalytics SDK | ||
| # (restricted-visibility rollout; see the Wave2 FR). Attach it | ||
| # dynamically so this generator works with internal SDK builds and | ||
| # degrades gracefully on public ones. | ||
| if "agent_context_reference" in { | ||
| f.name for f in type(bq_references).pb(bq_references).DESCRIPTOR.fields | ||
| }: | ||
| bq_references.agent_context_reference = gda.AgentContextReference( | ||
| context_set_id=context_set_id | ||
| ) |
There was a problem hiding this comment.
If agent_context_reference is not yet available in the public SDK, it is highly likely that the AgentContextReference class itself is also missing or restricted in that version. Referencing gda.AgentContextReference directly will raise an AttributeError at runtime, defeating the graceful degradation goal.
To make this fully robust, we should dynamically retrieve the class using getattr and check its existence. Additionally, we can use bq_references._pb to access the underlying protobuf message, which is more consistent with how it is done elsewhere in the codebase (e.g., query_context._pb in BaseDBConfigGenerator).
| # The agent_context_reference field on BigQueryTableReferences is not | |
| # yet available in the public google-cloud-geminidataanalytics SDK | |
| # (restricted-visibility rollout; see the Wave2 FR). Attach it | |
| # dynamically so this generator works with internal SDK builds and | |
| # degrades gracefully on public ones. | |
| if "agent_context_reference" in { | |
| f.name for f in type(bq_references).pb(bq_references).DESCRIPTOR.fields | |
| }: | |
| bq_references.agent_context_reference = gda.AgentContextReference( | |
| context_set_id=context_set_id | |
| ) | |
| # The agent_context_reference field on BigQueryTableReferences is not | |
| # yet available in the public google-cloud-geminidataanalytics SDK | |
| # (restricted-visibility rollout; see the Wave2 FR). Attach it | |
| # dynamically so this generator works with internal SDK builds and | |
| # degrades gracefully on public ones. | |
| agent_context_ref_cls = getattr(gda, "AgentContextReference", None) | |
| if agent_context_ref_cls and "agent_context_reference" in { | |
| f.name for f in bq_references._pb.DESCRIPTOR.fields | |
| }: | |
| bq_references.agent_context_reference = agent_context_ref_cls( | |
| context_set_id=context_set_id | |
| ) |
The init skill's database-type list, the generate_evalbench_configs tool docstring, and the evaluate skill's per-database references all enumerate supported types explicitly, so the BigQuery generator was unreachable from the agent workflows without these updates.
BigQuery connections are scoped to a project with no default dataset at query time, so SQL referencing bare table names fails with 'Table not found'. Add BigQuery dialect references (template, facet, value search) to the context-generation-guide mandating project.dataset.table qualification, and call the rule out in the dataset-generation drafting chain of thought.
| When tasked with creating new pairs, using the generation plan as the guideline and north star. For *every* generated pair, execute the following internal Chain of Thought: | ||
|
|
||
| 1. **Draft SQL (Schema-First):** Write syntactically perfect, dialect-compliant SQL using prioritized tables and conditions. Make sure to adhere to the technical schema and generation plan. Avoid inventing columns or tables based on business documents alone. | ||
| 1. **Draft SQL (Schema-First):** Write syntactically perfect, dialect-compliant SQL using prioritized tables and conditions. Make sure to adhere to the technical schema and generation plan. Avoid inventing columns or tables based on business documents alone. For **BigQuery** sources, every table reference must be fully qualified as `` `project`.`dataset`.`table` `` (taken from the `kind: source` block in `tools.yaml`) — BigQuery has no default dataset at query time, so bare table names fail with "Table not found". |
There was a problem hiding this comment.
We try to isolate dialect/product specific things to dedicated dialect files.
Do you need this here or can it go into references/template/bigquery.md used by context-generation-guide?
2 participants
Implements the BigQuery portion of the Wave2 FR: