Skip to content

feat: add validate_context_set MCP tool - #173

Open
wangauone wants to merge 1 commit into
mainfrom
feat/validate-context-set
Open

feat: add validate_context_set MCP tool#173
wangauone wants to merge 1 commit into
mainfrom
feat/validate-context-set

Conversation

@wangauone

Copy link
Copy Markdown
Collaborator

Summary

  • New validate_context_set MCP tool that reports structural and convention issues in a ContextSet JSON file. Reports only — fixes are applied by the agent via mutate_context_set.
  • v1 checks: invalid JSON (with a snippet of the offending text), Pydantic schema mismatch, exact-duplicate items within an array, and missing $value parameter in value_search.query.
  • Wired a "Validate" step into every place a ContextSet is produced: bootstrap, hillclimb Phase 2, and the context-generation-guide manual authoring workflow.

@wangauone
wangauone requested review from g-lynnzee and helloeve June 27, 2026 00:08

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a validation mechanism for ContextSet JSON files, adding a new validate_context_set MCP tool, a context_validator.py module, and updating relevant workflow guides and test suites. Feedback on the changes includes specifying encoding="utf-8" when opening files to prevent platform-specific decoding issues, and catching broader OSError exceptions in the MCP tool to gracefully handle permission or directory errors instead of crashing.

Comment thread src/google/cloud/db_context_enrichment/common/context_validator.py Outdated
Comment thread src/google/cloud/db_context_enrichment/main.py Outdated
@wangauone
wangauone force-pushed the feat/validate-context-set branch from de125c7 to e312b10 Compare June 27, 2026 00:24
@wangauone
wangauone force-pushed the feat/validate-context-set branch from e312b10 to b0fc1db Compare June 27, 2026 00:26
@wangauone

Copy link
Copy Markdown
Collaborator Author

/gcbrun

@g-lynnzee

Copy link
Copy Markdown
Collaborator

If the agent is calling mutate_context_set to create the context, then do we expect validation to fail?

If not, then the value of validation code seems to be when mutate_context_set is not called by the agent to create context. In that case, if the agent ignores directive on how to create context, why do we expect it to obey directive on how to create the context?

@@ -0,0 +1,173 @@
import json

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How do you decide what's the right validation to add here?

Why I ask: I don't think we should aim to be 100% comprehensive but to target the key ways we've seen the agent fail. Are these: bad json strcuture, dups, missing $value param issues that you've run into?

We do have a list of issues that we've observed in created context, from our debugging work.

@wangauone wangauone Jun 30, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a basic version of validation. The core goal is to only solve one problem: wrong JSON format that Erin has encountered along with some other possible cases that are easy to catch. We can add more based on the criticality later.

@wangauone

Copy link
Copy Markdown
Collaborator Author

If the agent is calling mutate_context_set to create the context, then do we expect validation to fail?

If not, then the value of validation code seems to be when mutate_context_set is not called by the agent to create context. In that case, if the agent ignores directive on how to create context, why do we expect it to obey directive on how to create the context?

We don't make assumptions where the context set comes from. The core work here is just to validate whether a context has the right JSON format we expected and we just give the agent a hint you should call this method to verify, there is no 100% guarantee and that's the nature of all agent I believe.

Which part are we thinking should be changed?

@wangauone
wangauone requested a review from g-lynnzee June 30, 2026 18:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants