feat: add validate_context_set MCP tool - #173
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces a validation mechanism for ContextSet JSON files, adding a new validate_context_set MCP tool, a context_validator.py module, and updating relevant workflow guides and test suites. Feedback on the changes includes specifying encoding="utf-8" when opening files to prevent platform-specific decoding issues, and catching broader OSError exceptions in the MCP tool to gracefully handle permission or directory errors instead of crashing.
de125c7 to
e312b10
Compare
e312b10 to
b0fc1db
Compare
|
/gcbrun |
|
If the agent is calling mutate_context_set to create the context, then do we expect validation to fail? If not, then the value of validation code seems to be when mutate_context_set is not called by the agent to create context. In that case, if the agent ignores directive on how to create context, why do we expect it to obey directive on how to create the context? |
| @@ -0,0 +1,173 @@ | |||
| import json | |||
There was a problem hiding this comment.
How do you decide what's the right validation to add here?
Why I ask: I don't think we should aim to be 100% comprehensive but to target the key ways we've seen the agent fail. Are these: bad json strcuture, dups, missing $value param issues that you've run into?
We do have a list of issues that we've observed in created context, from our debugging work.
There was a problem hiding this comment.
This is a basic version of validation. The core goal is to only solve one problem: wrong JSON format that Erin has encountered along with some other possible cases that are easy to catch. We can add more based on the criticality later.
We don't make assumptions where the context set comes from. The core work here is just to validate whether a context has the right JSON format we expected and we just give the agent a hint you should call this method to verify, there is no 100% guarantee and that's the nature of all agent I believe. Which part are we thinking should be changed? |
Summary
validate_context_setMCP tool that reports structural and convention issues in a ContextSet JSON file. Reports only — fixes are applied by the agent viamutate_context_set.$valueparameter invalue_search.query.bootstrap,hillclimbPhase 2, and thecontext-generation-guidemanual authoring workflow.