Add workflow composer validation and warnings#10
Draft
sezallagwal wants to merge 2 commits into
Draft
Conversation
dhairyashiil
requested changes
Jul 1, 2026
dhairyashiil
left a comment
dhairyashiil
left a comment
Collaborator
There was a problem hiding this comment.
Hi Sezal, could we improve the sampling/data-flow validation in this PR?
A plain text sampling step can currently be used later like it returned JSON, for example {{steps.rank.safe}}. That composes successfully, but if the sampling result is just text at runtime, the field access resolves to an empty value.
Could we make the composer catch this? A few possible options:
- treat non-JSON sampling output as a string and reject field access on it
- require
RESPONSE_FORMAT jsonor a response schema before allowingsteps.someSampling.field - or infer JSON mode when downstream steps clearly access fields
The main thing is: if a later step expects structured data, the composer should make that explicit instead of silently producing empty values.
Sampling results are plain text unless the step is in JSON mode (explicit responseFormat: json or JSON intent in the prompt), where the runtime parses them into an object. Field access is only valid against the object form. - Add shared detectJsonIntent heuristic (workflow/json-intent.ts) as the single source of truth for JSON-mode detection across compose-time validation and the runtime sampling executor. - validateDataFlowTypes now resolves sampling output type per-step and rejects field access on plain-text sampling with an actionable error. - Implement inferSamplingResponseSchemas: infer responseSchema for JSON-mode sampling from downstream field access, warning (SAMPLING_SCHEMA_MISMATCH) when a consumed field is absent from the prompt. - Add unit tests for data-flow validation, schema inference, and json-intent. Addresses review feedback on silent empty-value resolution when a text sampling result is used like structured JSON.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Depends on
#9