Skip to content

Add end-to-end evaluation tests and project documentation#15

Draft
sezallagwal wants to merge 3 commits into
feature/code-generationfrom
feature/evaluation-documentation
Draft

Add end-to-end evaluation tests and project documentation#15
sezallagwal wants to merge 3 commits into
feature/code-generationfrom
feature/evaluation-documentation

Conversation

@sezallagwal

Copy link
Copy Markdown
Collaborator

Summary

  • Add full pipeline integration tests verifying the complete DSL -> compose -> generate -> validate flow
  • Add architecture documentation with Mermaid diagrams covering module map, data flow, and compiler analogy
  • Add complete DSL reference with formal grammar, keyword tables, auto-typing rules, and examples
  • Add generated project anatomy documentation for consumers of generated servers
  • Add security model documentation covering the expression sandbox, AST allowlist, and threat model
  • Add testing guide with test suite map, conventions, and instructions for adding tests
  • Update README with full project description, capabilities, quick start, and documentation links
  • Update CONTRIBUTING with complete setup, development workflow, coding conventions, and PR checklist

Depends on

#14

- Add full pipeline integration tests (DSL -> compose -> generate -> validate output)
- Add architecture documentation with module map and data flow diagrams
- Add complete DSL reference with grammar, keywords, and examples
- Add generated project anatomy documentation
- Add security model documentation (sandbox, allowlist, threat model)
- Add testing guide with test suite map and conventions
- Update README with full project description, capabilities, and doc links
- Update CONTRIBUTING with complete setup, workflow, and conventions guide
@sezallagwal sezallagwal force-pushed the feature/evaluation-documentation branch 2 times, most recently from ace9a91 to 12e3b97 Compare June 28, 2026 09:46

@dhairyashiil dhairyashiil left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi Sezal, thanks for adding the docs and evaluation coverage. Could we adjust a few things here so the docs and tests match what the generator really does today?

Could we update the generated-project docs/README claims around tests? The docs currently say generated projects include src/tests/* and an npm test script, but the generator does not emit tests or a test script yet. We should either generate those files/scripts, or remove the claim for now.

Could we also update the DSL reference to include supported constructs like CONTENT_TEXT, CONTENT_IMAGE, and top-level WEBHOOK? The parser supports these, so the reference should include them if this is the main user-facing DSL contract.

For the generated TypeScript validity tests, could we add a stronger smoke test later in this PR or as a near follow-up? Right now the tests mostly check substrings like export / tool; a better signal would be writing a generated project to a temp dir and running tsc or npm run build there.

One stack-level thing: because these PRs are stacked and most target feature branches, the current GitHub CI only shows CLA on most of them. Could we update CI so stacked PRs also run npm run check, not only PRs targeting main? That will help us catch format/lint/test issues earlier while this repo grows.

Overall the documentation direction is helpful, I just want it to stay accurate from the first version.

@dhairyashiil dhairyashiil marked this pull request as draft July 1, 2026 09:05
…e test, CI on stacked PRs

- Generator now emits src/tests/setup.ts + per-workflow smoke tests and a 'test' script, so the generated-project docs/README claims are accurate

- Document CONTENT_TEXT, CONTENT_IMAGE (multimodal sampling) and the top-level WEBHOOK construct in the DSL reference (WEBHOOK noted as parsed-but-not-yet-emitted)

- Add generated-project.integration.test.ts: writes a project to a temp dir, runs 'tsc --noEmit' over it, and runs its generated npm test

- CI now runs on all PRs (not only those targeting main) so stacked PRs get format/lint/typecheck/test/build

- Fix pre-existing lint errors and repo-wide Prettier formatting surfaced by enabling CI; correct stale docs (engine filenames, GEMINI_API_KEY claim) and add missing lint:fix script
…ature/evaluation-documentation

# Conflicts:
#	src/generator/project.ts
#	src/tools/generate.ts
#	src/workflow/executor.ts
@sezallagwal sezallagwal requested a review from dhairyashiil July 7, 2026 21:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants