Add end-to-end evaluation tests and project documentation#15
Add end-to-end evaluation tests and project documentation#15sezallagwal wants to merge 3 commits into
Conversation
- Add full pipeline integration tests (DSL -> compose -> generate -> validate output) - Add architecture documentation with module map and data flow diagrams - Add complete DSL reference with grammar, keywords, and examples - Add generated project anatomy documentation - Add security model documentation (sandbox, allowlist, threat model) - Add testing guide with test suite map and conventions - Update README with full project description, capabilities, and doc links - Update CONTRIBUTING with complete setup, workflow, and conventions guide
ace9a91 to
12e3b97
Compare
dhairyashiil
left a comment
There was a problem hiding this comment.
Hi Sezal, thanks for adding the docs and evaluation coverage. Could we adjust a few things here so the docs and tests match what the generator really does today?
Could we update the generated-project docs/README claims around tests? The docs currently say generated projects include src/tests/* and an npm test script, but the generator does not emit tests or a test script yet. We should either generate those files/scripts, or remove the claim for now.
Could we also update the DSL reference to include supported constructs like CONTENT_TEXT, CONTENT_IMAGE, and top-level WEBHOOK? The parser supports these, so the reference should include them if this is the main user-facing DSL contract.
For the generated TypeScript validity tests, could we add a stronger smoke test later in this PR or as a near follow-up? Right now the tests mostly check substrings like export / tool; a better signal would be writing a generated project to a temp dir and running tsc or npm run build there.
One stack-level thing: because these PRs are stacked and most target feature branches, the current GitHub CI only shows CLA on most of them. Could we update CI so stacked PRs also run npm run check, not only PRs targeting main? That will help us catch format/lint/test issues earlier while this repo grows.
Overall the documentation direction is helpful, I just want it to stay accurate from the first version.
…e test, CI on stacked PRs - Generator now emits src/tests/setup.ts + per-workflow smoke tests and a 'test' script, so the generated-project docs/README claims are accurate - Document CONTENT_TEXT, CONTENT_IMAGE (multimodal sampling) and the top-level WEBHOOK construct in the DSL reference (WEBHOOK noted as parsed-but-not-yet-emitted) - Add generated-project.integration.test.ts: writes a project to a temp dir, runs 'tsc --noEmit' over it, and runs its generated npm test - CI now runs on all PRs (not only those targeting main) so stacked PRs get format/lint/typecheck/test/build - Fix pre-existing lint errors and repo-wide Prettier formatting surfaced by enabling CI; correct stale docs (engine filenames, GEMINI_API_KEY claim) and add missing lint:fix script
…ature/evaluation-documentation # Conflicts: # src/generator/project.ts # src/tools/generate.ts # src/workflow/executor.ts
Summary
Depends on
#14