feat: BYOND DreamMaker support (.dm/.dme/.dmi/.dmm/.dmf) - #884
feat: BYOND DreamMaker support (.dm/.dme/.dmi/.dmm/.dmf)#884kanya-approve wants to merge 8 commits into
Conversation
38cce49 to
879aefe
Compare
safishamsi
left a comment
There was a problem hiding this comment.
Exciting to see DreamMaker support! Two blocking issues before this can land:
1. Dangling include-edge targets
When an #include points to a file not in the corpus (e.g. engine headers, external libs), the edge target node doesn't exist in the graph — this creates orphan edges that break graph traversal and dedup. Fix: check whether the target node ID exists before emitting the edge, or mark it with "external": true so downstream code can skip it.
Example guard:
if target_id in known_node_ids:
edges.append({source: src_id, target: target_id, ...})2. Git dependency blocks PyPI publishing
tree-sitter-dreammaker @ git+... in the dependency list means we can't publish to PyPI (PyPI rejects git deps). Please either:
- Publish
tree-sitter-dreammakerto PyPI and depend on the PyPI release, or - Make it an optional extra (e.g.
graphifyy[dreammaker]) with a try/except import guard in the extractor, same pattern as our other optional grammars
Happy to help with either approach — just let us know.
879aefe to
c9c87b7
Compare
3d951d0 to
2005f0f
Compare
Wires tree-sitter-dm into the extraction pipeline so .dm and .dme files become first-class corpus members. PyPI's tree-sitter-dm wheel is broken right now, so the dep installs from the FeudeyTF/tree-sitter-dm git repo. DM's identity model is path-based, not block-based — `/datum/foo/proc/bar` defines `bar` on `/datum/foo` from top level — so the generic class-body walker doesn't fit. extract_dm is a custom AST walk that handles: - type_definition + nested type_proc_definition/type_proc_override - top-level proc_definition with optional type_path prefix - proc_override (no `proc` keyword, type_path required) - preproc_include (resolved to file nodes when the target exists) - call_expression / field_proc_expression as `calls` - new_expression as `instantiates` Same-name ambiguity (e.g. 8 path-distinct `f()` overrides in one file) is left unresolved rather than picked arbitrarily — matches the cross-file resolver's single-match rule. `..()` super-calls are skipped because resolving them needs corpus-wide type hierarchy we don't have at single-file extract time. Smoke-tested across all 660 files in spacestation13/dm-test-suite: 0 extraction errors, 1840 nodes, 1405 edges.
Beyond .dm source, a BYOND project ships three structured assets the
graph should know about:
- .dmi (icon sheets): PNG with a zTXt "Description" chunk holding BYOND
state metadata. extract_dmi parses it with stdlib struct+zlib and emits
one node per icon state name (the same names DM code references via
`icon_state = "X"`).
- .dmm (map files): tile dictionary entries name the types that compose
each tile, e.g. `"a" = (/obj/structure/table{...}, /area/maintenance)`.
extract_dmm parses the dictionary section (skipping the grid), splits
tile bodies on top-level commas, strips `{var=val}` overrides, and
emits `uses` edges from the map file to each referenced type. Format
reference: SpacemanDMM/crates/dmm-tools/src/dmm/read.rs — tree-sitter-dm
claims .dmm support but errors on the `"key" = (...)` syntax in
practice, hence the custom parser.
- .dmf (interface forms): hierarchical key-value text. extract_dmf emits
window and elem nodes with the BYOND control type encoded in the elem
label (e.g. `elem "map" [MAP]`), so winset/winget references in DM
code have something to point at.
All three are registered in _DISPATCH and CODE_EXTENSIONS. .dmi moves
out of IMAGE_EXTENSIONS now that it has a real extractor — the metadata
is more useful to a code graph than an LLM caption of the sprite sheet.
Smoke-tested across the full spacestation13/dm-test-suite corpus: 0
extraction errors. 13 new tests for the three extractors.
2005f0f to
fda0243
Compare
The contributing guide currently tells contributors `pip install pytest` as a separate step, and CI does the same. Move pytest into PEP 735 `[dependency-groups]` so it's declared in pyproject.toml and `uv sync` installs it by default (no `--with` workaround, no separate install line). Update CI to use astral-sh/setup-uv + `uv sync` + `uv run pytest`, and refresh the Contributing section of the README to match. `[dependency-groups]` is the right home (vs `[project.optional-dependencies]`) because pytest is dev-only and shouldn't appear in the published wheel's optional features list alongside things like `pdf` or `mcp`.
Made tree-sitter-dm an optional dep and labeling files outside of corpus as external now |
- move tree-sitter-dm>=0.25.1 into core dependencies alongside the other tree-sitter grammars (no longer optional) - remove [tool.uv.sources] git override - drop [dreammaker] extra and its install caveat from README - remove --extra dreammaker from CI
# Conflicts: # graphify/detect.py # graphify/extract.py # tests/test_languages.py
|
Implemented manually on v8 — the PR had merge conflicts with recent changes (uv migration, |
…, #1030) - Feat: extract_dm (tree-sitter-dm), extract_dmi (PNG icon states), extract_dmm (tile dict uses edges), extract_dmf (window/elem hierarchy) for .dm .dme .dmi .dmm .dmf; 26 tests, fixtures, pyproject.toml dep - Feat: graphify extract --mode deep flag; deep_mode threaded through all four LLM backends via extract_corpus_parallel - Fix: CHANGELOG 0.8.21 entries for #1050, #1046, #1047 that were missing Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Summary
Adds first-class graphify support for the full BYOND project surface, not just the DM language.
Commit 1 —
.dm/.dme(DreamMaker source)tree-sitter-dmlives in[project.optional-dependencies].dreammakerso the base PyPI install ofgraphifyystays clean (no git deps in[project], no broken-sdist resolution failures). Mirrors the pattern used for[sql].tree-sitter-dm0.25.0's PyPI artifact is currently broken (sdist missingscanner.c, no wheels).[tool.uv.sources]overrides the resolution to git for uv users, souv sync --extra dreammakerworks today. Pip users with the extra are documented as needing a manual git install until 0.25.1 lands.@pytest.mark.skipif(not _HAS_TREE_SITTER_DM, ...)so the suite stays green when the grammar isn't installed./datum/foo/proc/bar), so the generic class-body walker doesn't fit.extract_dmis a custom walk handlingtype_definition, nestedtype_proc_definition/type_proc_override, top-levelproc_definition/proc_overridewith optionaltype_pathprefix,preproc_include,call_expression/field_proc_expression, andnew_expression(→instantiatesedges).f()overrides in one file) is left unresolved rather than picked arbitrarily — matches the cross-file resolver's single-match rule...()super-calls are skipped because resolving them needs corpus-wide type hierarchy unavailable at single-file extract time.#includepaths get"external": trueon the edge so consumers reading raw extractions can skip them without re-deriving the check (build.py:160 already drops dangling targets at graph-build time).Commit 2 —
.dmi/.dmm/.dmf(project assets, no grammar dep).dmi(icon sheets): PNG with a zTXt "Description" chunk holding BYOND state metadata.extract_dmiparses it with stdlibstruct+zlib, emits one node per icon state name — the same names DM code references viaicon_state = "X"..dmm(map files): tile dictionary entries name the types composing each tile.extract_dmmparses the dictionary (skipping the grid), splits on top-level commas, strips{var=val}overrides, emitsusesedges from map → types. Format reference:SpacemanDMM/crates/dmm-tools/src/dmm/read.rs.tree-sitter-dmclaims.dmmsupport but errors on the"key" = (...)syntax in practice, hence the custom parser..dmf(interface forms): hierarchical key-value text.extract_dmfemits window + elem nodes with the BYOND control type baked into the elem label (e.g.elem "map" [MAP]).All three are pure stdlib — they don't need the
dreammakerextra.Test plan
uv sync --extra dreammaker && uv run --with pytest pytest tests/test_languages.py -k "dm or dmi or dmm or dmf"— 26 passuv sync && uv run --with pytest pytest tests/test_languages.py -k "dm or dmi or dmm or dmf"— 13 DM tests skip cleanly via@needs_dm, 13 BYOND-format tests passuv run pytest tests/— 792 passing without the extra (8 pre-existing SQL failures unchanged).dm/.dmefiles in spacestation13/dm-test-suite — 0 errors.dmi/.dmffiles in dm-test-suite parse without errorsuv run graphify update <dm-corpus>produces a graph with correct type nodes (/datum/object), proc nodes (/datum/object/New()), and clustered communitiesdreammakerextra or upstream cuts 0.25.1ARCHITECTURE.md checklist
extract_dm,extract_dmi,extract_dmm,extract_dmfingraphify/extract.py.dm,.dme,.dmi,.dmm,.dmfin_DISPATCH+collect_filesCODE_EXTENSIONSindetect.py(watch.py picks them up automatically)tree-sitter-dmtopyproject.toml(as an optionaldreammakerextra)sample.dm,sample.dmi,sample.dmm,sample.dmf) and tests intests/test_languages.pyRelated