Skip to content

feat: BYOND DreamMaker support (.dm/.dme/.dmi/.dmm/.dmf) - #884

Closed
kanya-approve wants to merge 8 commits into
Graphify-Labs:v8from
kanya-approve:feat/dm-extractor
Closed

feat: BYOND DreamMaker support (.dm/.dme/.dmi/.dmm/.dmf)#884
kanya-approve wants to merge 8 commits into
Graphify-Labs:v8from
kanya-approve:feat/dm-extractor

Conversation

@kanya-approve

@kanya-approve kanya-approve commented May 15, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds first-class graphify support for the full BYOND project surface, not just the DM language.

Commit 1 — .dm / .dme (DreamMaker source)

  • tree-sitter-dm lives in [project.optional-dependencies].dreammaker so the base PyPI install of graphifyy stays clean (no git deps in [project], no broken-sdist resolution failures). Mirrors the pattern used for [sql].
  • tree-sitter-dm 0.25.0's PyPI artifact is currently broken (sdist missing scanner.c, no wheels). [tool.uv.sources] overrides the resolution to git for uv users, so uv sync --extra dreammaker works today. Pip users with the extra are documented as needing a manual git install until 0.25.1 lands.
  • DM source-language tests are guarded by @pytest.mark.skipif(not _HAS_TREE_SITTER_DM, ...) so the suite stays green when the grammar isn't installed.
  • DM identity is path-based (/datum/foo/proc/bar), so the generic class-body walker doesn't fit. extract_dm is a custom walk handling type_definition, nested type_proc_definition / type_proc_override, top-level proc_definition / proc_override with optional type_path prefix, preproc_include, call_expression / field_proc_expression, and new_expression (→ instantiates edges).
  • Same-name ambiguity (e.g. 8 path-distinct f() overrides in one file) is left unresolved rather than picked arbitrarily — matches the cross-file resolver's single-match rule. ..() super-calls are skipped because resolving them needs corpus-wide type hierarchy unavailable at single-file extract time.
  • Unresolved #include paths get "external": true on the edge so consumers reading raw extractions can skip them without re-deriving the check (build.py:160 already drops dangling targets at graph-build time).

Commit 2 — .dmi / .dmm / .dmf (project assets, no grammar dep)

  • .dmi (icon sheets): PNG with a zTXt "Description" chunk holding BYOND state metadata. extract_dmi parses it with stdlib struct+zlib, emits one node per icon state name — the same names DM code references via icon_state = "X".
  • .dmm (map files): tile dictionary entries name the types composing each tile. extract_dmm parses the dictionary (skipping the grid), splits on top-level commas, strips {var=val} overrides, emits uses edges from map → types. Format reference: SpacemanDMM/crates/dmm-tools/src/dmm/read.rs. tree-sitter-dm claims .dmm support but errors on the "key" = (...) syntax in practice, hence the custom parser.
  • .dmf (interface forms): hierarchical key-value text. extract_dmf emits window + elem nodes with the BYOND control type baked into the elem label (e.g. elem "map" [MAP]).

All three are pure stdlib — they don't need the dreammaker extra.

Test plan

  • uv sync --extra dreammaker && uv run --with pytest pytest tests/test_languages.py -k "dm or dmi or dmm or dmf" — 26 pass
  • uv sync && uv run --with pytest pytest tests/test_languages.py -k "dm or dmi or dmm or dmf" — 13 DM tests skip cleanly via @needs_dm, 13 BYOND-format tests pass
  • uv run pytest tests/ — 792 passing without the extra (8 pre-existing SQL failures unchanged)
  • Stress: extracted all 660 .dm/.dme files in spacestation13/dm-test-suite — 0 errors
  • Stress: all .dmi / .dmf files in dm-test-suite parse without errors
  • End-to-end: uv run graphify update <dm-corpus> produces a graph with correct type nodes (/datum/object), proc nodes (/datum/object/New()), and clustered communities
  • CI green on Python 3.10 + 3.12 — DM tests will skip until either CI installs the dreammaker extra or upstream cuts 0.25.1

ARCHITECTURE.md checklist

  1. Added extract_dm, extract_dmi, extract_dmm, extract_dmf in graphify/extract.py
  2. Registered .dm, .dme, .dmi, .dmm, .dmf in _DISPATCH + collect_files
  3. Added all five suffixes to CODE_EXTENSIONS in detect.py (watch.py picks them up automatically)
  4. Added tree-sitter-dm to pyproject.toml (as an optional dreammaker extra)
  5. Added fixtures (sample.dm, sample.dmi, sample.dmm, sample.dmf) and tests in tests/test_languages.py

Related

@safishamsi safishamsi left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exciting to see DreamMaker support! Two blocking issues before this can land:

1. Dangling include-edge targets
When an #include points to a file not in the corpus (e.g. engine headers, external libs), the edge target node doesn't exist in the graph — this creates orphan edges that break graph traversal and dedup. Fix: check whether the target node ID exists before emitting the edge, or mark it with "external": true so downstream code can skip it.

Example guard:

if target_id in known_node_ids:
    edges.append({source: src_id, target: target_id, ...})

2. Git dependency blocks PyPI publishing
tree-sitter-dreammaker @ git+... in the dependency list means we can't publish to PyPI (PyPI rejects git deps). Please either:

  • Publish tree-sitter-dreammaker to PyPI and depend on the PyPI release, or
  • Make it an optional extra (e.g. graphifyy[dreammaker]) with a try/except import guard in the extractor, same pattern as our other optional grammars

Happy to help with either approach — just let us know.

@kanya-approve kanya-approve changed the title feat: add DM (BYOND DreamMaker) language extractor feat: BYOND DreamMaker support (.dm/.dme + .dmi/.dmm/.dmf) May 15, 2026
@kanya-approve
kanya-approve force-pushed the feat/dm-extractor branch 2 times, most recently from 3d951d0 to 2005f0f Compare May 15, 2026 23:00
Wires tree-sitter-dm into the extraction pipeline so .dm and .dme files
become first-class corpus members. PyPI's tree-sitter-dm wheel is broken
right now, so the dep installs from the FeudeyTF/tree-sitter-dm git repo.

DM's identity model is path-based, not block-based — `/datum/foo/proc/bar`
defines `bar` on `/datum/foo` from top level — so the generic class-body
walker doesn't fit. extract_dm is a custom AST walk that handles:

- type_definition + nested type_proc_definition/type_proc_override
- top-level proc_definition with optional type_path prefix
- proc_override (no `proc` keyword, type_path required)
- preproc_include (resolved to file nodes when the target exists)
- call_expression / field_proc_expression as `calls`
- new_expression as `instantiates`

Same-name ambiguity (e.g. 8 path-distinct `f()` overrides in one file) is
left unresolved rather than picked arbitrarily — matches the cross-file
resolver's single-match rule. `..()` super-calls are skipped because
resolving them needs corpus-wide type hierarchy we don't have at
single-file extract time.

Smoke-tested across all 660 files in spacestation13/dm-test-suite: 0
extraction errors, 1840 nodes, 1405 edges.
Beyond .dm source, a BYOND project ships three structured assets the
graph should know about:

- .dmi (icon sheets): PNG with a zTXt "Description" chunk holding BYOND
  state metadata. extract_dmi parses it with stdlib struct+zlib and emits
  one node per icon state name (the same names DM code references via
  `icon_state = "X"`).
- .dmm (map files): tile dictionary entries name the types that compose
  each tile, e.g. `"a" = (/obj/structure/table{...}, /area/maintenance)`.
  extract_dmm parses the dictionary section (skipping the grid), splits
  tile bodies on top-level commas, strips `{var=val}` overrides, and
  emits `uses` edges from the map file to each referenced type. Format
  reference: SpacemanDMM/crates/dmm-tools/src/dmm/read.rs — tree-sitter-dm
  claims .dmm support but errors on the `"key" = (...)` syntax in
  practice, hence the custom parser.
- .dmf (interface forms): hierarchical key-value text. extract_dmf emits
  window and elem nodes with the BYOND control type encoded in the elem
  label (e.g. `elem "map" [MAP]`), so winset/winget references in DM
  code have something to point at.

All three are registered in _DISPATCH and CODE_EXTENSIONS. .dmi moves
out of IMAGE_EXTENSIONS now that it has a real extractor — the metadata
is more useful to a code graph than an LLM caption of the sprite sheet.

Smoke-tested across the full spacestation13/dm-test-suite corpus: 0
extraction errors. 13 new tests for the three extractors.
The contributing guide currently tells contributors `pip install pytest`
as a separate step, and CI does the same. Move pytest into PEP 735
`[dependency-groups]` so it's declared in pyproject.toml and `uv sync`
installs it by default (no `--with` workaround, no separate install
line). Update CI to use astral-sh/setup-uv + `uv sync` + `uv run pytest`,
and refresh the Contributing section of the README to match.

`[dependency-groups]` is the right home (vs `[project.optional-dependencies]`)
because pytest is dev-only and shouldn't appear in the published wheel's
optional features list alongside things like `pdf` or `mcp`.
@kanya-approve

Copy link
Copy Markdown
Contributor Author

Exciting to see DreamMaker support! Two blocking issues before this can land:

1. Dangling include-edge targets When an #include points to a file not in the corpus (e.g. engine headers, external libs), the edge target node doesn't exist in the graph — this creates orphan edges that break graph traversal and dedup. Fix: check whether the target node ID exists before emitting the edge, or mark it with "external": true so downstream code can skip it.

Example guard:

if target_id in known_node_ids:
    edges.append({source: src_id, target: target_id, ...})

2. Git dependency blocks PyPI publishing tree-sitter-dreammaker @ git+... in the dependency list means we can't publish to PyPI (PyPI rejects git deps). Please either:

  • Publish tree-sitter-dreammaker to PyPI and depend on the PyPI release, or
  • Make it an optional extra (e.g. graphifyy[dreammaker]) with a try/except import guard in the extractor, same pattern as our other optional grammars

Happy to help with either approach — just let us know.

Made tree-sitter-dm an optional dep and labeling files outside of corpus as external now

@kanya-approve
kanya-approve requested a review from safishamsi May 15, 2026 23:17
@kanya-approve kanya-approve changed the title feat: BYOND DreamMaker support (.dm/.dme + .dmi/.dmm/.dmf) feat: BYOND DreamMaker support (.dm/.dme/.dmi/.dmm/.dmf) May 15, 2026
- move tree-sitter-dm>=0.25.1 into core dependencies alongside the
  other tree-sitter grammars (no longer optional)
- remove [tool.uv.sources] git override
- drop [dreammaker] extra and its install caveat from README
- remove --extra dreammaker from CI
# Conflicts:
#	graphify/detect.py
#	graphify/extract.py
#	tests/test_languages.py
@safishamsi

Copy link
Copy Markdown
Collaborator

Implemented manually on v8 — the PR had merge conflicts with recent changes (uv migration, .svh, OpenCode path fix, cluster remap). All four extractors landed: extract_dm (tree-sitter-dm, covers .dm/.dme), extract_dmi (PNG tEXt/zTXt chunk parser for icon states), extract_dmm (tile dictionary parser, uses edges), extract_dmf (window/elem/control-type hierarchy). Fixture files + 26 tests added, all green. tree-sitter-dm added to pyproject.toml dependencies. Thanks for the thorough contribution!

@safishamsi safishamsi closed this May 27, 2026
safishamsi added a commit that referenced this pull request May 27, 2026
…, #1030)

- Feat: extract_dm (tree-sitter-dm), extract_dmi (PNG icon states),
  extract_dmm (tile dict uses edges), extract_dmf (window/elem hierarchy)
  for .dm .dme .dmi .dmm .dmf; 26 tests, fixtures, pyproject.toml dep
- Feat: graphify extract --mode deep flag; deep_mode threaded through all
  four LLM backends via extract_corpus_parallel
- Fix: CHANGELOG 0.8.21 entries for #1050, #1046, #1047 that were missing

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants