Skip to content

feat(pipeline): Ensemble production topology routing pass for ObjectScript/IRIS - #1063

Open
isc-tdyar wants to merge 3 commits into
DeusData:mainfrom
isc-tdyar:feat/ensemble-routing
Open

feat(pipeline): Ensemble production topology routing pass for ObjectScript/IRIS#1063
isc-tdyar wants to merge 3 commits into
DeusData:mainfrom
isc-tdyar:feat/ensemble-routing

Conversation

@isc-tdyar

Copy link
Copy Markdown
Contributor

Adds pass_ensemble_routing — a pipeline pass that statically extracts InterSystems Ensemble/IRIS Interoperability production topology from ObjectScript .cls files at index time. No live IRIS instance required.

Closes #1061. Follows up on #467 (ObjectScript language support) — split out at reviewer request.

What it does

Pass A — EnsembleItem nodes: reads each EnsembleProduction class's ProductionDefinition XData block (XML), parses <Item Name="..." ClassName="..."> entries, and creates EnsembleItem nodes for each production component (business services, processes, operations).

Pass B — ROUTES_TO edges: for each EnsembleItem, reads the source .cls file and extracts routing rules referencing other components by class name, emitting ROUTES_TO edges. Uses segment-anchored matching (qn_ends_with_segment) to avoid false positives from partial class name matches.

WorkMgr dispatch: detects .Queue("##class(X).method", ...) calls and emits CALLS edges to the target method — the same static dispatch pattern as task queues in Celery/Airflow.

Changes from the version reviewed in #467

Per review feedback:

  • Language gate: early-exits immediately if the project contains no ObjectScript files — zero overhead for non-IRIS projects
  • Truncation logging: cbm_log_warn fires when MAX_ITEMS or MAX_SETTINGS caps are hit so silent truncation is visible in logs

Files changed

  • src/pipeline/pass_ensemble_routing.{c,h} — new pass
  • src/pipeline/pipeline.c — wired into run_predump_passes
  • Makefile.cbm — added to SRCS

@isc-tdyar
isc-tdyar requested a review from DeusData as a code owner July 12, 2026 19:58
@DeusData DeusData added enhancement New feature or request parsing/quality Graph extraction bugs, false positives, missing edges language-request Request for new language support priority/backlog Valuable contribution, lower scheduling urgency; review when maintainer capacity opens. labels Jul 14, 2026
@DeusData DeusData added this to the 0.9.2-rc milestone Jul 14, 2026
@DeusData

Copy link
Copy Markdown
Owner

Thanks for splitting the Ensemble topology pass from the base ObjectScript support. This is a 0.9.2-rc language enhancement. Review will focus on bounded parsing of embedded production XML, deterministic EnsembleItem identity, conservative ROUTES_TO matching, malformed-input handling, and representative IRIS fixtures without requiring a live server.

Adds pass_ensemble_routing.c, a pre-dump pipeline pass that reads
Ensemble production definitions (XData ProductionDefinition blocks)
and emits ROUTES_TO edges between production items.

The pass:
  - Early-exits when no ObjectScript source files (.cls/.mac/.int)
    are present in the graph buffer, making it zero-cost on non-IRIS
    projects.
  - Parses <Item> / <Setting> XML from each XData ProductionDefinition
    found in the graph buffer.
  - Emits EnsembleItem nodes and ROUTES_TO edges with confidence scores
    (0.95 for literal target names, 0.85 for property-resolved names).
  - Logs a cbm_log_warn when MAX_ITEMS (256) or MAX_SETTINGS (8) caps
    truncate the parsed production.

Signed-off-by: Thomas Dyar <tdyar@intersystems.com>
Signed-off-by: Thomas Dyar <tdyar@intersystems.com>
…_TO pipeline test

- class_name_matches() replaces strstr for method_belongs_to_production:
  exact or dot-prefixed suffix match prevents 'Ens' matching 'Ens.BusinessService'
- find_entry_point: strcmp instead of strstr for QN matching
- read_file: cbm_fopen for Windows UTF-8 path compatibility
- pipeline_ensemble_routing_routes_to_edges: full pipeline test with two
  .cls fixtures asserting >= 2 EnsembleItem nodes and >= 1 ROUTES_TO edge

Signed-off-by: Thomas Dyar <tdyar@intersystems.com>
@isc-tdyar
isc-tdyar force-pushed the feat/ensemble-routing branch from 72a8a0d to db30e5d Compare July 28, 2026 22:24
@DeusData

Copy link
Copy Markdown
Owner

Reviewed in full. This is a marked step up from #1064, and I want to say that first because the contrast is the most useful thing I can tell you.

It demonstrably works end to end. Your test runs the full pipeline on two real .cls fixtures and asserts at least two EnsembleItem nodes and at least one ROUTES_TO edge in the dumped store — return codes checked, post-dump, not at the extraction boundary. Every CI leg is green. That is exactly what #1064 was missing: there, the queries could not parse and the return codes were dropped, so it always returned empty while looking healthy. Here I could confirm the inputs genuinely exist (XData nodes are emitted at extract_defs.c:5798, parent_class at pass_definitions.c:288) and the outputs land in the store.

The review feedback from #467 was clearly absorbed too: the language gate, truncation warnings, the segment-anchored class_name_matches so Ens cannot match Ens.BusinessService, and cbm_fopen for Windows paths. The confidence scoring (0.95 literal, 0.85 property-resolved, −0.10 on the entry-point fallback) fits our existing cross-service linking model well.

The direction question is the maintainer's, and it is about schema rather than quality. EnsembleItem is a new node label and ROUTES_TO a new edge type — permanent core-schema surface, for one vendor's language. Whether vendor-specific labels belong in the core schema is his call, not mine, and I have put it to him.

Worth knowing while that is decided: EnsembleItem is excluded by all four hardcoded label IN ('Function','Method','Class') lists in store.c (arch boundaries, clustering, hotspots, vector search). For topology nodes that is arguably right — but it means get_architecture would show none of a production's topology, which cuts against the feature's stated purpose. Three separate contributors have now hit that same allowlist trap in this review round, so it is a systemic thing rather than a criticism of your PR.

One defect I would fix before merge, because it is a graph-accuracy problem.

scan_source_for_send_targets takes a method_name parameter and then discards it ((void)method_name, line 327). It scans the whole file and returns only the first SendRequestSync target. resolve_method_routes then attributes that single target to every production-member method in the file. So a class with N methods and M distinct targets produces N identical edges to the first target — wrong-method attribution and undercounting at the same time. Your single-method fixture cannot catch it; a two-method, two-target fixture would.

And one thing to correct in the description: the third bullet advertises WorkMgr dispatch — detecting .Queue("##class(X).method") and emitting CALLS edges. That code is not in the diff; there is no Queue or ##class handling anywhere. Either it belongs to a different branch or the body is ahead of the code. Worth fixing so the next reviewer is not looking for it.

Three smaller notes:

  • The "zero overhead for non-IRIS projects" claim is not quite right. has_objectscript_nodes walks every node in the graph buffer with a strlen and up to three suffix comparisons, and the visitor API has no early break — so it is O(N) per index, in both FAST and FULL modes. On an 8.5M-node graph that is tens of milliseconds; small, but not zero, and the discover layer already knows per-language file counts if you want a cheaper gate. Also .cls is shared with Apex, so Salesforce repos pass the extension gate too.
  • extract_xml_attr searches unbounded from the tag offset, so an Item or Setting missing an attribute silently inherits it from a later tag — a wrong Enabled or a mispaired ClassName. Bounding the search to the tag's closing > fixes it.
  • properties_json is assembled by snprintf without JSON-escaping the XML-derived strings, so a quote in an item or class name produces malformed JSON in the graph.

One genuine question rather than a defect: you match only TargetConfigName (singular), but real Ensemble routers commonly use TargetConfigNames (plural, comma-separated). Is the singular-only set deliberate?

Static extraction of production topology without a live IRIS instance is a real capability for IRIS shops. The open question on our side is the schema surface, not the work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request language-request Request for new language support parsing/quality Graph extraction bugs, false positives, missing edges priority/backlog Valuable contribution, lower scheduling urgency; review when maintainer capacity opens.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Ensemble production topology routing pass for ObjectScript/IRIS

2 participants