Skip to content

Record the paths of the content model as columns in the engine #615

Closed

Description

@yhay81

Context

DESIGN.md section 10.2: "Characters and content items are columnar buffers that NumPy and Arrow read without copying; records are views over them." #606 gave the paths of a page columns, Page.paths(), in the layout agreed there, and built them in the binding from the interpretation that Page.content() memoizes (section 5), as its Paths asked. That interpretation keeps one tuple per path in the engine's content model, ContentItem::Path: 288 bytes of enum and three heap allocations for the verbs, the coordinates, and the stroke's paint. The page memo holds those tuples after the call, and building them dominates what reading a CAD page costs once the Python objects are gone.

On the 200,000-line page of #606, on Windows 11 x86-64 (evidence/path-columns/results/windows-amd64.md):

  • Page.paths() takes 0.25 s, a little longer than rendering the page (0.21 s), against 4.0 s for Content.paths. Once the interpretation is memoized, building the columns takes 36 ms of that.
  • The columns hold 224 bytes a path: 118 for the path and 53 for each of its two segments.
  • The call grows the peak resident set by 581 bytes a path and leaves 693 held. About 470 of those 693 are the memoized tuples.

Goal

The content collector records paths as columns in the engine, the page memo holds them, and Page.paths() hands the memo's buffers to Python without a copy, so that reading the paths of a CAD page costs tens of bytes per segment in all, not only in the columns that Python holds.

Scope

In:

  • crates/pyropdf-engine/src/extract/model.rs: a columnar path table that the collector appends to, with the fill-and-stroke merge and the clip filter of today's records, and a PageContent whose items refer to its rows.
  • crates/pyropdf-python/src/buffers.rs and items.rs: PathColumns over the memoized table, and Page.content()'s path tuples built from it.

Out:

Paths

crates/pyropdf-engine/src/extract/model.rs, crates/pyropdf-engine/src/extract/mod.rs, crates/pyropdf-engine/src/extract/tests.rs, crates/pyropdf-python/src/buffers.rs, crates/pyropdf-python/src/items.rs, and evidence/path-columns/.

Acceptance

  • tests/test_path_columns.py and tests/test_content_model.py pass unchanged, and max_elements refuses with content_element_count at the same bounds
  • evidence/path-columns/measure.py rerun shows columns costing at most a few hundred bytes of peak and held memory per line on the 200,000-line page, with the results committed

Notes

  • The collector merges a stroke into the fill before it when both paint the same segments (record_path), and finish drops the items outside clip after the budget has counted them; the table needs both.

Activity

  1. added this to the Phase 1: Reading milestone on Nov 3, 2026
  2. added
    area:enginecrates/pyropdf-engine: interpretation, rendering, writing, understanding
    on Nov 3, 2026
  3. added
    in-progressClaimed; the latest comment names the branch
    on Nov 4, 2026
  4. yhay81 commented on Nov 4, 2026

    @yhay81
    OwnerAuthor

    Claiming: branch claude/615-engine-path-columns.

  5. yhay81 commented on Nov 4, 2026

    @yhay81
    OwnerAuthor

    Handoff: PR #642 on claude/615-engine-path-columns.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:enginecrates/pyropdf-engine: interpretation, rendering, writing, understandingin-progressClaimed; the latest comment names the branch

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions