Read the paths of a page as columnar buffers

Context

DESIGN.md section 10: "Characters and content items are columnar buffers that NumPy and Arrow read without copying; records are views over them." #99 did this for characters (Page.chars(), Chars.column(), and the Arrow PyCapsule interface) and left out "content items other than text, until the content model covers them". Page.content() now covers paths, as one Path record per path, each with its Stroke, paint, GraphicsContext, and Rect, and a MoveTo, LineTo, or CurveTo with its Points per segment.

CAD output often draws every line as a path of its own. An A1 page of such lines (x y m x y l S, one in ten red), read with page.content(text_runs=False, images=False).paths on an Apple silicon Mac with CPython 3.15.0:

Lines	pyropdf 0.2.1	with `gc.disable()`	Peak RSS growth	`pixmap(dpi=36)`	PyMuPDF 1.28.2 `get_cdrawings()`	`get_drawings()`
33,000	0.27 s	0.11 s	2.6 KB per line	0.02 s	0.08 s	0.16 s
200,000	3.26 s	0.69 s	2.5 KB per line	0.09 s	0.49 s	0.93 s

pylopdf 0.16.0 measures the same as pyropdf. Rendering the page, which interprets every path, takes 0.02 and 0.09 s; the rest is building Python objects, and the cyclic garbage collector that their number triggers makes the time grow faster than the page. The same 200,000 lines in one path take 0.21 s and 0.7 KB per line. Each such path counts 3 elements, so with the default max_elements (262,144) a page of more than 87,381 lines is refused with content_element_count, and raising the bound costs about 2.5 KB per line.

This came up in the downstream application of #600, which reads the stroked segments of plan pages by color (site boundaries, dimension lines). It bounds max_elements because of this memory, so it cannot read denser drawings.

Goal

The paths of a page as columns, one row per path and one per segment, that NumPy and Arrow read without copying, so that reading a CAD page costs about what rendering it costs, at tens of bytes per segment.

Scope

In:

The layout: per segment, its path, verb, and points, a curve's control points included; per path, its fill and stroke paint (sRGB and alpha, or a pattern), fill rule, stroke width, cap, join, and dashes, bounding box, and graphics context, with the contexts shared as Content already shares them.
The buffer protocol and the Arrow PyCapsule interface, as Chars has them, and an accessor for the columns.
max_elements counting the same elements, so that the limit and its code stay as they are.

Out:

Text runs and images, until a measurement shows the need.
Making the Content.paths records views over the columns, as section 10 says records are, which can follow the measurement here.

Paths

crates/pyropdf-python/src/items.rs, crates/pyropdf-python/src/buffers.rs, python/pyropdf/_content.py, the content documentation, and a new test file.

Acceptance

On every corpus page, the columns hold the same segments, paints, strokes, and contexts as Content.paths
NumPy reads the coordinates without a copy, and Arrow imports the columns through the PyCapsule interface
Time and memory against Content.paths and PyMuPDF's get_cdrawings(), on the page above and on the corpus page with the most paths, under evidence/

Notes

Agree the layout in a comment here before building on it; it is public API. #99's buffers.rs is the model.

Reproduction

python repro.py 200000 and python repro.py 200000 --no-gc print the time and peak RSS growth (bytes on macOS, KiB on Linux).

import gc
import resource
import sys
import time

import pyropdf

lines = int(sys.argv[1]) if len(sys.argv) > 1 else 200_000
collect = "--no-gc" not in sys.argv


def page_pdf(count: int) -> bytes:
    """An A1 page of `count` short lines, each its own stroked path, one in ten red, as CAD output draws them."""
    parts = [b"0.5 w"]
    for i in range(count):
        x, y = 10 + (i % 400) * 5, 10 + (i // 400) * 5
        parts.append(b"%s %d %d m %d %d l S" % (b"1 0 0 RG" if i % 10 == 0 else b"0 0 0 RG", x, y, x + 4, y + 3))
    stream = b"\n".join(parts)
    objects = [
        b"<< /Type /Catalog /Pages 2 0 R >>",
        b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>",
        b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 2384 1684] /Contents 4 0 R >>",
        b"<< /Length %d >>\nstream\n" % len(stream) + stream + b"\nendstream",
    ]
    out, offsets = bytearray(b"%PDF-1.7\n"), []
    for number, body in enumerate(objects, 1):
        offsets.append(len(out))
        out += b"%d 0 obj\n" % number + body + b"\nendobj\n"
    xref = len(out)
    out += b"xref\n0 %d\n0000000000 65535 f \n" % (len(objects) + 1)
    out += b"".join(b"%010d 00000 n \n" % offset for offset in offsets)
    out += b"trailer\n<< /Size %d /Root 1 0 R >>\nstartxref\n%d\n%%%%EOF\n" % (len(objects) + 1, xref)
    return bytes(out)


with pyropdf.open(page_pdf(lines)) as document:
    page = document.pages[0]
    started = time.perf_counter()
    page.pixmap(dpi=36)
    rendered = time.perf_counter() - started
    before = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
    if not collect:
        gc.disable()
    started = time.perf_counter()
    paths = page.content(text_runs=False, images=False, max_elements=3 * lines + 16).paths
    read = time.perf_counter() - started
    grown = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss - before  # bytes on macOS, KiB on Linux
    print(f"{len(paths)} paths: content() {read:.2f} s, pixmap(dpi=36) {rendered:.2f} s, peak RSS +{grown / lines:.0f} per path"
          f"{'' if collect else ' (gc disabled)'}")

Claimed on branch claude/606-path-columns. I will post the column layout here for the owner's agreement before building on it.

The layout of the path columns

The owner agreed to this layout in the working session of 2026-11-04, together with two decisions recorded at the end. It is public API; a change to it comes back here first.

Accessor

Page.paths(*, clip=None, annotations=True, max_elements=262_144) -> Paths: the paths that page.content(text_runs=False, images=False, clip=..., annotations=...) holds, in the same order, from the same memoized interpretation, so max_elements counts the same elements and refuses with content_element_count as before. It is a method named after what it returns, as Page.chars() is (DESIGN.md section 10.1).

`Paths`

A Sequence[Path] like Chars: len(), indexing, and iteration build each row's Path record on demand, equal to Content.paths, and a slice with a step of 1 is a view of the same columns. column(name) returns a column as a read-only memoryview without a copy, in the machine's byte order, and names lists the columns. Shared values, which the columns index:

contexts: tuple[GraphicsContext, ...], one per context of the interpretation; the records share these objects, as Content's items share theirs.
patterns: tuple[ShadingPattern | TilingPattern, ...], the distinct pattern paints.
soft_masks: tuple[SoftMask, ...], the distinct soft masks.

Per path (one item per row):

Column	Type	Holds
`x0`, `y0`, `x1`, `y1`	float64	`Path.bbox`
`segment_offsets`	int32, one more than the rows	Path `i`'s segments are rows `segment_offsets[i]` to `segment_offsets[i + 1]` of the segment columns
`has_fill`	uint8, 0 or 1	whether `Path.fill` is not `None`
`fill_red`, `fill_green`, `fill_blue`, `fill_alpha`	float32	the fill's `Color`, which hayro computes as float32, so the values are exact; NaN where the fill is a pattern or absent
`fill_pattern`	int32	the fill's index in `patterns`; -1 where it is a color or absent
`fill_rule`	uint8	`Path.fill_rule` as its position in `FillRule`: 0 nonzero, 1 even-odd
`has_stroke`	uint8, 0 or 1	whether `Path.stroke` is not `None`
`stroke_red`, `stroke_green`, `stroke_blue`, `stroke_alpha`	float32	the stroke paint's `Color`; NaN where it is a pattern or there is no stroke
`stroke_pattern`	int32	the stroke paint's index in `patterns`, or -1
`stroke_width`, `stroke_miter_limit`, `stroke_dash_offset`	float64	the `Stroke` fields; NaN without a stroke
`stroke_cap`, `stroke_join`	uint8	positions in `LineCap` and `LineJoin`; 0 without a stroke
`stroke_dashes_offsets`	int32, one more than the rows	path `i`'s dash lengths are `stroke_dashes[stroke_dashes_offsets[i]:stroke_dashes_offsets[i + 1]]`
`context`	uint32	the path's index in `contexts`
`blend_mode`	uint8	position in `BlendMode`
`soft_mask`	int32	the path's index in `soft_masks`, or -1

Per segment (one item per segment of every path of the page; a view shares them, as a Chars view shares text_data):

Column	Type	Holds
`path`	uint32	the row of the segment's path
`verb`	uint8	the ASCII code of the PDF operator: `m` (MoveTo), `l` (LineTo), `c` (CurveTo), `h` (ClosePath)
`to_x`, `to_y`	float64	the segment's `to`; for `h`, the start of its subpath, where the closing line ends
`c1_x`, `c1_y`, `c2_x`, `c2_y`	float64	a curve's control points; NaN for the other verbs

stroke_dashes (float64) holds the dash lengths of every path, which stroke_dashes_offsets index.

Lines are segments whose verb is l or h, running from the previous row's to to their own, so a stroked-segment reader needs no Python loop: stroke_red[path] gives each segment its path's color.

Arrow

Paths is an Arrow struct array through the Arrow PyCapsule interface (__arrow_c_schema__, __arrow_c_array__), imported without a copy. Its fields are the per-path columns, with has_fill and has_stroke as Arrow booleans, nulls where the columns hold NaN or -1, and two list fields in place of the offsets:

segments: list<struct<path, verb, to_x, to_y, c1_x, c1_y, c2_x, c2_y>> over segment_offsets and the segment columns, the control points null for verbs other than c;
stroke_dashes: list<double> over stroke_dashes_offsets and stroke_dashes, null without a stroke.

Size

118 bytes per path and 53 bytes per segment, 32 of them a curve's control points. A line of the CAD page in the issue, one path of two segments, takes 224 bytes of columns, against 2.2 KB of records on this machine (Windows 11, x86-64, Python 3.14.8): Content.paths of the 200,000-line page takes 5.77 s and grows the peak resident set by 2.2 KB per line; PyMuPDF 1.28.2's get_cdrawings() takes 1.36 s and 0.96 KB per line; pixmap(dpi=36) takes 0.37 s.

Decisions

Where the columns come from. This issue builds the columns in the binding from the memoized interpretation that Page.content() shares, inside the issue's Paths. The interpretation's per-path tuples stay in the page memo, about 490 bytes a path on this machine (ContentItem is 296 bytes), so they, not the columns, dominate the memory that remains. Recording the paths as columns in the engine (crates/pyropdf-engine/src/extract/model.rs), so that the memo and Python share one copy, becomes a follow-up issue: PR Move to hayro 0.9 with LaurenzV/hayro#1734 patched in, and read streams that name a file #609 (hayro-syntax: object numbers as u32 (I8) #18) changes the same file.
Paths beyond the issue's list. Page.paths() lives in python/pyropdf/_page.py, and the new class and function are registered in crates/pyropdf-python/src/lib.rs, python/pyropdf/_core.pyi, and python/pyropdf/__init__.py; the owner allowed these edits.

Handoff: branch claude/606-path-columns, PR #616, which waits for the owner's review. It is not merged, and auto-merge is off.

What changed.

Page.paths(clip=, annotations=, max_elements=) returns pyropdf.Paths in the layout agreed above.
The binding builds the columns from the interpretation that page.content(text_runs=False, images=False) memoizes, so max_elements and content_element_count are unchanged and the two calls share one interpretation.
Paths is a Sequence[Path] row view. Its records equal Content.paths and share contexts, patterns, and soft_masks.
The Arrow export nests segments (a list of structs) and stroke_dashes (a list of doubles).
Beyond the issue's Paths, as the owner allowed: _page.py, lib.rs, _core.pyi, and __init__.py.

Checks.

python tools/check.py --locked passes on Windows 11: pytest 6569 passed and 691 skipped, plus ruff, ty, cargo fmt, clippy, and cargo test.
The NumPy and pyarrow tests of tests/test_path_columns.py pass locally with --with numpy --with pyarrow. CI skips them because it installs neither.

Evidence. evidence/path-columns/, with results/windows-amd64.md. The machine is an AMD Ryzen 5 3600XT on Windows 11, so the script reads memory with GetProcessMemoryInfo instead of resource. Each row is the median of five processes, with CPU time too, because other processes kept the processor busy.

On the 200,000-line page:
- paths() takes 0.32 s and pixmap(dpi=36) 0.31 s.
- Content.paths takes 4.6 s, and get_cdrawings() 1.15 s.
- The columns hold 224 bytes a line.
- A call leaves 701 bytes a line held. About 480 of them are the memoized per-path tuples of the engine.
On f1040.pdf page 1, the corpus page with the most paths (502), paths() takes 10 ms, Content.paths 19 ms, and get_cdrawings() 28 ms.

Next.

The owner reviews Read the paths of a page as columns (#606) #616, with CI green on its head.
After the merge, Record the paths of the content model as columns in the engine #615 moves the recording into the engine, so the memo holds the columns alone. It waits for hayro-syntax: object numbers as u32 (I8) #18 (PR Move to hayro 0.9 with LaurenzV/hayro#1734 patched in, and read streams that name a file #609) and this issue.

Open point for the owner. The registry description of content_element_count (limits.toml) still names only Page.content. Page.paths reads within the same limit. Changing the description touches the generated limit files, which are outside the agreed Paths, so this PR leaves them unchanged.

Keep the Arrow capsules alive while the path tests read them (#606)

Merge main into the path columns (#606)

Name Page.paths in the description of content_element_count (#606)

Measure the path columns again on hayro 0.9 (#606)

Update on PR #616 (head 620971c):

The owner's decision on the open point. The registry's description of content_element_count now names Page.paths, in this PR: limits.toml and what tools/generate_limits.py generates from it. The limit and its code are unchanged.
main merged, with hayro 0.9 (Move to hayro 0.9 with LaurenzV/hayro#1734 patched in, and read streams that name a file #609). hayro no longer identifies the group of a soft mask, and the content model holds only the mask's type, so Paths.soft_masks holds at most one entry per SoftMaskType. The agreed layout is otherwise unchanged.
Evidence measured again on the merge (evidence/path-columns/results/windows-amd64.md). On the 200,000-line page:
- paths() takes 0.25 s, pixmap(dpi=36) 0.21 s, Content.paths 4.0 s, and get_cdrawings() 1.03 s.
- A call leaves 693 bytes a line held, about 470 of them the memoized tuples. Record the paths of the content model as columns in the engine #615's body has the same numbers.
Checks. python tools/check.py --locked passes on the merge: pytest 6587 passed and 691 skipped, plus ruff, ty, cargo fmt, clippy, and cargo test. The NumPy and pyarrow tests pass locally.

Next. The owner reviews #616, with CI green on its head.

Open point for the owner. Since a soft mask is now only its type, the soft_mask column could hold the position in SoftMaskType, as blend_mode does, without the soft_masks tuple. The PR keeps the agreed layout until the owner decides.

Read the paths of a page as columns (#606) (#616)

After #616 merged, the owner chose to change one point of the layout agreed above: #618 (PR #621) makes soft_mask the position in SoftMaskType, or -1, as an int8, and removes Paths.soft_masks. hayro 0.9 (#609) reduced a soft mask to its type, so the tuple no longer carries anything of its own. No release holds Paths yet.

Hold the type of a soft mask in the path columns (#618) (#621)

Navigation Menu

Repository navigation

Read the paths of a page as columnar buffers #606

Context

Goal

Scope

Paths

Acceptance

Notes

The layout of the path columns

Accessor

`Paths`

Arrow

Size

Decisions

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Read the paths of a page as columnar buffers #606

Description

Context

Goal

Scope

Paths

Acceptance

Notes

Activity

yhay81 commented on Nov 3, 2026

yhay81 commented on Nov 3, 2026

The layout of the path columns

Accessor

Paths

Arrow

Size

Decisions

yhay81 commented on Nov 3, 2026

yhay81 commented on Nov 3, 2026

yhay81 commented on Nov 3, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

`Paths`