Skip to content

DESIGN.md

Latest commit

 

History

History
997 lines (875 loc) · 57.3 KB

DESIGN.md

File metadata and controls

997 lines (875 loc) · 57.3 KB
Raw

pyropdf foundation design

This document is pyropdf's design: one versioned object store that every engine reads, the contracts between the store, the engines, and the Python API, and the principles that every change keeps (section 3), with the reasons for them. Section 2 records the evidence that motivated the design, and section 13 the phases that order the work. The issue tree records the work and how far it has come; this document does not.

1. Concept

pyropdf keeps the concept of pylopdf and rebuilds its foundation (sections 2 and 14).

  • Complete control. Every feature of PDF as ISO 32000-2:2020 defines it, with its errata and extensions and the PDF/A and PDF/UA profiles, is read without loss, interpreted, created, edited, and preserved. A specification map records each feature in each of these five dimensions with the tests that prove it. pymupdf's capabilities are the floor, not the goal.
  • Consistent edits. An edit keeps every structure that depends on what it changes valid, or refuses. What it cannot keep valid, such as a signature or a conformance claim, it reports before the document is saved.
  • An essential API. One way to do each task, typed values, standard Python protocols, and one coordinate space: display space, measured from the top-left corner of the visible page with /Rotate and the CropBox applied. Section 10 states the rules.
  • Engine-faithful. hayro interprets and renders, harfrust shapes text, and work that an engine owns lands upstream first (section 6). pyropdf builds the store, the binding, the resource policy, the consistency of edits, document understanding, and what it writes: content operators, images, and fonts (section 7).
  • Trustworthy. Hostile input meets typed, bounded refusal. Results carry their provenance, independent validators check what pyropdf writes, and benchmarks publish losses with wins.
  • Everywhere. GIL-enabled and free-threaded CPython, Pyodide and Cloudflare Python Workers (#257, #258), small wheels, and the MIT license.

Underneath is one store: every engine reads and writes the document through it (F1), and nothing reconciles separate object models after the fact.

2. Evidence from pylopdf

pylopdf's costs, measured and counted on 2026-10-24, led to this design. evidence/pylopdf-baseline records them in full, with the scripts and the conditions of each run.

2.1 Measurements

Four workloads, at the largest size measured. pylopdf's results are those of pylopdf 0.15.0, and pyropdf's those of the evidence that each row links, which records its conditions. The documents repeat pylopdf's tests/assets/real_world/bill-hr815.pdf (110 pages), and the form is its f1040.pdf.

Workload pylopdf, and the cause pyropdf
Rotate every page, then read it 1,760 pages: 28.68 to 30.41 s, 18 to 24 times as long as reading the text alone. Every edit drops hayro's snapshot and every derived cache, and the next read serializes the whole lopdf graph and parses it again. Rotating and rendering 1,760 pages takes 11.5 s, 3.6 times rendering alone, where pylopdf takes 22.7 s, 7.2 times (edit-render). What still grows with the document is hayro's re-read of each version, with the bytes that the bridge builds for it, and the page index walk (section 6).
Fill one text field 2.17 to 2.19 ms with 2,626 objects, 6.39 to 6.58 ms with 8,646. Field edits clone the whole lopdf document so that they can roll back on failure. 0.04 ms with 2,663 objects as with 6,903 (form-fill-cost).
Read the value of every field 2.18 to 2.24 s for 1,600 fields, while one walk of the form takes 1.22 to 1.25 ms. A Field handle walks the whole form on every property access to find itself. 7.77 ms to list and read 1,600 fields, against 4.72 ms for one walk, which the memo computes once (form-read-cost).
Stamp text in one OpenType face on 100 pages 100 embedded font programs and 330 KiB. dedupe=True merges them only when every page has the same text (29.6 KiB); with different text it leaves 82.7 KiB. Each page.draw() commit generates its own krilla document, so each commit embeds its own subset. One program: 37.1 KiB, or 19.7 KiB with dedupe=True, and 36.7 and 24.7 KiB with different text (font-subsets).

2.2 Structure

pylopdf's code shows the same cause. About 5,500 lines of Rust exist because an engine lacks a bound, an API, or linear-time behavior, among them the reconciliation of how lopdf and hayro read malformed files. 67 call sites choose by hand how much an edit invalidates, and page cleanup, structure-tree pruning, and form import are 3,654 lines of operation-specific code. Incremental save compares every object with the original bytes, and transactions copy the whole graph, because no editing method reports what it changed. lopdf stores real numbers as f32, so every save rewrites 35.6812345 as 35.681236.

On damaged input, hayro alone rendered 103 of 112 inputs (14 files under 8 kinds of damage), while pylopdf with lopdf 0.46 read the first page of 78 (pylopdf's roadmap, evidence log).

2.3 Diagnosis

pylopdf holds three object models: lopdf's editable graph, hayro's parse of file bytes, and the PDFs that krilla generates. It reconciles them after the fact, and its costs come from one decision. lopdf's graph is the editable source of truth, hayro reads an immutable snapshot of it, and every edit discards that snapshot. As a result:

  • each edit costs as much as the document;
  • each difference between the two readers becomes pylopdf code;
  • each optimization must prove which caches an edit may keep.

Incremental save diffs the whole graph, transactions copy it, and invalidation drops everything derived from it. These are separate mechanisms, and each pays for the whole document. The items on pylopdf's roadmap that would fix this are central change tracking, exact real numbers, interpretation over a source that is not file bytes, and provenance. They are parts of one mechanism, and pyropdf builds that mechanism first.

3. Foundation principles

  • F1 One store. Every engine reads the current version of the document through the store. Nothing parses a serialized copy of it.
  • F2 Versions, not snapshots. A version is immutable. A transaction produces the next version at a cost proportional to what it writes; a failed transaction is discarded.
  • F3 Lossless by construction. A save writes every object that no edit touched with the bytes it was read from. An edited object keeps the lexemes of the values that the edit did not change.
  • F4 Derived results record what they read. A cached result stays valid while nothing it read has changed. No editing method declares what it invalidates.
  • F5 One parser, one interpreter. One iterative, bounded parser reads objects and content, and one interpreter, whose items carry provenance, interprets them. A bound refuses with an error; it never truncates silently or exhausts the stack.
  • F6 One budget per operation. Bytes, elements, decoded amplification, interpreter fuel, a deadline, and cancellation are counted in one place.
  • F7 Identity, not position. Handles name objects, and indexes and positions are computed from the current version.
  • F8 Schema-driven objects. The Arlington PDF Model generates the typed object layer and the structural check.
  • F9 Evidence in the repository. Property tests, differential oracles, fuzzers, and benchmarks are committed with the scripts that run them. CI runs the tests, the fuzzers, and the oracles that it can install; a benchmark runs again with each change that claims an effect, and tests guard what it shows wherever that can be counted.

4. The store

4.1 Base, versions, and transactions

A document is a base and a chain of versions.

  • The base holds:

    • the original bytes, shared and never copied;
    • an index from object number to location: a byte offset, or a member of an object stream;
    • the security handler;
    • the revisions that the file contains;
    • the repair report.

    Opening a document builds the index and nothing else.

  • A version holds:

    • the base;
    • a persistent map from object number to a written object (its serialized bytes) or a free entry;
    • the trailer and the next free object number;
    • the object numbers that its transaction wrote.

    Versions share structure, so a new version costs time and memory in proportion to what its transaction wrote. The map is pyropdf's own chunk table (decided in #9; measured in evidence/persistent-map): copy-on-write chunks of 16 slots, indexed by object number, under 32-way branches. A write copies one chunk and the branches above it, at most six levels for 32-bit object numbers, so its cost grows with the height of the table and not with the number of objects. The budget charges every chunk and branch before it is allocated, which bounds a write into scattered numbers at one chunk and six branches.

  • A transaction starts from the current version, collects writes, and commits by publishing the new version. A refusal or an error drops the transaction; nothing is cloned and nothing needs rolling back. Undo republishes an earlier version, within the history budget below.

  • Allocation never returns an object number twice during a document's lifetime, including after undo or a dropped transaction. The document keeps a high-water mark above every number successfully allocated or written; an older version's next number cannot lower it. Unused reserved numbers leave gaps. A refused write does not advance the mark unless the number was already reserved by an allocation.

  • History is one line of immutable versions, with undo and redo. Every committed edit is one step, including an edit outside an explicit Python transaction. A commit that writes anything discards the redo steps; a dropped transaction or one that writes nothing keeps them.

    The budget counts what each retained version holds beyond its neighbor toward the current version: unshared chunks and branches, written objects and trailers, the version record and its change set, and a history entry. Parsed-value caches and unused collection capacity are excluded. The Python default is 256 MiB; None removes the limit and zero retains no steps. The store takes the limit explicitly and sets no default. Programs with tighter memory limits, including Workers, should lower it.

    First discard the farthest redo steps while redo alone exceeds the limit. Then discard the oldest undo steps until the whole history fits. This preserves undo steps when a large undone addition cannot remain as redo. Apply the same rule after commit, undo, redo, and a limit change. These choices were accepted in #12.

  • A saved history carries the history to the same document opened again (#428): doc.history.save() returns it as bytes in pyropdf's own format, and restore() reads them back. The bytes hold every version that undo and redo reach: each as what it holds differently from the version before it, the oldest as its difference from the base, and written objects as the writer wrote them. With them go the step names, the position, the trailers, the next-number bounds, and the marks that wait for settling (section 7). They hold no object of the base, so a SHA-256 fingerprint of the base (its file's bytes, or the objects of a document without a file) names the document that they need, and a SHA-256 checksum closes them.

    Restoring refuses another document, an edited one, and a call inside a transaction. It reads, checks, and builds every version under the document's limits before it takes the writer lock, then replaces the history in one step, so data that is corrupt or that a limit refuses leaves the document and its history unchanged. The history budget then applies as after a commit. Section 4.5 says what the bytes hold of an encrypted document.

  • The change set between two versions is the union of the object numbers that their transactions wrote. Saving, invalidation, and reports all read it.

hayro-syntax parses every object on demand from its bytes, in the base or in a version (F5), so the editor and the interpreter see the same object for every number. A parsed object is cached by object number. Base objects never change, and a written object is immutable.

4.2 Values and losslessness

Objects reach the editing API as owned values: null, booleans, integers, reals, names, strings, arrays, dictionaries, streams, and references. Each value read from bytes keeps its source span, so an unchanged value is written back as its original lexeme:

  • a real stays 35.6812345;
  • a string keeps its literal or hexadecimal form;
  • a dictionary keeps its key order.

A changed real is written with the shortest representation that round-trips its f64. A stream keeps its encoded data until a caller replaces the data or asks for recompression.

4.3 Index and repair

Opening checks every uncompressed in-use cross-reference entry against the object header it points to. When the cross-reference data cannot be read or an entry fails the check, the store rebuilds the index once by scanning object headers, and records the repair. An object stream's member table is checked against the entries that name the stream when the stream is first decoded, and a mismatch resolves by one fixed rule. Repair therefore never depends on the order of reads, and no consumer repairs on its own. A repaired base refuses incremental save, because its offsets are not the file's.

4.4 Saving

  • Incremental save writes the base bytes unchanged, then the objects of the change set since the base, a cross-reference section that links to the base's, and a trailer. The bytes that existing signatures cover stay unchanged, and before anything is written the change set tells which signed objects an edit has touched.
  • Full save writes every live object. An untouched base object is copied from its bytes (from its decoded member bytes for object streams), and a written object is copied as written. Renumbering, deduplication, object streams, and recompression happen only when an option asks for them.
  • Linearized save (linearize=True, #426) is a full save in the layout of ISO 32000-2 Annex F, so that a viewer can show the first page before the rest of the file arrives. The layout is qpdf's, so that a reader that computes the hint tables itself, as qpdf --check-linearization does, finds the tables that the file holds: the linearization dictionary with the first page's cross-reference section, the catalog and the objects that open the document, the hint stream, the first page with its objects, each other page with the objects that only it uses, the objects that pages share, and the rest. The save writes what the trailer, the catalog, and the pages reach, numbered anew, and moves the attributes that pages inherit into the pages. It writes the file twice: once to measure the offsets, which fill fields of fixed width, and once to the target. It writes a cross-reference table, so it refuses object streams, a cross-reference stream, and deduplication, and an incremental save refuses it.
  • Document.is_linearized reads whether the opened file is linearized: its first object, within its first 1,024 bytes, is a linearization dictionary whose /L is the file's length. An update appended to a linearized file changes that length, so the file then reads as not linearized.
  • Saving with the same encryption copies encrypted bytes as they are. Changing or removing encryption decrypts and encrypts values again, which keeps every value exactly.

4.5 Encryption

The store decrypts strings and streams lazily as they are read, through the base's security handler, and encrypts them on write. It writes AES-256 (revision 6) and AES-128, and reads RC4 but never writes it. New objects stay plaintext in memory until a save encrypts them. A saved history (section 4.1) holds the objects that its steps wrote as the store holds them: unencrypted, even for an encrypted document.

4.6 Concurrency

A Document holds its current version behind a short-lived lock. A read holds it only to clone the version's shared pointer, and a commit only to replace that pointer; the previous version is dropped outside this lock. A read then works on its version without locks. Writes serialize through one writer lock and publish complete versions atomically. On every runtime with threads, free-threaded CPython included, reads on one document can run in parallel, and a long render keeps reading the version it started with while edits publish newer ones.

5. Derived results

Every derived result is memoized together with the set of object numbers that computing it read, which the store's resolver records. Derived results include:

  • page interpretations, and text and table pages;
  • rendering caches (fonts, images, and glyphs);
  • the page index and the form index;
  • outline and name-tree indexes;
  • link resolution and placement reuse.

A result computed on version X is valid on version Y when the change set from X to Y is disjoint from its read set. Settings that change what a computation produces, such as fallback fonts, are part of its memo key.

A result that depends on a few objects, possibly of another document, such as the Form of a placed page (section 7), records their stamps instead: for each object, what a version holds for it, which is the base's object, a written object, or a free entry, remembered without keeping the version or its objects alive. The result stays valid while every stamp holds. That is the identity that change sets compare, so writing an object again invalidates the result even when it writes the same bytes (#503).

Read sets and stamps keep invalidation correct for a new editing method without any declaration (F4), where hand-chosen invalidation needs a rule for every method (section 2.2). Versions (section 4), read sets (this section), document resources (section 7), and identity handles (section 10) together make each workload of section 2.1 cost in proportion to what it changes or reads, not to the size of the document, once hayro reads versions directly (I1 in section 6).

6. Interpretation contract

pyropdf interprets through hayro-interpret and parses through hayro-syntax. It needs the following from them, and contributes each item upstream first; the hayro tracker (#7) follows each one.

Item Requirement
I1 Object source hayro-syntax resolves objects through a source that the caller supplies (a store version), not only through file bytes: object bytes by identifier, with object-stream members resolved and decryption applied, and the trailer. Opening a source does not resolve the whole page tree: hayro resolves a page when it is read, or takes the page list from the source. It needs no public cross-reference reader, repair, or decryption.
I2 Bounded construction Colour spaces, functions, Type 4 programs, and nested content and objects are built and read without unbounded recursion, and refuse with an error beyond their bounds instead of truncating.
I3 No panics Malformed input yields errors, not panics (LaurenzV/hayro#1546 and the rest).
I4 Provenance Begin and end of every content stream, Form XObject, annotation appearance, tiling pattern, and Type 3 glyph procedure; the operator index and byte span of every painted item; the resource names it used; a callback for sh; marked-content properties; colour spaces and components; font and image identity; image-mask streams.
I5 Fuel and cancellation A budget charged per operator, per function evaluation, and per rasterized area, with a cancellation check.
I6 Linear time Object stream member tables parsed once (LaurenzV/hayro#1545), a linear search for the end of inline image data, and an operand stack that holds the 33 operands of a DeviceN scn.
I7 Spans The byte span of every parsed object and dictionary entry, which F3 needs. pyropdf's content syntax already keeps each content operation's bytes over hayro's tokenizer (#93).
I8 Object numbers Object numbers read as u32.

hayro-syntax 0.8 keeps its cross-reference reader, its repair, and its decryption crate-private (root_xref, fallback, repair, and the crypto module), and Pdf::new resolves the whole page tree. Its FromBytes trait, which parses an object from a byte slice, is public. The store therefore reads cross-reference data and decrypts on its own, and parses every object through FromBytes.

The write side stays in pyropdf (decided in #10): versions, the writer, the cross-reference reader and repair, and the security handler. hayro-syntax stays a reader, and I1 and I7 are the whole interface between them. The store's repair runs once on open and does not depend on the order of reads (section 4.3), where hayro-syntax repairs on the first read that fails; with I1, hayro reads the store's objects and its own repair is not used.

Until I1 lands, the store hands hayro its current version as an in-memory incremental update: the base bytes, followed by the written objects (already serialized) and a cross-reference section. hayro-syntax reads that format natively. That section lists every object at the store's locations and has no /Prev, whether or not the base was repaired: hayro then never reads the file's own cross-reference data, which it can read differently from the store, and never repairs on its own. An object that the store reads as absent is listed free, which hayro reads as null. A new version then costs hayro a fresh read of the cross-reference data and the page tree, but no serialization or parse of unchanged objects. That remaining cost still grows with the document, and the page tree is most of it (evidence/hayro-page-list): I1 removes the cross-reference read, and its page-tree requirement the rest.

For an encrypted base (#13), the bridge encrypts the written objects with the base's security handler, as a save that keeps the encryption does, and copies untouched objects' encrypted bytes. The cost stays that of the written objects, and no decrypted copy of the document is made. hayro-syntax accepts a password, not a file key, so the base keeps the password that opened it: private to the store, never written or shown to Python, and dropped with the base. A version whose /Encrypt is not the base's is refused, as a save refuses it. I1 makes the password unnecessary.

No pyropdf release claims safety for untrusted input before I2 is released. If the upstream patches cannot land, pyropdf bounds that construction itself before hayro builds it: an iterative walk, charged to the operation's budget, measures how deeply colour spaces, functions, Type 4 procedures, and content and objects nest, and refuses with LimitError beyond its bounds. The walk is then pyropdf's own code, listed as an engine workaround in the hayro tracker with a link to the upstream discussion (#15), and it goes when hayro releases I2.

The other engines read what callers supply: fonts, images, and text. Each must work inside the operation's budget (F6): charge what grows from input before it grows, poll the deadline and cancellation in bounded steps, and return an error instead of panicking. Where one call of an engine cannot, pyropdf bounds the input before the call and polls between calls; what stays outside the budget is a residual for the owner to accept, and an upstream issue follows it.

Engine Use Requirement
vello_cpu Rasterizes what hayro interprets CMYK output, agreed with hayro (#433). hayro converts every colour to sRGB, and vello_cpu rasterizes RGBA8 only, so Page.pixmap renders gray and RGB.
harfrust Shapes the text of embedded faces (section 7) Limits and cancellation inside one shape call (#580). pyropdf counts the text against shaped_glyph_count before it shapes, and polls between runs. harfrust reads fonts through the release of read-fonts that hayro uses (section 15).
png (image-rs) Decodes the PNG images that callers supply Its inflate state and row buffers charged to the budget, and a poll within a wide row (#561). pyropdf charges the decoded size before decoding, and polls while it fills the output.
zlib-rs Compresses Flate streams A compressor initialization that fails with an error instead of asserting (#561).
unicode-segmentation Finds grapheme clusters for text layout Its pre-context protocol, which pyropdf answers a step at a time between polls (#552).

7. Writing contract and settling

pyropdf writes the canvas itself. Paths, shapes, graphics state, clipping, transforms, opacity, blending, images, placed pages, and text become content operators, image XObjects, Form XObjects, and fonts that pyropdf writes into the store, so one canvas commits one content stream and one transaction. Images and font faces are document resources: each is registered once per document, and drawing it again reuses its objects. krilla writes nothing: its public API returns a complete PDF per drawing, which pyropdf would have to parse and import (F1), and #466 chose to write text in pyropdf rather than wait for krilla to offer an object sink and document resources.

A canvas is one operation (section 9). It takes its budget from the document's limits when its block begins, and everything inside the block charges that budget before it allocates: validation, buffered content, text layout and encoding, the objects that it reads and writes, and the join at commit. A call's max_* option inside a canvas can only tighten the operation, and a canvas inside another is part of the outer one. The deadline and cancellation stop preparation as they stop writing, and a canvas that a limit refuses, that is cancelled, or that raises writes nothing and records no undo step (#500, #505).

Placing a page (Canvas.page) reads its geometry, content, and resources from one version of the source document, a snapshot taken under the lower of the operation's and the source's limits, so an edit of the source while the canvas runs cannot mix versions. The placed page becomes a Form XObject whose resources are copied from that version, from the same document as from another, so a later edit of the source does not change it. The target document keeps up to 64 such Forms, the least recently used leaving first, and reuses one while three things hold: the page's reference, its /Contents and inherited /Resources values, and its display box are equal; the stamps (section 5) of every object that its content and resources reach still hold in the source; and the Form and its resources are unchanged in the target. An edit of anything that the page draws therefore makes a new Form, and an edit of the page's annotations or of another page does not (#503).

Text is drawn in a Standard 14 font, which is not embedded, or in an embedded face. harfrust shapes the text of an embedded face, and pyropdf writes the face as a Type0 font with the Identity-H encoding, whose CIDs are the face's glyph IDs. One font dictionary, with its descendant CIDFont, descriptor, program, and ToUnicode CMap, serves every drawing in the face. A drawing adds the glyphs it uses to the widths and to the ToUnicode CMap in its own transaction, so undoing the drawing removes them. A glyph maps to its own character, or to its whole cluster when the shaper merged characters into it. A cluster whose glyphs do not read as its text is drawn inside one marked-content sequence with /ActualText of the whole cluster, with a glyph that has a width first and the text position left at the cluster's end. A glyph without text of its own maps to U+2060 until the document draws it for real text. Text extraction honours /ActualText of the outermost sequence (evidence/cluster-text, #637).

Until a save, a face in use is a provisional object with all its glyphs, so rendering and extraction see the right glyphs. For a CID-keyed CFF program, the provisional charset maps each CID to the glyph with that original glyph ID, even if the source charset did not. The save subsets the program: it keeps the glyphs that the document draws and only the tables that ISO 32000-2 section 9.9 reads from an embedded program, renumbers the kept glyphs, and maps the CIDs that content uses to them through a CIDToGIDMap for a TrueType program or through the charset of a CID-keyed CFF program. Content codes therefore stay the face's glyph IDs: content written earlier needs no rewrite, and a drawing after a save restores the provisional program with all its glyphs until the next save. Stamping one face on 100 pages embeds one program (evidence/embedded-text).

A variable face draws at its default instance unless a weight selects another on its weight axis; a weight outside the axis refuses. Loading that instance writes a static TrueType program: every glyph keeps its ID, composites are flattened, and instructions go. From there it is a face like any other, with its own font dictionary, provisional program, and subset at save. The program is written at load and not at save because the provisional program needs all its glyphs from the first drawing, and a face of 17,936 glyphs takes about a third of a second. The program stays under the limit on font input, since flattening composites can amplify, and a coordinate that the variations push outside i16 refuses. harfrust shapes the source at the instance, because kerning has variation deltas. CFF2 stays refused: it has no PDF font type. Drawing the instance's glyphs as paths in /ActualText is not an option, for the reason below (evidence/font-instances-colour, #598).

A CIDFont draws a glyph's outline (ISO 32000-2 section 9.7), so a colour glyph that a drawing cannot paint draws its monochrome fallback outline and reads as text: any in a form field or an annotation, and on the canvas COLR, the SVG table, and a strike that is not PNG. One whose outline is empty refuses, rather than draw nothing under text that reads back, and a face with neither outlines nor colour bitmaps refuses at load.

Colour itself is drawn as a Type 3 font whose glyph procedures paint the layers or the image, with a ToUnicode CMap, because that is the form that reads as text. /ActualText over paths or a Form XObject reads as nothing in pyropdf, xpdf, MuPDF, and PDFium, which honour it only over a shown glyph.

A Type 3 font holds 256 glyphs, one for each one-byte code, and a document holds as many fonts of a face as its drawn glyphs need. What they hold is read from the document, so a drawing adds its glyphs in its own transaction, undoing it removes them, and only what the document draws is ever written. A line selects these fonts and the face's Type0 font with Tf as its glyphs ask, and a drawing with no outline glyphs writes no Type0 font.

The canvas paints the PNG strikes of sbix and CBDT this way, through the image objects that it has (evidence/colour-bitmaps, #631). A drawing takes the smallest strike that holds the glyph at 300 pixels per inch at its size, else the largest, so a glyph holds one procedure and one image per strike. A face with bitmaps and no outlines draws only on the canvas. COLR also needs a writer for shadings and soft masks, since 97.7% of Segoe UI Emoji's colour glyphs use alpha in their gradients. The SVG table is not drawn: a renderer adds 23 crates and parses XML from the input with no hook for the budget.

Dependent structures are settled rather than rebuilt by every call. A transaction marks what its writes make stale: appearance streams, outline counts, the structure tree's parent tree, name- and number-tree limits, and font subsets. Settling runs:

  • before any read that depends on a stale structure;
  • at the end of every top-level transaction;
  • before a save, which is the only point where font subsets settle.

A program never observes an unsettled structure, so the rule that setters apply immediately (section 10) keeps its meaning, while three setters on one annotation draw its appearance once.

8. Typed objects and referential rules

A generator reads a pinned release of the Arlington PDF Model (Apache-2.0, credited in a NOTICE file) and produces three things:

  • a descriptor for every object type: its keys and their types, which keys are required or inheritable, the PDF version that introduced or deprecated each key, the possible values, and the types each key links to;
  • Python typed views with stubs;
  • the structural check.

The check evaluates the descriptors and the predicates that can be decided without rendering, and reports each departure with its object and clause. It implements first the expression, version, storage, and value predicates, which decide 98% of the model's predicate cells (#216), and counts each predicate that it meets but cannot decide as unchecked instead of skipping it.

The typed layer fills the specification map's Read dimension, and the plain cases of Create and Edit, mechanically. Hand-written code is left for meaning: appearances, forms, the structure tree, and page operations.

The same link information builds a reverse-reference index per version, maintained from change sets. It records which objects refer to an object, under which key, and in which role. A structural edit runs the rules registered for the roles that refer to what it removes or moves:

  • remove the entry;
  • make the destination unresolved;
  • remove the referrer;
  • point the reference elsewhere.

Page cleanup, structure-tree pruning, and form import become such rules rather than operation-specific code (section 2.2), and a new feature adds rules instead of revisiting every structural operation.

9. Budgets and errors

  • An operation's Budget comes from DocumentLimits and the call's max_* options. It counts allocated bytes, elements, decoded bytes, and interpreter fuel, and checks a deadline and a cancellation token. A collection that grows from input grows through the budget (BudgetVec), so fallible growth is the default; crates/clippy.toml refuses the growth of the standard collections and of the store's values in every crate under crates/ that has no clippy.toml of its own.

  • An operation can span several calls, as a canvas does (section 7). Its budget is then shared, and its configuration fixed, so that a call that reads a limit once never misses a later change. A call that needs a lower limit, or that reads another document under that document's limits as well, takes a view that charges the same meters and can only be stricter, and that refuses at once when the operation has already charged more than the view allows.

  • Limit codes come from one registry. It generates the Rust enum, the stable LimitError.code strings, and the security guide's table.

  • The binding borrows Python buffers and strings and checks their sizes once, in Rust, before any copy.

  • Opening reads only the index, so damage and limits surface when a part is first read. DocumentLimits(check_at_open=True) reads every object at open instead, so that damage and the limits of objects surface at open.

  • DocumentLimits.web() is the profile for documents from outside, read in bounded workers. It bounds the file, the pages, the objects, the decoded size of each stream, of a page's content, and of all the streams of an operation, the nesting of objects, and the data that the interpreter reads. It sets check_at_open, so that a damaged upload, or one whose objects pass a limit, is refused when it opens rather than when a part of it is first used.

  • Core crates return typed errors, and the binding maps them to one hierarchy. Every error that pyropdf raises for a document is a PdfError, which derives from Exception and not from ValueError:

    • MalformedPdfError for input that cannot be read as a PDF, empty input included;
    • InternalError for a Rust panic in pyropdf or an engine: a library defect that the input exposed, with the panic's message retained so that the caller can report it;
    • PasswordError for a password that matches neither password, and EncryptedDocumentError for an operation that needs one;
    • LimitError, whose stable code names the limit that refused;
    • StaleObjectError for a handle whose object the current version does not hold, with StalePageError for a page (section 10);
    • DocumentClosedError, and OcrError for OCR models and inference.

    An argument that a call cannot take raises TypeError or ValueError, and a missing file raises FileNotFoundError. A recoverable condition, such as glyphs that text extraction leaves out, warns with PyropdfWarning.

    Every binding entry point catches unwinding Rust panics from its work before they reach PyO3, so document operations never expose PanicException. A failed transaction is discarded (F2), and the same document remains usable after InternalError; a poisoned lock must not refuse later calls. The boundary leaves the process-wide panic hook unchanged, so the default hook still prints the panic to stderr. It cannot catch a panic that aborts instead of unwinding; the PyEmscripten wheel must verify unwinding when that target is built (section 11).

10. Python API

10.1 Rules

These rules bind every public API, new or changed; a change to a rule changes this section first.

  • One way. Each task has one way to do it: there are no aliases, synonyms, compatibility layers, or deprecated twins, and no convenience that only shortens another API. Two APIs overlap only where the PDF model itself has two levels, such as a form field and its widgets. Before 1.0, a redesign removes the old API in the same release, and the changelog names what replaces it.

  • Names.

    • Properties expose cheap state of a document or a page, such as page.rotation and doc.metadata. A property whose change is one atomic edit is assignable, and a removable one supports del, as del doc.xmp does. There are no get_ and set_ pairs.
    • Collections are properties that return live views with the standard protocols: doc.pages is a sequence that supports del, insert, and extend, doc.attachments and doc.form are mappings, and page.annotations and page.links are sequences with add.
    • Computed results are methods named after what they return, with keyword-only options: page.text(), page.words(), page.tables(), page.content(), page.pixmap(), and page.svg().
    • Actions are verbs, such as save, unlock, sanitize, and apply_ocr.
    • Booleans start with is_ or has_. Names use full words, except established PDF terms such as xref, bbox, and dpi.
    • One spelling per concept: rect for a rectangle that the PDF stores, such as a page's or an annotation's, bbox for an extent that pyropdf computes, such as a word's, index for a zero-based position, and xref for an object number. A pymupdf name is used where it is also the clearest name by these rules, as Rect, Matrix, and xref are; matching pymupdf is never the reason.

  • Types.

    Kind Use Rule
    Handle Document-backed objects: Page, Annotation, Link, Field, Widget, Attachment, OutlineItem, and low-level objects Names a document and an object (section 10.2); equality compares those, and a stale handle raises StaleObjectError.
    Value Geometry, colors, fonts, pixmaps, and content items Immutable, and hashable where its content allows.
    Record Results such as Word, Block, Table, OcrWord, and Probe A frozen dataclass with slots and keyword-only fields; no positional tuples.
    Specification Input that describes something to create, such as Highlight(...), Encryption(...), and PageLabelRange(...) A frozen dataclass, validated on construction.
    Enum Codes and flags IntEnum, IntFlag, or StrEnum classes; no module-level constants.
    Option A choice among strings, such as format="png" A Literal type, with one @overload per value where the value changes the return type.

    Geometry is Point, Rect, IRect, Quad, and Matrix: immutable tuples with geometric operators (| for union, & for intersection, * for transformation, ~ for inversion), validated against non-finite coordinates, with their constants as class attributes such as Matrix.IDENTITY and Rect.EMPTY. A result is a dictionary only through an explicit to_dict() that serializes a record for JSON.

  • Coordinates. Every position on a page, read or written, is in display space: points from the top-left corner of the visible page, y downward, after the CropBox offset and /Rotate. Page.rect describes this space, and the visible page is the CropBox clipped to the MediaBox. The page boxes are read and set in display space too.

  • Pages. Page indexes are zero-based everywhere, the outline and page labels included, and negative indexes count from the end, as they do for Python sequences.

  • Errors. The exceptions are those of section 9, and an engine's refusal reaches Python as one of them.

  • Limits. Untrusted-input caps are keyword-only max_* options with bounded defaults and None as the trusted opt-out, on top of DocumentLimits (section 9). Passing one raises LimitError and returns no partial result.

  • Mutation.

    • A method that changes the document is one transaction: it plans its fallible work before it commits, and on refusal nothing changes.
    • Setters on handles apply immediately, and the structures that depend on them, such as appearances, settle before anything reads them (section 7), so no handle has an update().
    • A builder that collects drawing before it writes, such as the canvas, is a context manager that commits once on a clean exit and writes nothing after an exception.
    • An editing method keeps valid every structure that depends on what it changes, such as appearances, widths and encodings, the structure tree, and name and number trees, and its docstring names them; when it cannot, it refuses. What it invalidates and cannot repair, such as a signature over changed bytes or a PDF/A or PDF/UA claim, the change report names before a save (section 10.2).

  • Concurrency. Section 4.6 says which calls run in parallel. No process-global state changes what an operation produces: there is no font registry, and output depends on the document, the call's arguments, the fonts passed in, and the data packages of section 14 that are installed, which the API that uses them documents.

  • Engines. Each public concept documents the engine that implements it. An engine's limitation surfaces as a typed error or a documented restriction, never as a silent gap, and its fix is engine work (section 6).

  • Python 3.11. The oldest supported Python is 3.11 (section 11). Its stable ABI includes the buffer protocol, so every wheel exposes Pixmap's buffer. Dataclasses use slots=True and kw_only=True, no syntax newer than 3.11 appears, and public type aliases stay TypeAlias assignments, because a PEP 695 type statement needs 3.12 and changes what typing.get_args returns at runtime.

10.2 What the foundation gives

The store gives the API these behaviors:

  • Handles. A handle names a document and a reference (object number and generation), with a path for a direct container. Its facts are cached per version, and when the version moves it reads its own object again, never the whole structure around it. Handle and memo caches do not keep a version alive; they remember it by a weak reference or serial number. A handle raises StaleObjectError while its reference is absent from the current version, and works again if undo or redo restores that reference. A generation mismatch is stale, and allocation never reissues a number after undo (section 4.1), so an old handle cannot silently name a new object. A Page names its page object and raises StalePageError while it is absent from the current page tree. It stays valid when other pages are inserted, moved, or deleted, and page.index costs constant time.
  • Reals. A real reaches Python as pyropdf.Real, an immutable float subclass with lexeme (the bytes it was read from, or None) and to_decimal(), the exact decimal that it writes. Equality and hashing are the float's, as the store compares f64 values, and arithmetic returns a plain float. Assigning a Real writes its lexeme, or its f64 in shortest form when it has none; a float writes its shortest form; a Decimal writes its exact digits in positional form; a non-finite value raises. The lexeme travels with the value, through copies, other documents, and pickle (evidence).
  • Transactions. with doc.transaction(): groups edits into one version and one undo step, at a cost that does not depend on the document's size. doc.history undoes and redoes the steps, and its save() and restore() carry them to the same document opened again (section 4.1).
  • Revisions. save(incremental=True) writes only the change set, and the file's revisions open as documents of their own.
  • Change reports. Before a save, doc.changes(incremental=...) reports what the edits since opening invalidate: the signatures whose objects they touched, and the PDF/A or PDF/UA claims they break (#157).
  • Glyph-level results. Characters and content items are columnar buffers that NumPy and Arrow read without copying; records are views over them.
  • Typed objects. The layer of section 8 sits between the raw object model and the domain APIs.
  • Threads. Several threads may read one document.
  • Limits. DocumentLimits bounds time as well, and carries a cancellation token (section 9).

A sketch:

doc = pyropdf.open("contract.pdf")               # reads the cross-reference data only
cover = pyropdf.open("cover.pdf")
page = doc.pages[3]
doc.pages.insert(0, cover.pages[0])
assert page.index == 4                           # the handle follows its page object

with doc.transaction():                          # one version, one undo step
    page.rotation = 90
    doc.form["name"].value = "Ada Lovelace"

print(doc.changes(incremental=True).signatures)  # signed objects that the edits touched
doc.save("contract.pdf", incremental=True)       # writes only what changed

11. Layout

crates/
  pyropdf-store    objects, index and repair, versions, change sets, the
                   writer, the security handler, budgets, read-set memos,
                   the page index, the reverse-reference index, and
                   settling                                        (hayro-syntax)
  pyropdf-schema   typed object descriptors generated from the Arlington
                   PDF Model, and the structural check
  pyropdf-engine   the hayro bridge, content model and provenance,
                   rendering, writing and fonts, appearances, page
                   operations, understanding and OCR, signing      (hayro, harfrust,
                                                                    RTen, RustCrypto)
  pyropdf-python   the PyO3 binding, the only crate that depends on PyO3
python/pyropdf     the Python API
tests              Python tests: the public contract
fuzz               fuzz targets, run in CI
evidence           measurements and the scripts that produce them
tools              the work report, the checks, the generators, and the
                   drivers of the oracles
docs               the guides in English and Japanese, and the generated
                   API reference, limit codes, and specification coverage
limits.toml        the registry of limit codes (section 9)

Dependencies point down the list and never up. The store depends on no interpreter or writer, and no crate but the binding names PyO3. A crate splits further only when compile times or dependency boundaries call for it.

maturin builds the package. A release publishes, through PyPI's Trusted Publishing, one abi3-py311 wheel for each of five targets (Linux x86_64 and aarch64, macOS arm64 and x86_64, and Windows x86_64), a free-threaded wheel for each of 3.14t and 3.15t on the same targets, since an abi3 wheel does not load on a free-threaded interpreter, and an sdist. A PyEmscripten wheel, for Pyodide and Cloudflare Python Workers, is a target for 1.0 (#257, #258). OCR models stay a separate data package, which the ocr extra installs (section 14), as do the CJK faces that render the fonts that a document does not embed, which the jp, ko, zh-cn, and zh-tw extras install; canvas text draws in a Standard 14 font or in a face that the program supplies (section 7).

Signing (issue #397, evidence/signing) uses the RustCrypto line of cms 0.3, x509-cert 0.3, der 0.8, p256 and p384 0.14, pkcs8 0.11, sha2 0.11, and hmac 0.13, with cms built without its builder feature so that the rsa crate stays out of the tree. pyropdf signs with ECDSA keys itself; RSA, HSM, and PKCS#11 keys sign through a callback, and a time-stamp is a callback that returns an RFC 3161 token, so the library makes no network access.

12. Verification

  • Python tests in tests/ are the public contract, and a change to public behavior changes them. Rust tests cover what no public API exposes: the invariants of the store and the engines, through property tests, differential oracles, and fuzz targets.
  • Property tests for the store:
    • without edits, an incremental save reproduces the input byte for byte, and a full save reproduces every value exactly;
    • random sequences of transactions keep the invariants of versions: change sets, undo, and atomicity on failure.
  • The preservation harness (tests/test_preservation.py) applies each editing operation to each corpus document and saves it in full and incrementally: every object that the edit did not touch must read back equal, and an untouched page must keep its text and its rendering. The consistency harness (tests/test_consistency.py) checks the same saves with the structural check, qpdf, and pikepdf, which must find no defect that the original lacks. A new editing operation adds its row to both.
  • Differential tests against frozen pylopdf (section 14) on the corpus cover every ported capability, and each difference is recorded as an improvement or a regression (evidence/phase1-parity). pymupdf, pdf.js, and pypdfium2 read the corpus and each save of its edits as development-only oracles (tools/oracles.py), outside the package's dependencies; CI runs pdf.js.
  • Independent tools check what pyropdf writes, and CI runs each of them: veraPDF validates the PDF/A and PDF/UA corpora and their saves, qpdf and pikepdf read saved documents, poppler renders embedded-font text for comparison with hayro, and pyHanko validates signatures.
  • Fuzz targets cover the index, repair, the hayro bridge, the writer, and every new parser.
  • The four workloads of section 2.1 are measured under evidence/ (edit-render, form-fill-cost, form-read-cost, and font-subsets), and tests guard what can be counted, such as one walk of a form and one font program per face.
  • pylopdf's real-document corpus comes over with its license records.

13. Phases

The work runs in five phases. Each is a GitHub milestone with a tracker under the roadmap (#1), and the tracker holds the phase's scope and checks its exit. Phases order the work but do not gate it: an issue waits only on the issues that block it.

Phase 0: Foundation

The store and what proves it (#2): sections 4 to 6 and 9, and the core of the binding. Exit:

  • without edits, saves reproduce every corpus document: incremental saves byte for byte, full saves in every value;
  • an edit followed by a page read serializes and parses only the objects the edit wrote, apart from hayro's re-read of the cross-reference data and the page tree (section 6), and the benchmark publishes its cost at 110, 440, and 1,760 pages;
  • a failed transaction leaves the current version unchanged without copying the document.

Phase 1: Reading

The content model with provenance and fuel, text and document understanding, and rendering (#3). Exit:

  • differential parity with the frozen oracle of section 12 on the corpus, with differences documented;
  • every content item names its stream and operators.

Phase 2: Writing and interaction

The canvas and its fonts, settling, appearances, annotations, links, forms and widgets with identity handles, page operations with referential rules, and the change report (#4). Exit:

  • stamping one face on 100 pages embeds one font program;
  • reading every value of a 1,600-field form costs about one walk of the form;
  • the frozen oracle's drawing, annotation, form, and page-operation tests (section 12) pass after migration.

Phase 3: Typed objects

The Arlington generator, the structural check, the roles of the reverse-reference index, and the specification map (section 8, #5). Exit:

  • every Arlington object maps to a descriptor;
  • the check finds every defect of a corpus broken on purpose.

Phase 4: Specification

The rest of the specification on this foundation (#6), in one tracker per milestone: the content model, the canvas, reading, rendering, writing, rewriting, structure, persistence, understanding, verification, conformance, and specification completion (#245 to #256), with the continuous tracks and the watchlist (#258). Its exit is the 1.0 release, whose gates #257 tracks. 1.0 stabilizes the API, and needs:

  • a specification map (section 8) without planned or engine entries: every feature is done in each dimension that applies to it, or partial or non-goal under a recorded release decision, every partial keeps its missing behavior and its issue, and the release notes name these limitations;
  • the preservation and consistency harnesses (section 12) passing for every editing operation;
  • every engine workaround removed, or kept as pyropdf's own code with its reason recorded (section 6);
  • a guide and an end-to-end test for each canonical workflow that #257 lists;
  • every artifact of section 11 verified on its target;
  • a published stability policy.

Shipping 1.0 does not claim complete control. A claim of complete control within a declared scope needs a published scope with its exclusions, every applicable dimension inside it done with tests that pass on that release, and the harnesses and validators of section 12 passing on it.

14. Moving from pylopdf

pylopdf's users move to pyropdf in these steps.

  1. Freeze. pylopdf takes no new features after its freeze commit, ebb01d3c (#95), which is the reference for differential tests (section 12). It takes security fixes until it is archived.

  2. Porting. A module comes over with its Python tests and names the pylopdf commit it came from. What pylopdf has only because of its foundation (section 2.2) is not ported:

    • the guards, whose bounds go upstream as I2 (section 6);
    • the reconciliation between lopdf and hayro;
    • the replacements for lopdf's helpers;
    • the rules of pylopdf's AGENTS.md. Its measured facts move into this repository's evidence.

  3. Data packages. pylopdf-fonts-* and pylopdf-ocr-models hold data only, so pyropdf can use them as they are: the ocr extra installs pylopdf-ocr-models, and the jp, ko, zh-cn, and zh-tw extras install pylopdf-fonts-jp, -ko, -zh-cn, and -zh-tw. Each is republished under a pyropdf- name when a release changes it.

  4. Migration. pylopdf's codemod learns to rename imports, so code written for any pylopdf release, and pymupdf code, migrates in one step.

  5. Archive. pylopdf is archived when two conditions hold:

    • pyropdf opens, extracts, renders, reorganizes pages, and saves at least as well as frozen pylopdf;
    • the codemod migrates pylopdf's example programs.

    Before that, pylopdf's README and roadmap point here, and the open pylopdf issues worth keeping move here by issue transfer, because an archived repository is read-only. pylopdf then publishes a final release whose description points here, and its repository and PyPI project are archived. PyPI's archived status keeps existing releases installable, blocks new uploads, and can be reversed. The documentation site becomes a migration notice.

15. Risks

  • Upstream dependence. The items of section 6 need their maintainers' agreement. The bridge removes the need to wait for I1, and pyropdf's own bounds are the fallback for I2 (section 6). Until the other engines' requirements land, the residuals that section 6 lists stay outside the operation's budget. The canvas, text included, waits on no engine (section 7).
  • Text is pyropdf's code. harfrust shapes text, but the font writer and the subset at save read font programs that users supply, so fuzz targets cover them (F9). harfrust reads fonts through read-fonts, as hayro does, and pyropdf takes the harfrust release that uses hayro's version, so that harfrust adds no copy of it.
  • Later failures. Lazy opening moves some refusals from open to first use. check_at_open and the web profile keep rejection at open available.
  • Overhead. Persistent maps and read-set recording add constant costs, which the benchmarks track.
  • WebAssembly. Under Pyodide a stack overflow is fatal to the runtime, and there are no threads that could give interpretation a larger stack, so I2 matters most there.
  • Name. Searches for "pyro" find Pyro (Python Remote Objects) and Pyro (probabilistic programming).