Skip to content

Move to hayro 0.9 with LaurenzV/hayro#1734 patched in, and read streams that name a file - #609

Merged
yhay81 merged 10 commits into
mainfrom
claude/18-hayro-0.9-u32
Nov 3, 2026
Merged

yhay81 merged 10 commits into
mainfrom
claude/18-hayro-0.9-u32

Conversation

@yhay81

@yhay81 yhay81 commented Nov 3, 2026 •

Copy link
Copy Markdown
Owner

Closes #18
Closes #224

pyropdf moves to hayro 0.9.0 with LaurenzV/hayro#1734 applied, for release in 0.2.2, by the owner's decision of 2026-11-03 (recorded on #18). Awaiting the owner's review. The owner chose all three below on 2026-11-04 (「推奨どおりで」, as recommended), and the tag of 3 is pushed.

The patch

Engine (the branch's earlier commits)

  • Object numbers are u32 at every hayro call site (I8), and Reference loses its i32 caveat.
  • Glyph runs are grouped by the font dictionary of the run (GlyphRun::font_dict()): the key is Dict::cache_key() of the font dictionary, as 0.8's font_cache_key was, so fonts over one program stay apart. Glyph use for subsetting, the fonts table, and text spans read it from GlyphStyle::font.
  • RenderContent replaces render_annotations, begin_marked_content reads the MCID from the property list, and hayro-jpeg2000 is 0.5.0, hayro's.

The failures of #18's last handoff

Failure Cause Settled by
content tail (Rust, Python), test_unreadable_content_leaves_the_document_unchanged ×2 hayro-syntax reads up to 33 operands (LaurenzV/hayro#1599) tests use 34; docs say 34th
page_content_joins_streams_as_hayro_does hayro skips /Contents items that are not streams (LaurenzV/hayro#1600) PageContent::read skips them too, instead of stopping; tests show the skip
test_text_render_modes_follow_the_painting text in mode 7 reaches the device as invisible text (LaurenzV/hayro#1677) the test expects an INVISIBLE run; docstrings say so; an extraction test shows the text is extracted
value::tests: [/AB#FG] hayro-syntax 0.9 reads the name as written the test checks the name and key order; the panic guard stays for panics not known yet (#19)
value::tests: -1 0 R object numbers are u32 two integers and a keyword read as null
value::tests: 65 levels hayro-syntax 0.9 parses 512 levels the store keeps 64 as its own bound, below
subset renders ×8, and test_font_subset_poppler.py in CI hayro hints text by default (LaurenzV/hayro#1705) pyropdf renders unhinted, as with 0.8, until the owner chooses (below)

Nesting (F6). Cloning, comparing, dropping, copying for edit, and decrypting a value recurse into it, as does the binding's conversion of an operand, all bounded by hayro-syntax 0.8's 64 levels. The store's conversion now refuses deeper values itself (value::MAX_NESTING), so objects nested 65 to 512 levels stay malformed input as in 0.2.1, and the comments that named hayro's bound name the store's.

#224

A stream whose /F names a file parses with its dictionary and bytes (hayro-syntax 0.9, LaurenzV/hayro#1681), and the store refuses to decode it, as hayro-syntax does, instead of returning the bytes to ignore; without a /Filter the store returned them unread. The round-trip test of every typed view writes /F on every stream type, and the spec map's engine rule is gone (read: 634 done; no engine status left).

Tests added

  • A file whose page tree, font, and content are numbered 2^31, 3,000,000,000, u32::MAX − 2, and u32::MAX − 1: they resolve, render, extract, take an edit, and save incrementally and fully (hayro-syntax: object numbers as u32 (I8) #18's acceptance).
  • The 64-level bound in the store, the engine, and Python; /F streams in the store and in Python; clip-only text in extraction; a luminosity soft mask; the deprecation warning of SoftMask.xref, and that nothing else of a mask warns.
  • The writer's property tests generate references to any u32 number and streams with /F, which they left out for hayro-syntax 0.8.

Owner's choices (#18, 2026-11-04)

Evidence: evidence/hinted-subsets. Rendered hinted, mhlw-doc.pdf after subset_fonts differs from the original in 5,907 and 1,349 pixels of its two pages, by up to 239 of 255, and unhinted in none; canvas text in a CID-keyed CFF face differs in 873 of 36,000 pixels after the save. The auto-hinter of TrueType outlines reads 32 Latin and 150 CJK reference characters through the cmap, which a subset empties; a compacted CFF program loses its hints.

  1. Hinting: pyropdf renders unhinted in 0.2.2, as this PR does, so that a patch release changes no render. Keeping the auto-hinter's reference characters in subsets is rejected: about 150 CJK glyphs more in each subset. Follow-ups: Render text hinted on request: a hinting option of page.pixmap and Document.render #612, a hinting=True option of page.pixmap and Document.render, which only adds to the API; Keep a CFF face's hints when a save compacts it #611, compacted CFF programs that lose their hints, which matters to other viewers too.
  2. SoftMask.xref: kept as 0, and reading it warns with DeprecationWarning. A private field holds the value, so building (SoftMask(subtype=...) and SoftMask(xref=..., subtype=...)), comparing, hashing, printing, and pickling a mask do not warn, and it stays a frozen dataclass. Not restored through hayro's Debug hash, which is no API. Remove the deprecated SoftMask.xref in the next minor release #610 removes it in the next minor release.
  3. The tag pyropdf/glyph-run-font-dict on 394e4a0d goes to yhay81/hayro, so that released sdists keep fetching the commit if the branch of hayro-interpret: Expose the font dictionary of a glyph run LaurenzV/hayro#1734 is force-pushed. Pushed: a signed annotated tag, 972ea320.

For 0.2.2's changelog

  • pyropdf builds against hayro 0.9.0 with hayro-interpret: Expose the font dictionary of a glyph run LaurenzV/hayro#1734 applied, which hayro has not released: every hayro crate comes from yhay81/hayro commit 394e4a0d, which also holds two fixes after 0.9.0 (ToUnicode maps written with cidrange, hayro-font: Read Unicode from a ToUnicode map written with cidranges LaurenzV/hayro#1730; fallible allocation in CCITT decoding, hayro-ccitt: Allocate fallibly LaurenzV/hayro#1733), and a build from the sdist fetches it from GitHub (hayro-syntax: object numbers as u32 (I8) #18).
  • Objects numbered above 2,147,483,647 resolve in rendering and text extraction (hayro-syntax: object numbers as u32 (I8) #18).
  • A stream whose /F names an external file reads, with its dictionary and its bytes, and decoding its data raises MalformedPdfError instead of returning the bytes to ignore (hayro-syntax: parse a stream whose dictionary names an external file #224).
  • Content operators with up to 33 operands, such as scn with up to 32 DeviceN components, are read, rendered, and extracted; reading stopped at the 11th operand, which also left the rest of the page unrendered.
  • An item of a page's /Contents array that is not a stream is skipped, and the streams after it are read; reading stopped at it.
  • Text in render mode 7, which only clips, is extracted, and page.content() reports it as an INVISIBLE text run.
  • A name whose # is not followed by two hexadecimal digits reads as written instead of raising MalformedPdfError.
  • SoftMask.xref is deprecated: reading it warns with DeprecationWarning, and it is 0, since hayro 0.9 no longer identifies the group of a soft mask. The next minor release removes it.
  • Operations.parse and page.operations raise MalformedPdfError for an operand nested deeper than 64 levels, where reading stopped before it.
  • hayro 0.9.0's fixes and performance work reach rendering and extraction, as its changelogs list them (patterns, soft masks, clipping, embedded fonts, the Unicode of glyph names, malformed files); text still renders unhinted.

Checks

On Windows, after rebasing onto v0.2.1:

  • cargo fmt --all --check, cargo clippy --workspace --all-targets --all-features --locked -- -D warnings, cargo test --workspace --all-features --locked: pass (engine 184, store 327 and 2 ignored, corpus 2)
  • uv run pytest: 6,544 passed, 689 skipped
  • uv run ruff check ., uv run ruff format --check ., uv run ty check: pass.
  • cargo check of fuzz/ against the patch: pass.
  • The sdist of maturin 1.15.0 installs from source into a fresh Python 3.14 environment (12 min 48 s, with Cargo's git cache warm), and the result opens, extracts, and renders mhlw-doc.pdf. GitHub serves 394e4a0d by its hash (git fetch --depth 1), as Cargo fetches a full rev.
  • CI on d30ae65: green, the six fuzz targets included.

Follow-ups: #603 (remove the patch), #608 (TilingPattern.opacity, which hayro 0.9 leaves to devices), #610 (remove SoftMask.xref in the next minor release), #611 (keep CFF hints in compacted programs), #612 (a hinting option). After this lands, #584 no longer blocks #19, #365, and #437, which the patched commit already serves.

🤖 Generated with Claude Code

yhay81 and others added 8 commits November 3, 2026 22:59
Bumps hayro, hayro-svg, hayro-syntax to 0.9.0 and hayro-jpeg2000 to
0.5.0, reads object numbers as u32, and follows 0.9's API changes.
It does not build against crates.io yet: it calls
OutlineGlyph::font_cache_key and SoftMask::group_id, which #584 asks
hayro to add.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Follows #584's redesign: glyph runs are grouped by the font dictionary
that hayro's proposed GlyphRun::font_dict reports, instead of
OutlineGlyph::font_cache_key, with the same key as before. SoftMask.xref
goes away, since hayro 0.9 removed the mask's group ID on purpose
(LaurenzV/hayro#1553); the engine passes the mask type as a code.

It does not build against crates.io yet: GlyphRun::font_dict is what
#584 asks hayro to add.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
hayro 0.9.0 lacks GlyphRun::font_dict() (LaurenzV/hayro#1734, #584).
By the owner's decision of 2026-11-03, main takes every hayro crate
from yhay81/hayro 394e4a0d, hayro main 4f7d16fa with #1734 on top,
through [patch.crates-io] until a hayro release contains it. The
crates name each other by path, so their types agree only when they
all come from that commit. fuzz/Cargo.toml repeats the table; #603
removes both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
hayro 0.9 hints text by default (LaurenzV/hayro#1705). Hinting reads
more of a font program than the glyphs drawn: the auto-hinter's
reference characters of a TrueType program, and the hints of a CFF
one. The programs that pyropdf subsets therefore draw differently
hinted, which fails the tests that compare renders before and after
subsetting. Until the owner chooses what to do about it, pyropdf
renders unhinted, as it did with hayro 0.8. evidence/hinted-subsets
measures the difference and lists the choices.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…224)

- Content: hayro-syntax reads up to 33 operands per operator
  (LaurenzV/hayro#1599), and a page's /Contents array skips the items
  that are not streams (LaurenzV/hayro#1600), which PageContent now
  mirrors instead of stopping there.
- Text in render mode 7 reaches the device as invisible text
  (LaurenzV/hayro#1677): an INVISIBLE run, and extracted text.
- Values: a name whose # is not followed by two hexadecimal digits
  reads as written, -1 0 R is no reference, and hayro-syntax parses 512
  levels of nesting. The store keeps 64 as its own bound, since
  cloning, comparing, dropping, copying, and decrypting a value recurse
  into it (F6).
- SoftMask.xref stays, deprecated and 0: hayro 0.9 no longer identifies
  a soft mask's group (LaurenzV/hayro#1553).
- A stream whose /F names a file reads, and decoding its data refuses
  instead of returning the bytes to ignore (#224). The round-trip test
  writes /F on every stream type, and the spec map's engine rule goes.
- A test of a file whose objects are numbered from 2^31 to
  u32::MAX - 1: they resolve, render, extract, and save (#18).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s properties (#18, #224)

The property tests of the writer kept references to i32 object numbers
and streams without /F, which hayro-syntax 0.8 could not read back.
hayro-syntax 0.9 reads both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The engine passes the type of a soft mask as a code since hayro 0.9; the test covered only alpha.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The owner chose on 2026-11-04 to keep SoftMask.xref as 0 and to warn
with DeprecationWarning when it is read. A private field holds the
value, so that building, comparing, hashing, printing, and pickling a
mask read no xref and do not warn, and SoftMask(subtype=...) and
SoftMask(xref=..., subtype=...) both still build one. #610 removes it
in the next minor release.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@yhay81
yhay81 merged commit af7f21e into main Nov 3, 2026
14 checks passed
yhay81 added a commit that referenced this pull request Nov 3, 2026
hayro 0.9 no longer identifies the group of a soft mask (#609), and the
content model now holds only its type, so `Paths.soft_masks` holds at most
one entry per type.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
yhay81 added a commit that referenced this pull request Nov 3, 2026
* Hold the type of a soft mask in the path columns (#618)

hayro 0.9 reduced a soft mask to its type (#609), so the `soft_mask` column
now holds the position in `SoftMaskType` directly, or -1, as an int8 like
the other enum columns, and `Paths.soft_masks` is gone. No release holds
`Paths` yet. The owner chose this change to the layout agreed in #606.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Measure the path columns with the type of a soft mask (#618)

The columns hold 221 bytes a path on the line pages, 3 fewer than with the
index, and the other figures stay within the machine's noise.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
yhay81 added a commit that referenced this pull request Nov 4, 2026
Since hayro 0.9's patch (#609), which only the root and fuzz manifests repeat, the probe no longer builds whatever it passes render::raster, and an evidence crate records what its measurement ran.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
yhay81 added a commit that referenced this pull request Nov 4, 2026
…600) (#613)

* Render CJK fonts that a document does not embed with fallback faces (#600)

hayro asks its font resolver for a font that a document does not embed,
and its default resolver answers with a Standard 14 font, which has no
CJK glyphs, so such text rendered blank. A resolver ported from pylopdf
4ff4a266 (rust/src/document/fonts.rs) now answers first: the language
comes from the character collection (Adobe-Japan1, Korea1, GB1, CNS1),
or from the BaseFont name for Adobe-Identity, and a name that marks a
serif face, such as a Mincho, takes the serif face.

- pyropdf-engine: render::Fallback holds a sans and a serif face per
  language, a loaded face or a font file read the first time that a page
  needs it. Every rendering takes it (raster, batch, svg), and the
  thread's render cache is keyed by its faces as well as the bridge
  (DESIGN.md section 5).
- pyropdf-python: each _Document holds its faces, which
  set_fallback_faces replaces; page and annotation pixmaps, batches,
  and SVG render with them.
- Python: FallbackFont and doc.fallback_fonts, a mapping by language
  that holds the faces of the installed pylopdf-fonts-* packages until
  it is edited. The jp, ko, zh-cn, and zh-tw extras install them, and
  the dev group installs them for the tests.
- The differential harness renders tests/assets/text/nonembedded-cjk.pdf
  with the extras' faces in both engines, and the corpus without.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Explain the parity differences of the fallback pages and of #460

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Measure the fallback pages and the corpus against frozen pylopdf again

run.py now builds frozen pylopdf with its CJK font extras, dumps the
corpus without fallback fonts and tests/assets/text/nonembedded-cjk.pdf
with them, and measures pdfium-ocg-invisible.pdf too. All 163 rasters
match byte for byte. The run also finds what changed since 2026-10-28:
the highlight of pdfium-links-highlights-annots now renders alike (#172),
and page 2 of patent-us223898 keeps its headings whole (#460).
compare.py reads the text of a layout's spans from their characters.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Keep the fallback settings crate-private and drop an unused package field

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Render without fallback faces in the bridge and interpret fuzz targets

render::raster and render::svg take the faces that draw CJK fonts that a document does not embed; the fuzz targets pass none, which renders as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Render without fallback faces in the edit-render split probe

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Leave the edit-render split probe as it was measured

Since hayro 0.9's patch (#609), which only the root and fuzz manifests repeat, the probe no longer builds whatever it passes render::raster, and an evidence crate records what its measurement ran.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Measure the parity evidence again on hayro 0.9

Frozen pylopdf renders on hayro 0.8, and pyropdf now on hayro 0.9, whose
glyph cache moves filled text by less than a pixel on 152 of the 163
rasters. With the cache off, 162 match byte for byte, so the report names
it as their cause, and #622 asks the owner whether 0.2.2 keeps it. The
reading outputs stay as they were.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

hayro-syntax: parse a stream whose dictionary names an external file hayro-syntax: object numbers as u32 (I8)

1 participant