Repository navigation
Move to hayro 0.9 with LaurenzV/hayro#1734 patched in, and read streams that name a file - #609
Merged
Merged
Conversation
Bumps hayro, hayro-svg, hayro-syntax to 0.9.0 and hayro-jpeg2000 to 0.5.0, reads object numbers as u32, and follows 0.9's API changes. It does not build against crates.io yet: it calls OutlineGlyph::font_cache_key and SoftMask::group_id, which #584 asks hayro to add. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Follows #584's redesign: glyph runs are grouped by the font dictionary that hayro's proposed GlyphRun::font_dict reports, instead of OutlineGlyph::font_cache_key, with the same key as before. SoftMask.xref goes away, since hayro 0.9 removed the mask's group ID on purpose (LaurenzV/hayro#1553); the engine passes the mask type as a code. It does not build against crates.io yet: GlyphRun::font_dict is what #584 asks hayro to add. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
hayro 0.9.0 lacks GlyphRun::font_dict() (LaurenzV/hayro#1734, #584). By the owner's decision of 2026-11-03, main takes every hayro crate from yhay81/hayro 394e4a0d, hayro main 4f7d16fa with #1734 on top, through [patch.crates-io] until a hayro release contains it. The crates name each other by path, so their types agree only when they all come from that commit. fuzz/Cargo.toml repeats the table; #603 removes both. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
hayro 0.9 hints text by default (LaurenzV/hayro#1705). Hinting reads more of a font program than the glyphs drawn: the auto-hinter's reference characters of a TrueType program, and the hints of a CFF one. The programs that pyropdf subsets therefore draw differently hinted, which fails the tests that compare renders before and after subsetting. Until the owner chooses what to do about it, pyropdf renders unhinted, as it did with hayro 0.8. evidence/hinted-subsets measures the difference and lists the choices. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…224) - Content: hayro-syntax reads up to 33 operands per operator (LaurenzV/hayro#1599), and a page's /Contents array skips the items that are not streams (LaurenzV/hayro#1600), which PageContent now mirrors instead of stopping there. - Text in render mode 7 reaches the device as invisible text (LaurenzV/hayro#1677): an INVISIBLE run, and extracted text. - Values: a name whose # is not followed by two hexadecimal digits reads as written, -1 0 R is no reference, and hayro-syntax parses 512 levels of nesting. The store keeps 64 as its own bound, since cloning, comparing, dropping, copying, and decrypting a value recurse into it (F6). - SoftMask.xref stays, deprecated and 0: hayro 0.9 no longer identifies a soft mask's group (LaurenzV/hayro#1553). - A stream whose /F names a file reads, and decoding its data refuses instead of returning the bytes to ignore (#224). The round-trip test writes /F on every stream type, and the spec map's engine rule goes. - A test of a file whose objects are numbered from 2^31 to u32::MAX - 1: they resolve, render, extract, and save (#18). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The engine passes the type of a soft mask as a code since hayro 0.9; the test covered only alpha. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Nov 3, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2 of 3 tasks
The owner chose on 2026-11-04 to keep SoftMask.xref as 0 and to warn with DeprecationWarning when it is read. A private field holds the value, so that building, comparing, hashing, printing, and pickling a mask read no xref and do not warn, and SoftMask(subtype=...) and SoftMask(xref=..., subtype=...) both still build one. #610 removes it in the next minor release. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Nov 3, 2026
4 tasks
yhay81
added a commit
that referenced
this pull request
Nov 3, 2026
hayro 0.9 no longer identifies the group of a soft mask (#609), and the content model now holds only its type, so `Paths.soft_masks` holds at most one entry per type. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Nov 3, 2026
yhay81
added a commit
that referenced
this pull request
Nov 3, 2026
* Hold the type of a soft mask in the path columns (#618) hayro 0.9 reduced a soft mask to its type (#609), so the `soft_mask` column now holds the position in `SoftMaskType` directly, or -1, as an int8 like the other enum columns, and `Paths.soft_masks` is gone. No release holds `Paths` yet. The owner chose this change to the layout agreed in #606. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Measure the path columns with the type of a soft mask (#618) The columns hold 221 bytes a path on the line pages, 3 fewer than with the index, and the other figures stay within the machine's noise. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
4 tasks
yhay81
added a commit
that referenced
this pull request
Nov 4, 2026
Since hayro 0.9's patch (#609), which only the root and fuzz manifests repeat, the probe no longer builds whatever it passes render::raster, and an evidence crate records what its measurement ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
yhay81
added a commit
that referenced
this pull request
Nov 4, 2026
…600) (#613) * Render CJK fonts that a document does not embed with fallback faces (#600) hayro asks its font resolver for a font that a document does not embed, and its default resolver answers with a Standard 14 font, which has no CJK glyphs, so such text rendered blank. A resolver ported from pylopdf 4ff4a266 (rust/src/document/fonts.rs) now answers first: the language comes from the character collection (Adobe-Japan1, Korea1, GB1, CNS1), or from the BaseFont name for Adobe-Identity, and a name that marks a serif face, such as a Mincho, takes the serif face. - pyropdf-engine: render::Fallback holds a sans and a serif face per language, a loaded face or a font file read the first time that a page needs it. Every rendering takes it (raster, batch, svg), and the thread's render cache is keyed by its faces as well as the bridge (DESIGN.md section 5). - pyropdf-python: each _Document holds its faces, which set_fallback_faces replaces; page and annotation pixmaps, batches, and SVG render with them. - Python: FallbackFont and doc.fallback_fonts, a mapping by language that holds the faces of the installed pylopdf-fonts-* packages until it is edited. The jp, ko, zh-cn, and zh-tw extras install them, and the dev group installs them for the tests. - The differential harness renders tests/assets/text/nonembedded-cjk.pdf with the extras' faces in both engines, and the corpus without. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Explain the parity differences of the fallback pages and of #460 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Measure the fallback pages and the corpus against frozen pylopdf again run.py now builds frozen pylopdf with its CJK font extras, dumps the corpus without fallback fonts and tests/assets/text/nonembedded-cjk.pdf with them, and measures pdfium-ocg-invisible.pdf too. All 163 rasters match byte for byte. The run also finds what changed since 2026-10-28: the highlight of pdfium-links-highlights-annots now renders alike (#172), and page 2 of patent-us223898 keeps its headings whole (#460). compare.py reads the text of a layout's spans from their characters. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Keep the fallback settings crate-private and drop an unused package field Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Render without fallback faces in the bridge and interpret fuzz targets render::raster and render::svg take the faces that draw CJK fonts that a document does not embed; the fuzz targets pass none, which renders as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Render without fallback faces in the edit-render split probe Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Leave the edit-render split probe as it was measured Since hayro 0.9's patch (#609), which only the root and fuzz manifests repeat, the probe no longer builds whatever it passes render::raster, and an evidence crate records what its measurement ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Measure the parity evidence again on hayro 0.9 Frozen pylopdf renders on hayro 0.8, and pyropdf now on hayro 0.9, whose glyph cache moves filled text by less than a pixel on 152 of the 163 rasters. With the cache off, 162 match byte for byte, so the report names it as their cause, and #622 asks the owner whether 0.2.2 keeps it. The reading outputs stay as they were. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Nov 4, 2026
Closed
This was referenced Nov 4, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #18
Closes #224
pyropdf moves to hayro 0.9.0 with LaurenzV/hayro#1734 applied, for release in 0.2.2, by the owner's decision of 2026-11-03 (recorded on #18). Awaiting the owner's review. The owner chose all three below on 2026-11-04 (「推奨どおりで」, as recommended), and the tag of 3 is pushed.
The patch
GlyphRun::font_dict(), hayro: expose the font dictionary of a glyph run #584),[patch.crates-io]takes every hayro crate fromyhay81/hayro394e4a0d: hayromain4f7d16fa(0.9.0 plus hayro-font hayro-font: Read Unicode from a ToUnicode map written with cidranges LaurenzV/hayro#1730 and hayro-ccitt hayro-ccitt: Allocate fallibly LaurenzV/hayro#1733) with #1734 on top. The hayro crates name each other by path, so patching hayro-interpret alone would bring a second hayro-syntax whose types are not the store's; from one commit,Cargo.locklists each hayro crate once.fuzz/Cargo.tomlrepeats the table, andfuzz/Cargo.lockfollows (the fuzz targets type-check against it). Build against released hayro again: remove the [patch.crates-io] of LaurenzV/hayro#1734 #603, blocked by hayro: expose the font dictionary of a glyph run #584, removes both.maturin sdist(1.15.0) keeps the table and the git sources ofCargo.lock, so a build from the sdist fetches394e4a0dfrom GitHub.Engine (the branch's earlier commits)
u32at every hayro call site (I8), andReferenceloses itsi32caveat.GlyphRun::font_dict()): the key isDict::cache_key()of the font dictionary, as 0.8'sfont_cache_keywas, so fonts over one program stay apart. Glyph use for subsetting, the fonts table, and text spans read it fromGlyphStyle::font.RenderContentreplacesrender_annotations,begin_marked_contentreads the MCID from the property list, and hayro-jpeg2000 is 0.5.0, hayro's.The failures of #18's last handoff
test_unreadable_content_leaves_the_document_unchanged×2page_content_joins_streams_as_hayro_does/Contentsitems that are not streams (LaurenzV/hayro#1600)PageContent::readskips them too, instead of stopping; tests show the skiptest_text_render_modes_follow_the_paintingINVISIBLErun; docstrings say so; an extraction test shows the text is extractedvalue::tests:[/AB#FG]value::tests:-1 0 Ru32value::tests: 65 levelstest_font_subset_poppler.pyin CINesting (F6). Cloning, comparing, dropping, copying for edit, and decrypting a value recurse into it, as does the binding's conversion of an operand, all bounded by hayro-syntax 0.8's 64 levels. The store's conversion now refuses deeper values itself (
value::MAX_NESTING), so objects nested 65 to 512 levels stay malformed input as in 0.2.1, and the comments that named hayro's bound name the store's.#224
A stream whose
/Fnames a file parses with its dictionary and bytes (hayro-syntax 0.9, LaurenzV/hayro#1681), and the store refuses to decode it, as hayro-syntax does, instead of returning the bytes to ignore; without a/Filterthe store returned them unread. The round-trip test of every typed view writes/Fon every stream type, and the spec map'senginerule is gone (read: 634 done; noenginestatus left).Tests added
/Fstreams in the store and in Python; clip-only text in extraction; a luminosity soft mask; the deprecation warning ofSoftMask.xref, and that nothing else of a mask warns.u32number and streams with/F, which they left out for hayro-syntax 0.8.Owner's choices (#18, 2026-11-04)
Evidence:
evidence/hinted-subsets. Rendered hinted, mhlw-doc.pdf aftersubset_fontsdiffers from the original in 5,907 and 1,349 pixels of its two pages, by up to 239 of 255, and unhinted in none; canvas text in a CID-keyed CFF face differs in 873 of 36,000 pixels after the save. The auto-hinter of TrueType outlines reads 32 Latin and 150 CJK reference characters through thecmap, which a subset empties; a compacted CFF program loses its hints.hinting=Trueoption ofpage.pixmapandDocument.render, which only adds to the API; Keep a CFF face's hints when a save compacts it #611, compacted CFF programs that lose their hints, which matters to other viewers too.SoftMask.xref: kept as 0, and reading it warns withDeprecationWarning. A private field holds the value, so building (SoftMask(subtype=...)andSoftMask(xref=..., subtype=...)), comparing, hashing, printing, and pickling a mask do not warn, and it stays a frozen dataclass. Not restored through hayro'sDebughash, which is no API. Remove the deprecated SoftMask.xref in the next minor release #610 removes it in the next minor release.pyropdf/glyph-run-font-dicton394e4a0dgoes toyhay81/hayro, so that released sdists keep fetching the commit if the branch of hayro-interpret: Expose the font dictionary of a glyph run LaurenzV/hayro#1734 is force-pushed. Pushed: a signed annotated tag,972ea320.For 0.2.2's changelog
cidrange, hayro-font: Read Unicode from a ToUnicode map written with cidranges LaurenzV/hayro#1730; fallible allocation in CCITT decoding, hayro-ccitt: Allocate fallibly LaurenzV/hayro#1733), and a build from the sdist fetches it from GitHub (hayro-syntax: object numbers as u32 (I8) #18)./Fnames an external file reads, with its dictionary and its bytes, and decoding its data raisesMalformedPdfErrorinstead of returning the bytes to ignore (hayro-syntax: parse a stream whose dictionary names an external file #224).scnwith up to 32 DeviceN components, are read, rendered, and extracted; reading stopped at the 11th operand, which also left the rest of the page unrendered./Contentsarray that is not a stream is skipped, and the streams after it are read; reading stopped at it.page.content()reports it as anINVISIBLEtext run.#is not followed by two hexadecimal digits reads as written instead of raisingMalformedPdfError.SoftMask.xrefis deprecated: reading it warns withDeprecationWarning, and it is 0, since hayro 0.9 no longer identifies the group of a soft mask. The next minor release removes it.Operations.parseandpage.operationsraiseMalformedPdfErrorfor an operand nested deeper than 64 levels, where reading stopped before it.Checks
On Windows, after rebasing onto
v0.2.1:cargo fmt --all --check,cargo clippy --workspace --all-targets --all-features --locked -- -D warnings,cargo test --workspace --all-features --locked: pass (engine 184, store 327 and 2 ignored, corpus 2)uv run pytest: 6,544 passed, 689 skippeduv run ruff check .,uv run ruff format --check .,uv run ty check: pass.cargo checkoffuzz/against the patch: pass.394e4a0dby its hash (git fetch --depth 1), as Cargo fetches a fullrev.d30ae65: green, the six fuzz targets included.Follow-ups: #603 (remove the patch), #608 (
TilingPattern.opacity, which hayro 0.9 leaves to devices), #610 (removeSoftMask.xrefin the next minor release), #611 (keep CFF hints in compacted programs), #612 (ahintingoption). After this lands, #584 no longer blocks #19, #365, and #437, which the patched commit already serves.🤖 Generated with Claude Code