Repository navigation
Recover Type 3 character codes when Unicode mappings are absent #437
Description
Context
DESIGN.md sections 1, 3, 6 and 12 require engine-faithful text extraction with documented provenance. The PyMuPDF comparison #419 compares tests/assets/real_world/pdfium-type3.pdf: PyMuPDF 2.0 extracts A\nA\nA\n, while current pyropdf skips three glyphs and warns; unmapped="glyph_id" produces replacement characters for Type 3. The Type 3 font in this PDF has no /ToUnicode, and its /Encoding /Differences maps character code 65 to the custom glyph name /C1. PyMuPDF's result is a fallback to investigate, not proven ground truth: arbitrary Type 3 character codes do not guarantee Unicode semantics. tests/test_text.py::test_type3_glyphs_without_unicode_have_no_glyph_id documents current behavior. #247 lists unknown-Unicode options.
Goal
Expose enough original Type 3 character-code and font-encoding information to let callers recover or choose a documented best-effort fallback when Unicode mappings are absent, without silently claiming the inferred character is correct.
Scope
In: inspect hayro's Type 3 glyph callbacks, propose and implement upstream support if necessary, adopt a release, and expose a bounded, explicit fallback policy or original code through pyropdf. Honor valid /ToUnicode and standard glyph-name mappings when available. Preserve or deliberately document default skip-and-warning behavior. Out: inventing Unicode for arbitrary custom glyph names or treating PyMuPDF as semantic ground truth.
Paths
- Upstream hayro as needed; local proposals and evidence under
evidence/ crates/pyropdf-engine/src/extract/,crates/pyropdf-python/src/text.rs,python/pyropdf/_text.py,python/pyropdf/_page.pytests/test_text.pyand Type 3 fixturesdocs/en/api/text-and-content.mdanddocs/en/api/pages.md(generated from the public option documentation and verified by tests/test_docs.py)
Acceptance
- Document the information hayro provides for Type 3 glyphs and any upstream design and release needed; obtain owner approval before posting upstream.
- On the corpus fixture, the original character code 65 is recoverable or a documented opt-in fallback returns
A; default behavior and its warning are explicitly tested. - Valid
/ToUnicode, standard names, a custom non-Latin glyph name, and an invalid or ambiguous mapping are tested; no ambiguous Unicode is silently asserted as known. - Text, words, and character results have consistent mappings; limits and malformed input remain bounded.
Notes
Compare with PyMuPDF 2.0 and at least one independent reference, and label heuristic output as such. The current implementation handles Glyph::Type3(_) as a replacement when unmapped="glyph_id".
Activity
yhay81 commentedon Nov 2, 2026
Claimed by claude on branch
claude/437-type3-char-codes: investigating what hayro 0.8.0 and main provide for Type 3 glyphs (first acceptance item).yhay81 commentedon Nov 2, 2026
Handoff for the first acceptance item. PR: #563 (
claude/437-type3-char-codes, evidence only, awaiting owner review).What hayro gives (details and probe output in
evidence/type3-char-codes/README.mdon that branch):as_unicode()from/ToUnicode. No character code, no glyph name, and even a standard name such as/Ais unmapped. That is why pyropdf skips the fixture's glyphs.main: hayro-interpret: Fall back to glyph names for Type 3 Unicode mapping LaurenzV/hayro#1512 (merged 2026-10-17, fixes Type3 fonts without a ToUnicode CMap:as_unicode()is alwaysNone, and the character code is not accessible either LaurenzV/hayro#1331, supersedes the closed hayro-interpret: Improve Unicode recovery for Type3 fonts LaurenzV/hayro#1332) addsGlyph::char_code()(65 onpdfium-type3.pdf). It also adds a glyph-name fallback that keeps valid/ToUnicodefirst, skips a U+0000 entry, and maps standard,uniXXXX, and AGL non-Latin names. Custom/C1still givesNone, so hayro asserts nothing. hayro-interpret: AddType3Glyph::advance_widthLaurenzV/hayro#1678 addsType3Glyph::advance_width. The font layer moved tohayro-font(#1696), so expect import-path changes on adoption.Decision: no new upstream change or draft is needed. The required change is merged and waits on a hayro release. I moved the label from
upstream:todotoupstream:merged(not removingupstream, since pyropdf cannot build on it until a release). I also released the claim.Next action: when a hayro release contains #1512, drop
upstreamandupstream:merged, labeladoption, and implement the remaining acceptance items onGlyph::char_code(). Interpreting code 65 asAstays an opt-in, labeled pyropdf policy. The encoding's glyph name is not exposed by hayro, but #437 does not need it.Document what hayro gives for Type 3 glyphs (part of #437) (#563)
yhay81 commentedon Nov 3, 2026
Triage, 2026-11-03: hayro 0.9.0 (released 2026-11-02) contains the upstream fixes this issue waited on, so I replaced
upstreamandupstream:mergedwithadoption. pyropdf cannot adopt 0.9 yet: it dropsOutlineGlyph::font_cache_key()andSoftMask::id(), which the engine reads. #584 tracks putting them back (LaurenzV/hayro#1734, awaiting review), so this issue is now blocked by #584. The 0.9 bump itself resumes on #18's branchclaude/18-hayro-0.9-u32.Next action: once a hayro release contains LaurenzV/hayro#1734 and the bump lands, adopt this issue's change.
yhay81 commentedon Nov 5, 2026
Dependency triage, 2026-11-06: removed the obsolete blocked-by link to #584. PR #609 already merged the owner-approved hayro 0.9-based commit
394e4a0d37baf8af513f6ebda15f4bada85d6f23, and both the workspace and fuzz manifests use it. This issue's 0.9 capability is available now; #603 separately retains the #584 dependency for returning to published crates. This does not close #584 or waive any acceptance criterion here.Next action: Implement the remaining explicit Type 3 character-code recovery policy and tests using the already available Glyph::char_code API.
No implementation claim is held by this triage. Recheck the live queue, claim a branch and complete the existing acceptance before closing.
yhay81 commentedon Nov 6, 2026
Claimed on codex/437-type3-char-code. Implement the opt-in Type 3 fallback and verify text, words, characters, mapping precedence and limits against the corpus and synthetic cases.
yhay81 commentedon Nov 6, 2026
Implemented on codex/437-type3-char-code in #712, head 0b305bb. The shared extraction path recovers Type 3 source codes on explicit request. Tests cover the corpus and synthetic mappings, budget refusals, and consistent text/word/character results. Compared PyMuPDF 2.0 and independent PDFium 156, and updated the generated public reference. All Rust and static checks passed. After correcting the stale generated reference, all 180 documentation and text tests passed. Next: verify current-head CI, review threads and acceptance, then squash-merge.
Expose opt-in Type 3 character-code recovery (#437) (#712)
yhay81 commentedon Nov 6, 2026
Merged #712 at 742fea1 after Python, Rust, audit, documentation build and all nine fuzz jobs passed. The default warning, opt-in code recovery, mapping precedence, three output forms and extraction limits are covered. Closed and released the claim.