Repository navigation
Render text in non-embedded CJK fonts with fallback faces #600
Description
Context
DESIGN.md section 14: pylopdf is archived once pyropdf "opens, extracts, renders, reorganizes pages, and saves at least as well as frozen pylopdf", and the pylopdf-fonts-* packages hold data only, so pyropdf can use them as they are. Text in a CJK font that the PDF does not embed is a rendering gap against pylopdf. One page that shows 計画地 in MS-Gothic, with no FontFile*:
| Library | page.text() |
Dark pixels in page.pixmap(dpi=72) |
Empty glyph outlines in page.svg() |
|---|---|---|---|
pyropdf main 47744b9 and PyPI 0.2.0, with or without pylopdf-fonts-jp installed |
計画地 |
0 | 3 of 3 (<path id="g0" d=""/>) |
pylopdf 0.16.0 with the jp extra |
計画地 |
1,437 | 0 |
| pylopdf 0.16.0 without it | 計画地 |
0 | 3 of 3 |
The result is the same for a CIDFontType0 and a CIDFontType2 descendant (Adobe-Japan1), and for the UniJIS-UCS2-H and 90ms-RKSJ-H CMaps. Text extraction already works; only the glyphs are missing.
No crate sets hayro's font_resolver: crates/pyropdf-engine/src/render/mod.rs and render/svg.rs build InterpreterSettings from its default, so a FontQuery::Fallback for a CJK collection reaches hayro's default resolver, which has no CJK face. pylopdf wraps that resolver and answers CJK queries with the faces of the installed font packages.
This came up while moving a downstream application that reads Japanese architectural plans from pylopdf 0.16.0. On 62 pages whose fonts are all embedded, its text, words, path segments, and renders through pyropdf match pylopdf (crops are one pixel wider, from outward rounding). A plan whose Japanese fonts are not embedded would lose its text in the page images that the application renders.
Goal
Text in a non-embedded CJK font renders with a fallback face in pixmap(), svg(), and every other hayro rendering, as in frozen pylopdf, from the pylopdf-fonts-* packages that are installed.
Scope
In:
- A resolver for hayro's fallback queries. It picks the language from the character collection (Adobe-Japan1, GB1, CNS1, Korea1), or from the BaseFont name for Adobe-Identity, and the serif or sans face from the name.
- Discovery of the installed
pylopdf_fonts_*packages, an extra for each (such asjp, asocrinstallspylopdf-ocr-models), and a way to set or clear a fallback face per language. - A fixture of the page above, with both descendant types and both CMaps.
Out:
- Text extraction, which already maps these CMaps.
- Fallback faces for scripts other than CJK, which pylopdf's renderer does not pick either.
Paths
crates/pyropdf-engine/src/render/, the engine's font modules, crates/pyropdf-python/, python/pyropdf/_fonts.py, python/pyropdf/_document.py, pyproject.toml, and their tests.
Acceptance
- With
pylopdf-fonts-jpinstalled, the page above draws its three glyphs inpixmap()andsvg(), and a Mincho BaseFont picks the serif face - GB1, CNS1, and Korea1 pages pick their language's face when its package is installed
- Without an installed face, rendering is as now, and the documentation names the extras
- The differential tests against frozen pylopdf (DESIGN.md section 12) include such a page
Notes
pylopdf code to port (v0.16.0, 4ff4a266):
pick_cjk_fallback, the name hints,set_fallback_font, andset_fallback_faceinrust/src/document/fonts.rsinterpreter_settingsinrust/src/document/rendering.rs_FONT_PROVIDER_SPECSand_bundled_font_providersinsrc/pylopdf/_fonts.py_load_fallback_providersinsrc/pylopdf/_document.py
hayro 0.9 moves fonts into hayro-font (#584), so the resolver may change with the adoption (#18).
Reproduction
python repro.py pyropdf or python repro.py pylopdf prints one line per case. It needs numpy.
import sys
import numpy as np
library = __import__(sys.argv[1] if len(sys.argv) > 1 else "pyropdf")
def page_pdf(encoding: str, code: str, subtype: str) -> bytes:
"""One page that shows 計画地 in MS-Gothic, which the file does not embed."""
stream = f"BT /F1 40 Tf 20 30 Td <{code}> Tj ET".encode()
objects = [
b"<< /Type /Catalog /Pages 2 0 R >>",
b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>",
b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 200 100]"
b" /Resources << /Font << /F1 5 0 R >> >> /Contents 4 0 R >>",
b"<< /Length %d >>\nstream\n" % len(stream) + stream + b"\nendstream",
f"<< /Type /Font /Subtype /Type0 /BaseFont /MS-Gothic /Encoding /{encoding}"
" /DescendantFonts [6 0 R] >>".encode(),
f"<< /Type /Font /Subtype /{subtype} /BaseFont /MS-Gothic"
" /CIDSystemInfo << /Registry (Adobe) /Ordering (Japan1) /Supplement 2 >>"
" /FontDescriptor 7 0 R /DW 1000 >>".encode(),
b"<< /Type /FontDescriptor /FontName /MS-Gothic /Flags 4 /FontBBox [0 -141 1000 859]"
b" /ItalicAngle 0 /Ascent 859 /Descent -141 /CapHeight 709 /StemV 80 >>",
]
out, offsets = bytearray(b"%PDF-1.7\n"), []
for number, body in enumerate(objects, 1):
offsets.append(len(out))
out += b"%d 0 obj\n" % number + body + b"\nendobj\n"
xref = len(out)
out += b"xref\n0 %d\n0000000000 65535 f \n" % (len(objects) + 1)
out += b"".join(b"%010d 00000 n \n" % offset for offset in offsets)
out += b"trailer\n<< /Size %d /Root 1 0 R >>\nstartxref\n%d\n%%%%EOF\n" % (len(objects) + 1, xref)
return bytes(out)
for encoding, code, subtype in [
("UniJIS-UCS2-H", "8A08753B5730", "CIDFontType0"),
("UniJIS-UCS2-H", "8A08753B5730", "CIDFontType2"),
("90ms-RKSJ-H", "8C7689E6926E", "CIDFontType2"),
]:
with library.open(page_pdf(encoding, code, subtype)) as doc:
page = doc.pages[0]
dark = int((np.asarray(page.pixmap(dpi=72)).min(axis=2) < 128).sum())
empty = page.svg().count(' d=""')
print(f"{encoding:13} {subtype:12} text={page.text().strip()!r} dark_pixels={dark} empty_svg_glyphs={empty}")
Activity
yhay81 commentedon Nov 3, 2026
Claimed. Branch:
claude/600-cjk-fallback-faces.yhay81 commentedon Nov 3, 2026
Handoff: PR #613, branch
claude/600-cjk-fallback-faces. Ready for the owner's review; it has no merge authorization.What changed. A resolver for hayro's fallback font queries, ported from pylopdf
4ff4a266(the same files at the freezeebb01d3c), draws a CJK font that a document does not embed in the faces ofdoc.fallback_fonts: the language from the character collection, or from the name for Adobe-Identity, and the serif face for a Mincho-like name. Until edited, the mapping holds the installedpylopdf-fonts-*packages (the newjp,ko,zh-cn, andzh-twextras), each file read the first time that a page needs it.Page.pixmap,Document.render,Page.svg, andAnnotation.pixmaprender with them; text extraction does not.Checks.
python tools/check.py --lockedpassed on Windows (pytest: 6,563 passed, 689 skipped); CI on #613 is running.Evidence.
evidence/phase1-parityran again with frozen pylopdf built with its CJK extras: the 7 pages oftests/assets/text/nonembedded-cjk.pdf(the page above in both descendant types and both CMaps, MS-Mincho, GB1, CNS1, Korea1) render byte for byte as pylopdf renders them; the page above draws its 1,437 dark pixels. The rerun also records what changed since the run of 2026-10-28 apart from this issue: #460 keeps the headings ofpatent-us223898page 2 whole, #172 made the highlight render alike, andpdfium-ocg-invisible.pdfis measured now.Next action. The owner's review of #613. #609 (hayro 0.9) and #613 conflict only in the
InterpreterSettingsliterals ofcrates/pyropdf-engine/src/render/mod.rsandrender/svg.rs; whichever merges second keeps bothrender_contentand..fallback.settings().For the owner to decide.
words(exact_bboxes=True)differs by up to 7.28 points on those pages (fallback-exact-boxesin the parity report). Parity there would be its own issue: extraction would then depend on the fallback fonts, which DESIGN.md section 5 makes part of its memo key.Proposed DESIGN.md wording (not changed in the PR)
This issue makes the installed font extras the default faces of
doc.fallback_fonts, so rendering output now also depends on whichpylopdf-fonts-*packages are installed, asOcrEngine()already depends onpylopdf-ocr-models. The PR keeps the faces a setting of eachDocumentobject (not saved, likedoc.form.font), adds no mutable registry, and reads an installed face only when a page needs it. Three sentences of DESIGN.md do not say so yet; if you agree, a separate DESIGN.md pull request would read:ocrextra installs (section 14), as do the CJK faces that render the fonts that a document does not embed, which thejp,ko,zh-cn, andzh-twextras install; canvas text draws in a Standard 14 font or in a face that the program supplies (section 7)."ocrextra installspylopdf-ocr-models, and thejp,ko,zh-cn, andzh-twextras installpylopdf-fonts-jp,-ko,-zh-cn, and-zh-tw."yhay81 commentedon Nov 4, 2026
Owner's decisions on the two choices of the handoff above (2026-11-04, 「推奨どおりで」, as recommended):
Neither is approval of PR #613 itself: it still waits for the owner's review and has no merge authorization.
Next action. Merge
maininto PR #613 once more after #612's PR #623 lands, since the two share 7 files. The owner chose on #622 to turn hayro's glyph cache off for 0.2.2, and that work is onclaude/622-glyph-cache-off; whichever of it and PR #613 lands second measuresevidence/phase1-parityagain.yhay81 commentedon Nov 4, 2026
Handoff update: PR #613 (branch
claude/600-cjk-fallback-faces) is merged withmainat614fb83and pushed as0bab51a. Still ready for the owner's review, with no merge authorization.What changed. The conflicts with #609 (hayro 0.9) and #620 are resolved:
InterpreterSettingsinrender/mod.rsandrender/svg.rstakesrender_contentand..fallback.settings(), anduv.lockholds both sides. The resolver itself needed no change on hayro 0.9.evidence/edit-render/splitis back as it was measured.Checks.
python tools/check.py --lockedpassed on Windows at0bab51a(pytest: 6,613 passed, 691 skipped),cargo check --manifest-path fuzz/Cargo.toml --binspassed, and so didtests/test_parity.pywith its redump against frozen pylopdf. CI on the new head is running.Evidence.
evidence/phase1-parityis measured again on hayro 0.9. The fallback pages' reading outputs are as before. Their renders, and those of the corpus's other text pages, no longer match frozen pylopdf byte for byte, and not because of this PR: hayro 0.9's glyph cache, which #609 leaves on, changes 152 of the 163 rasters. With the cache off, 162 match, all 7 fallback pages among them.Next action. The owner's review of #613, and the choice on #622, whether 0.2.2 keeps the cache. Whichever of #613 and #622's change lands second measures the evidence again. The two questions in the handoff above, the DESIGN.md wording and the exact word boxes, are still open.
DESIGN.md: output depends on the installed CJK font extras (#625) (#626)
Render CJK fonts that a document does not embed with fallback faces (#…
Pass a Fallback in the render tests of hinting (#634) (#635)