Repository navigation
Rendering speed: close MuPDF's lead on the corpus #365
Description
Context
On the corpus, pyropdf renders 2.65 times slower than MuPDF at 72 dpi and 3.97 times at 144 dpi, and up to 8 times on bill-hr815 (#354). pylopdf milestone Rendering (#248) names glyph and image caches.
Goal
Rendering every corpus page is within 1.5 times MuPDF at 72 and 144 dpi, measured by evidence/pymupdf-comparison/run.py.
Scope
In: profiling per document, upstream changes in hayro and vello_cpu (glyph and image caches, clip shortcuts), and pyropdf-side caching. Out: GPU rendering.
Paths
evidence/pymupdf-comparison/crates/pyropdf-python/src/(pyropdf-side caching only)
Acceptance
- A profile of bill-hr815 and usrguide names the hot paths, committed under
evidence/. - Each upstream proposal is drafted here first; publishing it waits for the owner.
- The comparison reaches 1.5 times or better.
Notes
Upstream issues and PRs publish on the owner's behalf: ask first (AGENTS.md).
Activity
yhay81 commentedon Oct 29, 2026
Claimed. Branch:
claude/365-render-profile(profiling bill-hr815 and usrguide at 72 and 144 dpi, first acceptance criterion).yhay81 commentedon Oct 30, 2026
Profile and improvement plan
PR #390 (branch
claude/365-render-profile) commitsevidence/pymupdf-comparison/profile.pyandresults/profile-darwin-arm64.md. It covers the first acceptance box. It says "Part of #365", not "Closes", because the speed-up itself is still to do.Hot paths (shares of samples, main thread)
hayro::Renderer::fill_outline_run→RenderContext::fill_pathper glyph (vello flatten,make_tiles_analytic_aa, tile sort,strip::render)Dispatcher::rasterize)Pixmap::take_rgba8(ImageAlphaType::Alpha)(un-premultiply), inlined intopyropdf_engine::render::drawatrender/mod.rs:361hayro-syntax, fonts, and image decoding are each under 4%. pyropdf-side caching of documents or objects has almost nothing to win.
Measurement fix (in this issue's paths)
bench.pycompares unlike work. pyropdf runsto_bytes(), which encodes a PNG. PyMuPDF readssamples, which is raw. The PNG adds 40–90% to pyropdf. Withbytes(pixmap)on our side (the buffer protocol already exists in 0.1.0), the ratio on these two documents is 4.5–5.6, not 6.4–10.6. Proposal: switchbench.pytobytes(page.pixmap(dpi=...))and rerunrun.py. This changes the metric that the Goal names, so it needs an owner OK.pyropdf side
drawclears the target tooptions.background. When that alpha is 255, source-over keeps every pixel opaque, so premultiplied and straight RGBA are equal.take_rgba8(ImageAlphaType::AlphaPremultiplied)then moves the buffer without touching it. This edit is incrates/pyropdf-engine/src/render/mod.rs, outside this issue's paths, so it should be a follow-up issue (small, no upstream dependency).Resourcesper thread next toRenderCacheinrender::with_cache.drawpasses&mut Resources::default()on every page. This only helps once hayro draws through vello's glyph atlas (5).crates/pyropdf-python/src/: none worth doing. The store, the bridge, and opening are ≤1.2%, and theRenderCacheper thread already shares fonts and outlines across pages.Upstream drafts (not published; owner approval needed)
fill_outline_runsets the transform and callsfill_path(cached_outline(..))for every glyph, so vello re-flattens, re-tiles, re-sorts, and re-strips identical glyphs. Draft for LaurenzV/hayro:GlyphRunBuilder/ the atlas accept caller-supplied outlines with a caller key, so that PDF fonts (Type 1, CFF, Type 3 outlines) can use the atlas.Pixmap::take_rgba8(Alpha)runsunpremultiply_rgba8whenevermay_have_transparencyis set. Rendering sets it even when the target was cleared to an opaque color. Draft: keepmay_have_transparency = falseafterTargetInit::Clearwith an opaque color and source-over-only content, or bail out of the SIMD loop's division when every alpha in a chunk is 255. Item 2 removes the need for pyropdf, so this is low priority.vello_common::clip::intersectis 1–2% here. Not worth a proposal from these documents.Next action
Read pyropdf's samples in the render benchmark (#365)
yhay81 commentedon Oct 30, 2026
Benchmark fixed in #390 with the owner's approval (item 1 of the plan above):
bench.pynow readsbytes(page.pixmap(dpi=...))like PyMuPDF'ssamples, andrun.pywas rerun (pyropdf 0.1.0 against PyMuPDF 2.0, 15 documents). Corpus totals: Render 72 dpi 2.65 → 2.03, Render 144 dpi 3.97 → 2.34. Worst documents: bill-hr815 4.34 / 4.62, usrguide 2.89 / 3.67. Next actions are unchanged: a follow-up issue for skipping the un-premultiply (item 2), and the owner's call on publishing upstream drafts 5–7.Attribute the post-raster time to to_layout, and measure a fix (#365)
yhay81 commentedon Oct 30, 2026
Correction to item 2 of the plan, after review on #390 (commit bc02138).
The 7–20% that I attributed to
take_rgba8's un-premultiply isto_layout:render/mod.rs:361is theto_layoutcall, and line numbers in optimized code could not separate the two.evidence/pymupdf-comparison/patches/split-draw.patchkeeps both out of line. With it,to_layout(scalar RGBA→RGB) takes 7.7 / 20.1 / 7.7 / 17.9% (bill 72, bill 144, usrguide 72, usrguide 144), andtake_rgba8(Alpha)takes no samples, since vello_cpu skips the division when the opaque clear leavesmay_have_transparencyunset.Revised item 2 (pyropdf side): for RGB output without alpha, use vello_cpu's SIMD
Pixmap::try_take_rgb8instead ofto_layout, and keepto_layoutas the fallback (patches/rgb8-draw.patch; render tests pass with it). Measured with the samples read (results/profile-darwin-arm64-rgb8.md): bill-hr815 408.7 → 375.9 ms at 72 dpi (−8%) and 766.1 → 638.3 ms at 144 dpi (−17%); usrguide 77.1 → 71.0 ms (−8%) and 135.4 → 112.4 ms (−17%). The step drops to 1.3–3.6%. The change is incrates/pyropdf-engine, outside this issue's paths, so it remains a follow-up issue.Item 7 withdrawn: the un-premultiply already does not run here, so there is nothing to propose upstream.
yhay81 commentedon Oct 30, 2026
Item 2 of the plan (
to_layout→try_take_rgb8) is now #393, a sub-issue of #355 in Phase 1. Remaining here: the owner's call on publishing upstream drafts 5 and 6 (glyph coverage cache in hayro, an outline entry point to vello_cpu's glyph atlas), and the review of #390.Profile rendering of bill-hr815 and usrguide (#365) (#390)
yhay81 commentedon Oct 30, 2026
Draft 6 is published upstream with the owner's approval: linebender/vello#2041 proposes letting caller-supplied outlines use vello_cpu's glyph atlas. Draft 5 stays unposted because hayro's maintainer is doing glyph caching (LaurenzV/hayro#1613). Draft 7 is unneeded since #393. Next: answer the vello maintainers.
21 remaining items
yhay81 commentedon Nov 5, 2026
Handoff: pushed
a8d2f7d9613752688b6b8bb1bd866455243a68afoncodex/365-render-remeasurement, draft #709. Copy/initialization evidence.Bounded native stacks now attribute 122 of 235 memmove samples to
Pixmap::newat render/mod.rs:440, 96 to Python buffer-copy stacks, and 17 to other paths. Every copy sample is accounted for; leaf symbols resolve and context reads succeed. Python parent export labels are approximate; the Rust source line uses the matching PDB. The isolated constructor probe shows why allocation-only timings overstate gains: touching every byte leaves ratios 1.006/0.919/0.762 for 612x792, 1224x1584, 2480x3508. No product implementation changed.Created #710 for the out-of-scope engine path, as a child of #355 in Phase 1, and linked it as a blocker of #365. It owns the existing-API zeroed-storage implementation, exact-output/limit coverage and full-render comparison. #365 retains its <=1.5 MuPDF target; #709 stays draft and must not merge on this constructor evidence alone.
Validation: all eight locked checks pass (7,398 Python tests passed, 705 skipped); native helper /W4 /WX; isolated probe fmt/clippy/audit and actionlint pass. Working tree clean. All local sessions from this turn are terminal; no build/benchmark is running. Next: claim #710 in a fresh work session/branch from main, preserve #709's evidence, and implement/measure its scoped change. Reuse the ordinary baseline wheel target/365-wheels/pyropdf-0.2.2-cp311-abi3-win_amd64.whl (SHA d9232afcac19d40db410853da4cd9af5979a26fc9064ac8337e522b64b632c63); the PDB-enabled wheel is for profiling only.
Releasing the #365 implementation claim while following #710. Initial discarded profiling traces remain under ignored target/365-profile-windows/: optional cleanup was rejected by automatic command policy (blocked by policy), so no deletion was performed. They are not evidence; use committed JSON/report files or leaf-samples-debug.json. No need to retry cleanup to proceed.
yhay81 commentedon Nov 6, 2026
The #710 experiment closed in #711 (0194497). Both measurement orders show a 144 dpi regression, so the renderer retains its previous allocation. Full results: https://github.com/yhay81/pyropdf/blob/019449775f80ca8e08bf78eeb46f79bc46851412/evidence/pixmap-initialization/README.md . Draft #709 remains open with the original acceptance target. Next actionable investigation: measure reuse of repeated glyphs at exact device positions (including subpixel phase) on bill-hr815 and usrguide before proposing a bounded cache that preserves pixels. Keep raw pixel extraction and compare against MuPDF after any successful candidate.
yhay81 commentedon Nov 6, 2026
Resuming codex/365-render-remeasurement (#709) to measure exact two-axis subpixel reuse and assess an output-preserving, bounded renderer optimization after #710.
Measure exact glyph reuse and evaluate prepared outlines (#365)
Test bounded batching of disjoint glyph fills (#365)
yhay81 commentedon Nov 6, 2026
On codex/365-render-remeasurement, draft #709 now includes two independent local candidates at 925da35. The prepared-outline wheel matched all 1,872 pixel cases but reached only 1.84/1.53 times MuPDF in the official runner. A separate ordinary baseline run was 1.94/1.70, and paired artifact timings showed little consistent benefit. The disjoint-path wheel changed 164 hashes and regressed timing, so neither candidate should ship. Source patches, repeated measurements, all mismatches and reproduction steps are in the PR. All eight locked local checks passed, the native phase probe reproduced all rows, and the second probe verified its overlapping columns against those results. The working tree is clean. Main CI passed at 98fd892. Next: test the existing vello_cpu multi-worker dispatcher with an explicit small count in a disposable wheel, compare exact pixels, end-to-end timings and peak resources. Do not enable the default of up to eight threads or merge #709 on profiling alone.
Measure bounded raster workers, context reuse and lifetime (#365)
yhay81 commentedon Nov 6, 2026
Handoff on codex/365-render-remeasurement, draft #709, pushed 66c3578. Worker, memory, lifetime and per-document evidence: https://github.com/yhay81/pyropdf/blob/66c3578a76dedca1c99c222fe754737a8ad60e10/evidence/pymupdf-comparison/results/reused-workers/README.md . All eight locked checks passed (7,398 Python tests, 705 skips). New scripts passed Ruff and ran against the identified release artifacts. All local builds and measurements finished. Four reused workers preserved corpus output and met the aggregate target, but partial reads retain native workers after close and some documents exceed the target. #715 owns the production implementation and cross-platform/resource gates. Its parent, milestone and blocking link are verified. Next: implement safe resource ownership in #715, then remeasure the shipping wheel before reassessing #365. Releasing this claim while #365 waits on #715.
yhay81 commentedon Nov 6, 2026
Following the owner's request to advance the two open PRs, the completed research in #709 is now scoped to evidence-only issue #723. It will preserve benchmark tooling, profiles and all experiment outcomes without applying product patches or claiming the performance target. #709 will close #723, not this issue. The original goal and acceptance remain unchanged: implement #715, then validate the actual shipping wheel at both resolutions. No performance gate or upstream-release requirement has been waived.
Preserve rendering profiles and experiment measurements (#709)
yhay81 commentedon Nov 6, 2026
Research PR #709 has now merged as 97ca201, closing evidence-only #723. The profiles, benchmark scripts, rejected experiments and worker measurements are available on main. This performance issue remains open; no runtime optimization was applied. Next: complete #715 after its upstream dependencies, then measure the actual shipped wheel against the unchanged 72/144 dpi target.