Conversation
There was a problem hiding this comment.
Pull request overview
This pull request qualifies the Principia 34 corpus using preservation-aware, book-specific processing rules and adds deterministic, metadata-only audit/qualification artifacts plus regression tests to keep the qualification stable and reproducible.
Changes:
- Replaces broad cleanup/dehyphenation with evidence-backed, attestation-driven dehyphenation and book-local page-furniture removal logic across many Principia handlers.
- Adds qualification/audit test suites (including optional external Parquet integration checks) and focused regression fixtures for previously fragile books.
- Documents the qualification policy and links it from the README, keeping committed artifacts metadata-only (hashes/counts/tiers) rather than source text.
Reviewed changes
Copilot reviewed 48 out of 49 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_transforms.py | Adds/updates unit tests covering conservative dehyphenation behaviors and evidence-driven variants. |
| tests/test_principia_validation_pair.py | Adds focused regressions for Psychology and Federalist validation edge cases. |
| tests/test_principia_qualification.py | Adds deterministic contract tests for the frozen qualification and audit JSON artifacts (plus optional external Parquet checks). |
| tests/test_principia_batch_b.py | Adds regression coverage for “batch B” Principia book handlers (page furniture, scan garbage, boundary repairs). |
| tests/test_principia_batch_a.py | Adds regression coverage for “batch A” handlers and confirms broad OCR heuristics are not used. |
| tests/test_audit_principia_quality.py | Adds unit/integration coverage for the quality audit builder and CLI behaviors (no text leakage). |
| README.md | Links Principia 34 inventory to the new qualification documentation. |
| grokken/transforms/whitespace.py | Introduces evidence-backed dehyphenation utilities and tightens dehyphenation to avoid crossing blank lines. |
| grokken/books/principia/vicarious_sacrifice_bushnell.py | Replaces broad OCR cleanup with book-specific page-furniture removal and attested dehyphenation. |
| grokken/books/principia/stowe_writings.py | Adds book-local header detection/removal and confirmed boundary repairs; switches to attested dehyphenation. |
| grokken/books/principia/sociology_spencer.py | Implements page-furniture heuristics and targeted repairs; switches to attested dehyphenation. |
| grokken/books/principia/sociology_giddings.py | Adds page-furniture removal and targeted corrections; switches to attested dehyphenation. |
| grokken/books/principia/psychology_james.py | Adds preservation lists + evidence-aware dehyphenation, plus scan-noise/header handling and verified repairs. |
| grokken/books/principia/political_science_leacock.py | Adds running-head removal and targeted repairs; switches to attested dehyphenation. |
| grokken/books/principia/political_economy_mill_1884.py | Implements exact page-furniture removal and targeted repairs; switches to attested dehyphenation. |
| grokken/books/principia/political_economy_mill_1870.py | Implements exact page-furniture removal and targeted repairs; switches to attested dehyphenation. |
| grokken/books/principia/political_economy_bowen.py | Adds header removal and targeted joins; switches to attested dehyphenation. |
| grokken/books/principia/poetical_works_whittier.py | Tightens header removal for poetry and applies only a few verified wrap joins (avoids aggressive changes). |
| grokken/books/principia/pathologic_histology_mallory.py | Adds extensive verified wrap repairs and tighter running-head removal; switches to attested dehyphenation. |
| grokken/books/principia/natural_philosophy_comstock.py | Adds exact running-head removal and verified corrections; switches to attested dehyphenation. |
| grokken/books/principia/middle_ages_emerton.py | Adds exact running-head removal and verified OCR corrections; switches to attested dehyphenation. |
| grokken/books/principia/logical_theory_dewey.py | Adds exact running-head removal and one verified word-boundary repair; switches to attested dehyphenation. |
| grokken/books/principia/international_law_woolsey.py | Adds exact running-head removal and targeted repairs (e.g., §); switches to attested dehyphenation. |
| grokken/books/principia/international_law_davis_1903.py | Adds exact running-head removal and targeted repairs; switches to attested dehyphenation. |
| grokken/books/principia/international_law_davis_1900.py | Adds page-furniture removal, targeted joins, and trims library plate after terminator; switches to attested dehyphenation. |
| grokken/books/principia/history_religions_toy.py | Adds page-furniture removal, targeted joins, and due-slip trimming; switches to attested dehyphenation. |
| grokken/books/principia/hermeneutical_manual_fairbairn.py | Adds page-furniture removal, Greek-specific dehyphenation, and verified joins; switches to attested dehyphenation. |
| grokken/books/principia/federalist.py | Adds preservation-aware dehyphenation, exact seam repairs, and quote fixed-point reflow logic. |
| grokken/books/principia/evolution_conn.py | Adds page-furniture removal and due-slip trimming; switches to attested dehyphenation. |
| grokken/books/principia/ethics_jouffroy.py | Adds page-furniture removal, wrapper trimming, and explicit end trimming; switches to attested dehyphenation. |
| grokken/books/principia/english_literature_shaw.py | Adds page-furniture removal, targeted confusable fixes, and due-slip trimming; switches to attested dehyphenation. |
| grokken/books/principia/doctrines_friends_bates.py | Adds page-furniture removal and explicit end trimming; switches to attested dehyphenation. |
| grokken/books/principia/dickens_works.py | Adds page-furniture removal, quote-glyph fixes, and due-slip trimming; switches to attested dehyphenation. |
| grokken/books/principia/cornerstone_abbott.py | Adds front-matter normalization and monotone-folio-based header stripping; switches to attested dehyphenation. |
| grokken/books/principia/church_building_cram.py | Adds front-marker and tail trimming plus monotone-folio header stripping; switches to attested dehyphenation. |
| grokken/books/principia/chaucer_student.py | Adds monotone-folio header stripping and terminator trimming; keeps poetry-safe behavior. |
| grokken/books/principia/channing_works.py | Adds monotone-folio header stripping, verified join(s), and preserves verified lexical hyphenation. |
| grokken/books/principia/bible_literature_wood.py | Adds front-marker and due-slip trimming plus monotone-folio header stripping; switches to attested dehyphenation. |
| grokken/books/principia/atonement.py | Adds front/end trimming plus monotone-folio header stripping; switches to attested dehyphenation. |
| grokken/books/principia/animal_histology_dahlgren.py | Adds wrapper trimming, ad trimming, header stripping, and evidence-backed page-gap dehyphenation. |
| grokken/books/principia/algebra_day.py | Adds wrapper trimming, header stripping, and one verified page-gap join; switches to attested dehyphenation. |
| docs/principia_quality.md | Documents the frozen artifacts, qualification policy, and reproduction workflow for Principia 34. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+18
to
+30
| # Conservative, evidence-backed dehyphenation for quality-sensitive corpora. | ||
| _RE_DEHYPHENATE_ATTESTED = re.compile( | ||
| r"(?P<left>\p{L}[\p{L}\p{M}]*)-\n[ \t]*(?P<right>[a-z][\p{L}\p{M}]*)" | ||
| ) | ||
| _RE_DEHYPHENATE_ATTESTED_PAGE_GAP = re.compile( | ||
| r"(?P<left>\p{L}[\p{L}\p{M}]*)-\n(?:[ \t]*\n)?[ \t]*" | ||
| r"(?P<right>\p{Ll}[\p{L}\p{M}]*)" | ||
| ) | ||
| _RE_WORD = re.compile(r"(?<![\p{L}\p{M}\p{N}_])[\p{L}\p{M}]+(?![\p{L}\p{M}\p{N}_])") | ||
| _RE_INLINE_HYPHENATED_WORD = re.compile( | ||
| r"(?<![\p{L}\p{M}\p{N}_])(?P<left>\p{L}[\p{L}\p{M}]*)-" | ||
| r"(?P<right>[a-z][\p{L}\p{M}]*)(?![\p{L}\p{M}\p{N}_])" | ||
| ) |
Comment on lines
239
to
+248
| # === REMOVE ROMAN NUMERAL PAGE NUMBERS === | ||
| # The introduction uses roman numerals (ii, iii, iv, etc.). | ||
| # Require 2+ chars to avoid matching the pronoun "I" or single letters. | ||
| # Case-consistent to avoid matching words like "ill" or "Civil". | ||
| text = re.sub(r"^\s*(?:[ivxlc]{2,6}|[IVXLC]{2,6})\s*$", "", text, flags=re.MULTILINE) | ||
| text = re.sub( | ||
| r"^\s*(?:[ivxlc]{2,12}|[IVXLC]{2,6})\s*$", | ||
| "", | ||
| text, | ||
| flags=re.MULTILINE, | ||
| ) |
eous
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Corpus gate
9c9fb8bdacc4438ce86103c4a6ac202fca23bd787c2cf990d0cf2e69845e893421501257be8e25a8c9db18a04155b7228f577c643729b776d0e96a7e7b9a68de9b53fa5a1447b4f2f0f8f4c3864563358e2d2aa617c4ad860320fc74d661a4c6Cushing is a clean native-context validation source, but its length does not validate 262K or 1M behavior. Raw and processed book Parquets are deliberately excluded from this PR. The committed JSON contains hashes, counts, classifications, and evidence metadata only, with no source text or excerpts.
Validation
git diff --check: passed.9b53fa5a...a4c6exactly and is location-independent.The repository-wide format check still identifies three pre-existing formatting defects in unchanged files (
README.md,grokken/generation/books/principia/psychology_james.py, andtests/test_base.py); this PR does not broaden scope to reformat them.