feat(llm): schema-aware enhanced strategy for /graph/extract (#74)#369
Closed
YanhaoLv wants to merge 20 commits into
Closed
feat(llm): schema-aware enhanced strategy for /graph/extract (#74)#369YanhaoLv wants to merge 20 commits into
YanhaoLv wants to merge 20 commits into
Conversation
…ine)
Introduce a request-level extract_strategy ("baseline" | "enhanced") flag
and an include_debug flag, threaded through GraphExtractRequest → flow
prepare → WkFlowInput → ExtractNode → PropertyGraphExtract. Baseline is
byte-compatible: the flag is only surfaced in the shared context, and the
existing chunk-level parse/normalize path runs unchanged.
Subsequent commits introduce the schema-aware graph quality layer behind
the enhanced flag (parser → normalizer → document-level assembler →
quality gate → structured warnings). For this commit only the plumbing
lands; enhanced still falls through to baseline logic and emits an INFO
log noting the layer is not yet wired.
- Pydantic Literal validation returns 422 on unknown extract_strategy.
- Unknown values passed via internal constructors are normalized to
baseline (defence in depth against non-API entry points).
- WkFlowInput.reset() and WkFlowState.setup() now clear the new fields to
prevent state leakage across pooled pipeline reuse.
- Tests cover: request defaults, explicit baseline/enhanced acceptance,
unknown-strategy rejection, scheduler kwargs pass-through, flow prepare
capture/defaults, WkFlowInput reset, PropertyGraphExtract constructor
captures / normalizes strategy, and run() surfaces resolved strategy
in the returned context.
Add a runtime schema index for the enhanced graph extraction strategy.
GraphSchemaIndex compiles a validated HugeGraph schema (dict or JSON
string) into deterministic lookups used by the schema-aware quality
layer: vertex/edge label membership, allowed property sets, property
data-type and cardinality queries, canonical vertex id generation, safe
property value coercion, and edge endpoint compatibility.
The canonical id rule ({vertex_label.id}:{pk1}!{pk2}) intentionally
mirrors PropertyGraphExtract._primary_key_id — the enhanced strategy
inherits baseline behavior including the None fallback for schemas
without id_strategy=PRIMARY_KEY or a populated vertex_label.id. This
keeps the write-to-graph and baseline evaluator paths byte-compatible.
Coercion rules follow the design contract:
- TEXT/UUID/BLOB: string passthrough (str() for non-strings).
- INT/LONG/BYTE: strict integer only; rejects booleans and lossy floats.
- FLOAT/DOUBLE: numeric or parseable string; rejects booleans and
non-numeric strings.
- BOOLEAN: accepts bool, "true"/"false"/"yes"/"no"/"1"/"0", or 0/1 int.
- DATE: only YYYY-MM-DD; no fuzzy parsing.
- LIST/SET: element-wise coercion with partial-drop warning; SET
additionally deduplicates.
The module is pure additive — no existing file is touched. It performs
no HugeGraph I/O and does not resolve named-graph schema strings.
Callers must hand in the concrete schema (SchemaNode already does).
50 unit tests cover: constructor validation (dict/JSON/rejects), label
lookups, property allowlists and type/cardinality queries, single and
multi primary-key canonical ids, missing/empty PK values, CUSTOMIZE
id_strategy, schemas without vertex_label.id, edge endpoint spec and
direction check, each data-type coercion path (success and failure),
and LIST/SET cardinality including partial-drop and deduplication.
Introduce the structured warning surface for the enhanced graph extraction
strategy. Every material deviation from the schema-clean happy path (a
dropped item, a coerced property value, a merged duplicate, an
unresolvable endpoint) becomes a StructuredWarning that will flow into
meta.structured_warnings, the effect report, and the debug payload.
WarningCode is a (str, Enum) subclass so codes JSON-serialize as bare
strings without a custom encoder. Codes are grouped by producer:
- Candidate parser: JSON_NOT_FOUND, JSON_DECODE_FAILED,
GRAPH_SECTION_MISSING, ITEM_NOT_OBJECT, ITEM_TYPE_MISMATCH
- Normalizer (vertex): VERTEX_LABEL_NOT_IN_SCHEMA,
VERTEX_PRIMARY_KEY_MISSING, VERTEX_PRIMARY_KEY_INVALID,
VERTEX_ALIAS_RECORDED
- Normalizer (edge): EDGE_LABEL_NOT_IN_SCHEMA, EDGE_ENDPOINT_MISMATCH
- Property: PROPERTY_NOT_IN_SCHEMA, PROPERTY_COERCED,
PROPERTY_COERCION_FAILED
- Document assembler: ENDPOINT_PENDING_REPAIR, ENDPOINT_UNRESOLVED,
ENDPOINT_AMBIGUOUS, DUPLICATE_VERTEX_MERGED, DUPLICATE_EDGE_MERGED,
PROPERTY_CONFLICT
StructuredWarning is a frozen dataclass, hashable, defensively copies
its context mapping, and rejects invalid item_type or negative chunk_id
at construction. is_surface_affecting distinguishes codes that
represent a real change to the emitted graph from purely observational
markers (alias recorded, pending repair, first-wins conflict), which
the effect report uses to explain why item counts changed.
warning_code_distribution() aggregates a warning list into a
{code: count} histogram, silently skipping foreign entries so a
malformed tail can never sink the effect report.
22 unit tests cover: enum surface, JSON encoding without custom
encoder, dataclass validation and immutability, defensive context
copy, to_dict serialization (full and None-omitting shapes),
is_surface_affecting classification, and the distribution helper.
Parse a single chunk's raw LLM output into a CandidateGraph plus a list
of StructuredWarning. Handles the five formats produced by different
prompt styles:
- grouped JSON object: {"vertices": [...], "edges": [...]}
- Markdown-fenced JSON (```json ... ``` or bare ```)
- JSON wrapped in surrounding prose
- flat array: [{"type": "vertex", ...}, {"type": "edge", ...}]
- explicit-type items inside grouped sections
JSON discovery is layered: json.loads on the stripped text first (fast
path when the LLM emits clean JSON), then greedy {...} and [...]
extraction for the wrapped-in-prose case. A failed attempt with any
opening brace or bracket present yields JSON_DECODE_FAILED so callers
can distinguish "LLM tried and produced garbage" from "LLM emitted
prose only" (JSON_NOT_FOUND).
Item collection drops non-dict candidates (ITEM_NOT_OBJECT), items in
flat-array position without a routable type (ITEM_TYPE_MISMATCH), and
items in grouped sections whose explicit type disagrees with the
containing array (ITEM_TYPE_MISMATCH). Missing vertices/edges sections
in a grouped payload each produce a GRAPH_SECTION_MISSING warning; the
missing side is normalized to an empty list so downstream stages can
still run.
Item type is normalized to the expected value on every kept candidate
so the normalizer never needs a defensive default. Original caller
dicts are not mutated (each kept item is a shallow copy).
Also introduces types.py holding the shared CandidateGraph dataclass
(frozen, default_factory-safe lists, is_empty helper) that the
normalizer and assembler will consume in the next commits.
27 unit tests cover: the dataclass shape, all five input formats (with
and without fences), prose surrounding JSON, empty and whitespace-only
inputs, missing/non-list sections, flat-array items without type,
non-dict items in both formats, truncated JSON, scalar payloads,
grouped explicit-type conflicts, chunk_id propagation, item type
normalization, and immutability of caller-side dicts.
Consume a CandidateGraph plus a GraphSchemaIndex and produce a NormalizedChunkGraph in which every vertex is schema-valid with canonical ids when possible, every edge has both endpoints resolved (or explicit pending hints for the document-level assembler), and every deviation is recorded as a StructuredWarning tagged with the originating chunk_id. Processing order (per design section 6.4): - Vertex: label ∈ schema? → filter properties by allowed_properties → coerce values against propertykeys[].data_type / .cardinality → primary-key completeness check → canonical_vertex_id. - Edge: label ∈ schema? → filter properties → resolve out/in labels (falling back to the schema's edge spec when the LLM omits them) → is_endpoint_compatible → resolve endpoints via (a) legacy source/ target dicts against the schema, (b) explicit outV/inV against the chunk's alias table. An edge that can be neither fully resolved at chunk level nor cleanly ruled out is emitted with _pending_out / _pending_in hints for the assembler, plus an ENDPOINT_PENDING_REPAIR warning. Edges with schema-invalid endpoint directions are dropped immediately with EDGE_ENDPOINT_MISMATCH. The vertex alias table is seeded with identity entries in addition to (original_id → canonical_id) so downstream edges can resolve endpoints regardless of whether the LLM referenced a vertex by its raw id or its canonical form. Property coercion emits the soft PROPERTY_COERCED warning only when the value's Python type actually changed — same-type same-value coerces stay silent to avoid drowning the report. Also adds NormalizedChunkGraph, PENDING_OUT_KEY, PENDING_IN_KEY to the shared types module. Frozen dataclass with default_factory lists and dict — mutation of the collections is allowed but rebinding the fields is not, matching CandidateGraph's contract. 29 unit tests cover: vertex label validity and drop, property filter, type coercion (soft warning on type change, non-PK failure drops property, PK failure drops vertex), primary-key completeness (missing or empty), LLM-original-id alias recording (including the no-op case where original matches canonical), edge label validity, edge property filter, endpoint direction check and schema-fill fallback, legacy source/target resolution, chunk-alias resolution, unresolvable endpoint marking, and an end-to-end parser+normalizer integration smoke that verifies the two work together on a realistic grouped payload.
Assemble every chunk's NormalizedChunkGraph into a final DocumentGraph: cross-chunk vertex merge, document-level endpoint repair, and edge deduplication. This is the last stage the enhanced strategy needs before the graph reaches the API envelope. Vertex merge (per design section 6.5): - Key: (label, id). First-appearance wins for merge target and conflicting property values. - Non-conflicting properties from later occurrences are folded in. - Property conflicts emit PROPERTY_CONFLICT (soft — the first value stays); every merge event emits DUPLICATE_VERTEX_MERGED so the quality gate can count exact duplicates. - Vertices without an id are kept verbatim (no way to identify a merge partner). - Output order preserves first appearance for a deterministic result. Endpoint repair: - Union every chunk's alias table into a document-level index. Conflicting mappings for the same (label, key) are tracked in an ambiguous set — resolving through such a key yields ENDPOINT_AMBIGUOUS rather than silently picking one candidate. - Edges the normalizer left pending (with _pending_out / _pending_in hints) are resolved via the merged alias index. Successful repairs bump endpoint_repair_count on the resulting DocumentGraph. - Unresolvable pending endpoints drop the edge with ENDPOINT_UNRESOLVED; ambiguous ones drop with ENDPOINT_AMBIGUOUS. Edge dedup: - Key: (label, outVLabel, outV, inVLabel, inV, properties_signature). Property signatures use json.dumps(sort_keys=True) so LIST/SET cardinality property values (unhashable when tupled) still dedupe. - Edges with the same endpoints but different property signatures are both kept — losing a distinct fact is worse than surfacing a near duplicate. - Emits DUPLICATE_EDGE_MERGED per merge event, preserves first- appearance order. Also introduces DocumentGraph in the shared types module. Beyond vertices and edges, the dataclass carries pre_merge_vertex_count, pre_merge_edge_count, and endpoint_repair_count so the quality gate (next commit) can compute duplicate reduction and repair-rate metrics without re-scanning inputs. 15 unit tests cover: vertex merge across chunks, property conflict first-wins, missing property completion from later chunks, distinct keys stay separate, output ordering, no-id vertex handling, cross-chunk pending-endpoint repair, unresolved and ambiguous drops, resolved-at-chunk-level pass-through, identical edge dedup, distinct- property-signature preservation, and LIST-property dedup safety. Two integration tests drive parser + normalizer + assembler end-to-end.
Aggregate the structured warnings emitted throughout the enhanced pipeline plus the counts tracked by DocumentGraph into a single QualityMetrics bundle. Powers the meta.quality_metrics field on the API response and feeds the baseline-vs-enhanced comparison report. Must metrics per design section 6.6: - schema_valid_vertex_ratio / schema_valid_edge_ratio — fraction of candidate items that survived normalization. - endpoint_resolution_rate — pending edges resolved by the assembler vs. pending edges surfaced by the normalizer. - duplicate_vertex_reduction / duplicate_edge_reduction — merge/dedup effectiveness across chunks. - property_valid_ratio — kept properties / (kept + invalidated). PROPERTY_ CONFLICT is *not* counted as invalid because the first value survives. - dropped_item_count — sum of vertex/edge/item/property drop codes. Excludes DUPLICATE_*_MERGED (consolidated, not dropped) and PROPERTY_COERCED (value changed, item survived). - coerced_property_count, endpoint_repair_count — flow-through counters. Should metrics that fell out of the same aggregation cheaply: - property_conflict_count. - warning_code_distribution (delegated to warnings.warning_code_ distribution helper). Zero-input safety is a design contract, not an accident: every ratio returns a plain float in [0, 1] regardless of empty inputs. Ratios where "no candidates, no problems" is a passing state default to 1.0; reduction ratios (where "nothing to reduce" reads as "no reduction achieved") default to 0.0. NaN cannot escape the gate. QualityMetrics is a frozen dataclass; to_dict rounds ratios to 4 decimals so the API response is compact and floating-point-stable. Non-StructuredWarning entries in the warning list are silently skipped so a malformed tail at the very end of a long pipeline cannot sink the report. 26 unit tests cover: zero-input safety (all ratios in [0,1], no NaN, correct defaults), schema-validity ratios with normalization drops, endpoint resolution (full/partial/ambiguous/zero-pending), duplicate reduction, property validity (kept/dropped/conflict-doesn't-count), counter sums (drop codes / duplicates-and-coercions excluded / coerced count / repair count from graph), warning code distribution, serialization (rounding, JSON round-trip, frozen enforcement), robustness against malformed warning entries, and a realistic multi-metric integration scenario.
Activate the schema-aware graph quality layer behind
extract_strategy="enhanced". Requests continue to default to baseline
and stay byte-compatible; only an explicit opt-in reaches the new
pipeline.
Enhanced path per chunk:
raw LLM output
→ CandidateGraphParser.parse
→ SchemaAwareNormalizer.normalize (with GraphSchemaIndex)
→ NormalizedChunkGraph
Then across all chunks:
→ DocumentGraphAssembler.assemble
→ GraphQualityGate.compute
The final DocumentGraph vertices and edges replace context["vertices"]
and context["edges"]. Warnings from every stage accumulate into
context["structured_warnings"]; quality metrics land in
context["quality_metrics"]; when include_debug=true a per-chunk
diagnostic block (raw output truncated at 2 KB, candidate/normalized
counts, per-chunk warning count) plus the warning-code distribution
land in context["debug_info"].
New prompt_contract module:
- build_prompt_contract(schema_index) returns a short structural
constraint block enumerating the schema's vertex and edge labels
plus the six rules from design section 6.1 (schema-only labels,
schema-only properties, PK-required, endpoint forms, omit-not-
fabricate, JSON-only output).
- Appended after example_prompt so caller-supplied framing keeps
its shape; the downstream quality layer remains the effective
guarantor of output validity.
Flow post_deal now surfaces enhanced-only fields (extract_strategy,
chunk_count, call_count, structured_warnings, quality_metrics,
debug_info) on top of the baseline JSON envelope. Baseline output
stays exactly as before (unchanged log message, unchanged payload
shape) so existing callers see no diff.
API layer routes those fields into meta:
- meta always carries the three baseline counts when
include_meta=true;
- when strategy=="enhanced": additionally extract_strategy,
chunk_count, call_count, token_usage="unavailable",
structured_warnings, quality_metrics;
- when include_debug=true: debug_info lands in meta regardless of
include_meta (matches design section 7.3);
- top-level warnings[] gains a short "N structured warning(s)"
summary so legacy callers reading only warnings see the enhanced
activity signal.
WkFlowState grows four enhanced-output fields
(chunk_count / structured_warnings / quality_metrics / debug_info)
plus setup() clears them so pipeline-pool reuse cannot leak state
across requests.
Tests: 8 new enhanced-strategy tests on PropertyGraphExtract
(full-pipeline canonical ids, prompt appends constraint block,
schema-invalid item drop with warning, cross-chunk dedup, include_
debug records, quality_metrics shape). 4 new API-layer tests
(enhanced meta carries strategy+chunk+call+token_usage; structured_
warnings + quality_metrics reach meta; debug_info gated on include_
debug; baseline meta byte-compat regression). Full test suite is
green at 562 passed with coverage at 55.50% (floor 34%).
Offline evaluator that scores a predicted property graph against a labelled reference using set-based precision/recall/F1 (matched by (label, id) for vertices, (label, outV, inV) for edges) plus property fidelity metrics (valid_ratio, exact_match_rate). Empty-graph edge cases match standard conventions: both empty → 1.0, one-sided empty degrades the affected component. Ten deterministic scenarios drive both extraction strategies through the same FakeLLM per-chunk responses so their outputs can be compared apples-to-apples. Design invariants are asserted: enhanced never regresses F1 relative to baseline, and enhanced wins strictly on cross-chunk edges, alias mismatch, and INT coercion. Report at docs/quality/schema-based-graph-extract-report.md captures the numeric results (baseline avg F1 0.84 → enhanced 1.00, +0.16). Numbers are reproducible via 'uv run --directory hugegraph-llm pytest .../test_property_graph_benchmark.py -s'. Also adds pytestmark = pytest.mark.contract to the schema/warnings/ quality_gate/postprocess/evaluator/benchmark test modules so CI's '-m "unit or contract"' filter picks them up. Coverage of the enhanced package now sits at 95% (evaluator 94%); 392 unit/contract tests green.
…dd usage doc Adds four scenarios that close the rubric coverage matrix: - s11 missing endpoint referent (edge points at a vertex no chunk defines) - s12 wrong edge direction (Movie -> Person against schema Person -> Movie) - s13 duplicate edge across chunks (enhanced dedupes at raw level) - s14 multi-primary-key vertex (Employee keyed by (name, company)) Benchmark now measures per-scenario post-LLM latency and asserts LLM call count equals chunk count for both strategies. Table gains latency, call-count, and raw-vertex-count columns; explicit F1 relative-gain line at the footer. Failure-case analysis in the effect report replaces the terse "where enhanced wins" note with concrete LLM outputs + baseline vs enhanced behavior + production impact for scenarios s03/s04/s06/s07/s10. Adds docs/quality/schema-based-graph-extract-usage.md covering opt-in via API and Python, response meta layout, structured-warning codes, applicable scenarios, and known limitations (prompt-token overhead, best-effort coercion, no cross-request state). 14-scenario benchmark: baseline avg F1 0.89 -> enhanced 1.00 (+12.9% relative); property_exact_match_rate 0.93 -> 1.00; enhanced never regresses. 396 unit/contract tests pass; enhanced package coverage stays at 95%.
…ction scripts/graph_extract_live_benchmark.py runs both baseline and enhanced strategies against DeepSeek Chat on a 3-chunk Tom Hanks corpus with a labelled ground truth, capturing wall-clock latency, per-call token usage, and an approximate USD cost (rate card 2026-07). Full run archive lands at .workflow/deepseek_live_run.json (gitignored). Effect report gains a Live LLM Benchmark section with: - Aggregate table (F1, match rate, latency, tokens, cost). - Per-call breakdown so token overhead of enhanced's constraint block is visible per chunk (baseline ~413 vs enhanced ~640 prompt tokens/call). - Failure analysis on the two baseline losses in this run: DeepSeek emitted the character 'Chuck Noland' and the pronoun 'He' as Persons. Baseline propagated both; enhanced's constraint block nudged DeepSeek to drop them, and the cross-chunk assembler linked chunk 3's Woody edge back to Tom Hanks from chunk 1. - Cost/quality trade-off discussion (+12.5% cost, +33.3% F1 relative). pyproject.toml: allow T20 (print) in scripts/ so the CLI benchmark passes ruff. Live DeepSeek results (single run, temp 0.0): - baseline: F1 0.75, 5.44s, $0.000987 - enhanced: F1 1.00, 5.22s, $0.001110 Mock benchmark unchanged; 396 unit/contract tests still pass.
Two new scenarios covering cases where neither strategy can win: - s15 character-promoted-to-person. LLM emits "Chuck Noland" (a fictional role from Cast Away) as a first-class Person vertex. The name is schema-valid; neither strategy has world knowledge to catch it. Both F1 caps at 0.86. Reproduces the live DeepSeek failure mode. - s16 pronoun-ghost-no-prior-context. Single-chunk "He voiced Woody in Toy Story." Enhanced's document assembler needs a prior chunk to bind "He" to; without one it emits the ghost Person and its spurious outgoing edge. Both F1 caps at 0.50. Two dedicated tests assert baseline F1 == enhanced F1 on these, so the "enhanced ties baseline on domain limits" claim is machine-checked, not just prose in the report. Design threshold DESIGN_F1_RELATIVE_GAIN_MIN = 0.05 is now asserted in test_benchmark_produces_comparison_table: mock average F1 gain must be >= +5% relative or CI fails. Regressions can no longer sneak in as a number-check in the report; they break the test. Mock benchmark aggregate moves from 14 scenarios (baseline 0.89 -> enhanced 1.00, +12.9%) to 16 scenarios (baseline 0.86 -> enhanced 0.96, +11.6% relative). The lower enhanced ceiling is deliberate: it prevents the report from claiming "enhanced always reaches 1.00" -- a claim that would be dishonest given the live-track ceiling of 0.72. 398 unit/contract tests pass (was 396 with the +2 new assertions).
…ty API) The earlier live benchmark used a hand-authored 3-chunk Tom Hanks corpus with a hand-authored 7-item ground truth. That evaluation suffered from two-layer selection bias: the same person picked the text (choosing spots where baseline was known to fail) AND wrote the answer key. F1=1.00 on that corpus is not defensible as effect evidence. This script builds a reproducible public corpus instead: - Text source: Wikipedia lead extract (via MediaWiki API action=query prop=extracts exintro), pinned by article revid. - Ground truth source: Wikidata P161 (cast member) claims verified per film via wbgetentities. Only films whose entity is instance-of (P31) a film class AND whose P161 lists the actor's Q-id are admitted; TV shows, franchises, and disambiguation-wrong Q-ids are filtered out and recorded in rejected_mentions for auditability. - No SPARQL: Wikidata Query Service is currently under an outage that rate-limits SPARQL to 1 req/min. The Entity API (wbsearchentities + wbgetentities) is separately available and covers this workflow. Politeness/robustness: - Wikimedia-compliant User-Agent with contact URL. - 5s inter-request pacing on Wikidata (429s during the outage are aggressive; empirically this rate works). - 30s -> 60s -> 90s -> 120s -> 150s backoff on 429. - Per-actor cap of 15 candidate films to bound runtime (~15 min total for 8 actors). - Cross-actor Q-id cache (film titles overlap across actors' bios). Output: hugegraph-llm/src/tests/data/public_actor_corpus.json with per-corpus wikipedia_url, actor_qid, verified_films, rejected_mentions so a reviewer can independently audit every admit/reject decision.
Frozen output of scripts/build_public_actor_corpus.py at build time 2026-07-05. Committed so live benchmark reruns produce comparable numbers without needing a fresh Wikipedia/Wikidata fetch (which is sensitive to article rev changes and Wikidata rate limits during the current WDQS outage). Contents: - 8 actors: Tom Hanks, Meryl Streep, Leonardo DiCaprio, Denzel Washington, Julia Roberts, Anthony Hopkins, Nicole Kidman, Morgan Freeman. - 24 chunks (3 per actor, sentence-boundary split at ~800 chars). - 65 GT vertices, 57 GT edges (all Person -> Movie ACTED_IN). - Per-corpus wikipedia_revision, wikipedia_url (permalink), actor_qid, verified_films list, rejected_mentions list with reason codes. Text is CC-BY-SA (Wikipedia); attribution is preserved via the wikipedia_url field per corpus entry. Ground truth is Wikidata (CC0). No PII, no author-fabricated content. To rebuild: python scripts/build_public_actor_corpus.py --output hugegraph-llm/src/tests/data/public_actor_corpus.json. Rebuild will differ if the underlying Wikipedia articles have been edited since the pinned revids.
…aggregation
New CLI:
python scripts/graph_extract_live_benchmark.py \
--corpus <path-to-public_actor_corpus.json> \
--runs 3 \
--output <archive.json>
Legacy behavior preserved for callers who omit --corpus: the previous
hand-authored 3-chunk Tom Hanks CORPUS + GROUND_TRUTH is kept as a
smoke/sanity sample (name: legacy_tom_hanks_3chunk).
Aggregation shape:
- Every (corpus, strategy, run) tuple stored individually in the
archive under "runs" so per-call token usage and raw predicted
output stay auditable.
- per_corpus_strategy_aggregation: {mean, std, min, max, n} over
runs for F1 / vertex F1 / edge F1 / property match / latency /
tokens / cost.
- per_strategy_aggregation: same, pooled across all corpora.
- delta.f1_absolute and delta.f1_relative_percent.
Why multi-run at temperature 0.0: DeepSeek at temp=0 is
near-deterministic but not fully; token counts vary a few percent
across identical requests due to server-side batching. std across 3
runs of the same (corpus, strategy) turns out non-trivial for a few
corpora, so multi-run is not vestigial.
Public-corpus results on 8 actors x 3 runs = 48 samples:
- baseline F1 = 0.418 +/- 0.251, enhanced F1 = 0.439 +/- 0.188
- delta: +0.020 absolute (+4.8% relative)
- latency: 16.57s -> 14.80s (-10.7%)
- cost/document: $0.003776 -> $0.003508 (-7.1%)
- enhanced >= baseline on 5/8 corpora, regresses on 3/8
Numbers documented in
docs/quality/schema-based-graph-extract-report.md (Live LLM Benchmark
on Public Corpus section) in the docs commit that follows.
…hmark An earlier revision of this report leaned on a hand-authored 3-chunk Tom Hanks corpus with a hand-authored 7-item ground truth and reported F1=1.00 on enhanced (+33.3% relative vs baseline). That evaluation suffered from two-layer selection bias -- the same person picked the text and wrote the answer key. The +33.3% number is not defensible as effect evidence. This commit rewrites the report on top of: - Mock rubric benchmark (16 scenarios, deterministic FakeLLM). Serves as rubric coverage. Enhanced 0.86 -> 0.96, +11.6% relative. - Live DeepSeek benchmark on public corpus (8 Wikipedia articles, Wikidata-verified GT, 3 runs each = 48 samples). Serves as effect evidence. Enhanced 0.418 +/- 0.251 -> 0.439 +/- 0.188, +4.8% relative. The public-corpus numbers are much smaller than the previous hand- authored numbers. That gap is the price of an honest evaluation. Real effect is: - Modest F1 gain (+4.8% relative average) - Substantial variance reduction (-25% std) - Big worst-case rescue (Tom Hanks: 0.052 -> 0.327) - Latency down 10.7%, cost/document down 7.1% - Enhanced regresses on 3 of 8 corpora where baseline was already strong (Julia Roberts, Denzel Washington, Morgan Freeman) New sections: - Evaluation Methodology: explains why the two tracks exist and what each measures. - Live corpus provenance: Wikipedia REST + Wikidata Entity API workflow; per-actor stats table with rejection counts. - Per-corpus breakdown: shows the 5/3 win/loss split explicitly. - Live failure case analysis: two dominant failure modes documented (Person canonical name mismatch across bio-style writing, LLM extracting films Wikidata hasn't cast-tagged). - Expanded Scope and Caveats: strict-GT limits, WDQS-outage note, sample-size discussion. Usage doc gains: - "No knowledge-base entity resolution" limitation entry describing the Wikipedia "Thomas Jeffrey Hanks" vs Wikidata "Tom Hanks" canonical alias problem both strategies hit. - "When enhanced does NOT help" section describing the 3/8 regression pattern. Design threshold for enhanced (>= +5% relative F1 average on the 16-scenario mock benchmark) is now baked into a CI assertion; the report references it rather than repeating the number.
…driver The live benchmark script previously accepted --corpus as optional and fell back to a hard-coded 3-chunk Tom Hanks corpus with hand-authored ground truth when the flag was omitted. That fallback contradicts the public-corpus track's narrative: the whole point of moving to Wikipedia lead + Wikidata P161 GT was to remove selection bias, and keeping an in-script hand-authored corpus silently reintroduces it if anyone runs the script without --corpus. Changes: - Delete the hard-coded CORPUS list and GROUND_TRUTH dict. - Make --corpus required; argparse now rejects missing values. - Update module docstring to explain the deliberate absence of a fallback so future maintainers do not "helpfully" re-add one. The committed public corpus at hugegraph-llm/src/tests/data/public_actor_corpus.json is now the only input path.
Adds the full 48-run archive (8 corpora x 2 strategies x 3 runs) from the live DeepSeek benchmark. Previously this file lived only in .workflow/, which is repo-locally excluded — so reviewers looking at the effect report could see the aggregated numbers but had no way to cross-check them against per-run data. The archive contains, for each of the 48 runs: F1 (overall/vertex/edge), property match rate, per-LLM-call prompt/completion tokens and latency, predicted vertices/edges, and the full evaluator output. Aggregated mean/std/min/max blocks per (corpus, strategy) and per strategy are included at the top level. Reviewers can jq every number in the report's live-track section directly from this file — see the new "How to Reproduce This Report" section for the exact queries. The file is 612 KB. It contains no API keys, no raw LLM prompt or completion text — only token counts, latencies, and post-LLM parsed graph output. Model identifier "deepseek/deepseek-chat" and cache-miss pricing are publicly known.
…tionale
Two additions to the schema-based-graph-extract-report:
1. New "Why the live delta (+4.8%) is smaller than the mock delta
(+11.6%)" subsection in the executive summary. Explains, in four
parts:
- The number is smaller than the previous (+33.3%) claim because
that claim was double-selection-biased; +4.8% is what survives
honest measurement.
- The F1 mean is the least-interesting metric in the results —
variance reduction (-25%), worst-case improvement (4.4x),
latency (-11%), and cost (-7%) are the load-bearing claims.
- Mock (+11.6%) measures the pipeline in isolation; live (+4.8%)
measures pipeline + LLM. Both numbers are real; they measure
different things.
- The +5% CI threshold is asserted on the mock track, not the
live track, because live is subject to server-side variance and
Wikidata-GT completeness limits that would either loosen the
threshold to uselessness or tighten it below normal noise.
2. New top-level "How to Reproduce This Report" section listing every
committed artifact (public corpus JSON, live-benchmark archive JSON,
builder script, driver script) with paths, plus the exact
commands to regenerate each track. Includes three jq queries that
let a reviewer cross-check every live-track number against the
committed archive without re-running the benchmark.
The two per-track "Reproducing X" subsections that were duplicating
this material have been shortened to backlinks.
No numbers changed.
dosubot
Bot
added
size:XXL
Move the 48-run live-benchmark archive out of the PR to shrink the diff. The archive was ~600 KB / 21 k lines, roughly 67% of the total change; removing it lets reviewers focus on the actual feature code and tests. All numbers in the effect report remain valid. The archive is available on request; report now includes sha256 for integrity verification, and the jq examples still describe how to derive every number once the archive is obtained. sha256(live_benchmark_public_actors.json) = 8c7b7a8c22451405d9a6f4403dae09a534777c097a396cfcb4e563fc9e04b1e7
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an opt-in enhanced strategy for
/graph/extractthat appliesa schema-aware quality layer on top of the existing LLM-based
extraction. The current pipeline (renamed here as baseline) stays
byte-compatible: enhanced only runs when the caller sends
extract_strategy: "enhanced".The new layer runs a candidate parser → schema-aware normalizer →
cross-chunk document assembler → quality gate on every LLM response,
producing structured warnings + a quality-metrics block alongside the
usual
vertices/edges.Closes #74.
Effect
Two independent evaluation tracks — the mock rubric benchmark is for
coverage of every rubric edge case; the live public-corpus benchmark
is for effect evidence under externally-authored ground truth.
Design-stage threshold (baked into a CI assertion inside the mock
benchmark test): enhanced ≥ +5 % relative F1 on the 16-scenario rubric
average. Regressions below this floor fail CI.
Live-track secondary metrics (across all 48 runs):
Interpretation. Enhanced is a safety-net strategy: it rescues the
worst cases (Tom Hanks corpus F1: 0.052 → 0.327) and dramatically
narrows the F1 variance (−25 %) while sending +40 % prompt tokens but
producing −24 % completion tokens. Net effect: modest average F1 gain,
much more predictable outcomes, slightly cheaper and faster. It is not
a strict Pareto improvement — on 3 / 8 corpora where baseline was
already strong, enhanced's stricter dedup merges near-duplicate film
titles and regresses F1 by 0.10-0.20. This is documented in the report,
not glossed over.
Full analysis, per-corpus tables, and concrete failure cases (including
the Wikipedia "Thomas Jeffrey Hanks" ↔ Wikidata "Tom Hanks" canonical
name mismatch that hits both strategies) in
docs/quality/schema-based-graph-extract-report.md.
Backwards Compatibility
None broken. Two new optional request fields
(
extract_strategy,include_debug, both defaulted) and enhanced-onlymetakeys that appear only when the caller opts in. Baselineresponses are byte-identical to the pre-PR shape.
How to Enable
Full usage guide (API + Python + limitations) at
docs/quality/schema-based-graph-extract-usage.md.
What's in the box
New subpackage hugegraph-llm/src/hugegraph_llm/operators/llm_op/property_graph_extract_enhanced/:
schema_index.py— compiled runtime schema index (label lookup,canonical id, property coercion).
candidate_parser.py— tolerant JSON extractor covering 5LLM-output formats.
normalizer.py— per-chunk schema-aware normalizer (propertyfilter → coerce → PK check → canonical id).
document_assembler.py— cross-chunk alias-table union withambiguity detection and endpoint repair.
quality_gate.py— 11-metric quality aggregate per run.warnings.py— structured warning registry (20 codes).evaluator.py— offline evaluator (precision/recall/F1 + propertyfidelity metrics).
prompt_contract.py— schema-derived constraint block thatenhanced appends to the LLM prompt.
Threaded through the existing pipeline:
api/graph_extract_api.py,flows/graph_extract.py,operators/llm_op/property_graph_extract.py,state/ai_state.py,nodes/llm_node/extract_info.py.Public-corpus builder + live benchmark driver (outside CI, network
required):
scripts/build_public_actor_corpus.py— fetches Wikipedia leadextracts + verifies GT against Wikidata
P161 (cast member)claimsvia the entity API. Output pinned to
hugegraph-llm/src/tests/data/public_actor_corpus.json.scripts/graph_extract_live_benchmark.py— accepts--corpus(required, no hand-authored fallback) and
--runs; produces themean ± std, per-corpus, per-strategy JSON archive.
LLM call's prompt/completion tokens, latency, and raw predicted
output) is not bundled in this PR to keep the diff reviewable —
the archive alone would add ~21 k lines (~67 % of the total diff).
Available on request; sha256:
8c7b7a8c22451405d9a6f4403dae09a534777c097a396cfcb4e563fc9e04b1e7.Every number in the effect report's live section is
jq-derivablefrom the archive (queries in the report's "How to Reproduce This
Report" section).
Tests (all
unit or contract, deterministic):test_property_graph_schema.pytest_property_graph_warnings.pytest_property_graph_postprocess.py(parser + normalizer + assembler)test_property_graph_quality_gate.pytest_property_graph_evaluator.pytest_property_graph_benchmark.py(16 rubric scenarios, incl.s15/s16 domain-limit cases where enhanced ≡ baseline)
test_property_graph_extract.py(extended for enhanced dispatch)test_graph_extract_api.py(extended for API-layer meta)Test Plan
parser 97 %, assembler 90 %, quality_gate 100 %, warnings 100 %).
-m "unit or contract"filter, well above the 34 % CI floor.baseline meta remains
{vertex_count, edge_count, text_count}.≥ baseline F1 on every scenario.
avg F1 gain ≥ +5 % relative. Actual: +11.6 %.
latency, tokens, cost) for all 8 corpora × 3 runs = 48 samples.
Scope Notes
corpora and is the biggest single F1 loss source. Wikipedia leads
say "Thomas Jeffrey Hanks"; Wikidata canonicals say "Tom Hanks". No
schema-only pipeline can bridge that without either LLM alignment or
an entity-resolution step. Both are out of scope for Issue feat: Add progress bars and print information for building vector indexes #74. This
is documented as a known limitation in the usage doc, not hidden.
token_usagefield inmetaremains a placeholder pending anLLM adapter change to surface token metadata through
BaseLLM. Thelive benchmark script tracks tokens by calling the LiteLLM completion
API directly.
duplication doesn't get double-counted; raw counts are still exposed
in
ItemMetrics.predicted_count_rawfor downstream commit-loadmonitoring.