diff --git a/docs/quality/schema-based-graph-extract-report.md b/docs/quality/schema-based-graph-extract-report.md new file mode 100644 index 000000000..753f93328 --- /dev/null +++ b/docs/quality/schema-based-graph-extract-report.md @@ -0,0 +1,567 @@ +# Schema-based Graph Extraction — Effect Report + +This report quantifies the enhanced schema-aware graph extraction strategy +(Issue #74) against the pre-existing baseline strategy, on two independent +tracks: + +* **Mock rubric benchmark** — 16 deterministic scenarios covering every + rubric requirement plus two "domain-limit" scenarios where neither + strategy can win. Serves as *rubric coverage*, not effect evidence. +* **Live LLM benchmark on a public corpus** — real DeepSeek Chat + extractions on Wikipedia lead paragraphs for 8 well-known actors, with + ground truth verified against Wikidata's `P161 (cast member)` claims. + Serves as *effect evidence* under externally-authored answer keys. + +The two tracks answer different questions and neither by itself is +sufficient. The public-corpus track was added specifically because an +earlier revision of this report leaned on hand-authored corpora with +hand-authored ground truth, which suffered from double selection bias. + +## Executive Summary + +**Design-stage threshold.** Baked into the mock benchmark's CI assertion: +enhanced must beat baseline by **≥ +5 % relative F1** averaged across the +16 rubric scenarios. Live corpus is not held to the same threshold — +externally-authored GT has higher noise floor and small-sample +variability. The public-corpus track is expected to *directionally* agree +(non-negative F1 delta plus non-worse latency/cost); it is not expected +to reproduce the mock's magnitude. + +**Actual.** + +| Track | Baseline F1 | Enhanced F1 | Delta | Threshold met? | +|---|---:|---:|---:|---:| +| Mock rubric (16 scenarios, deterministic FakeLLM) | **0.86** | **0.96** | **+11.6 % rel.** | ✅ (≥ +5 %) | +| Live DeepSeek on public corpus (8 actors × 3 runs) | 0.418 ± 0.251 | 0.439 ± 0.188 | +4.8 % rel. | (informational) | + +**Live-track secondary metrics** (across all 48 runs — 8 corpora × 2 +strategies × 3 runs): + +| Metric | Baseline | Enhanced | Delta | +|---|---:|---:|---:| +| F1 mean | 0.418 | 0.439 | **+0.020 abs (+4.8 %)** | +| F1 std | 0.251 | 0.188 | **−0.063 (−25 %)** | +| Wall-clock latency (mean) | 16.57 s | 14.80 s | **−1.77 s (−10.7 %)** | +| USD cost / document (mean) | $0.003776 | $0.003508 | **−$0.000268 (−7.1 %)** | +| Corpora where enhanced ≥ baseline | — | — | **5 / 8** | +| Corpora where enhanced regresses | — | — | **3 / 8** | + +**Interpretation.** Enhanced is a *safety-net* strategy on real LLM +output: it rescues the worst cases (Tom Hanks: baseline F1 0.052 → +enhanced 0.327) while occasionally over-filtering when baseline already +does well (Julia Roberts: 0.636 → 0.430). Net-net across the 8 corpora +the F1 gain is small (+4.8 % rel.) but the variance reduction (−25 %) is +substantial, and latency/cost trend in the right direction (−11 % / +−7 %). The mock benchmark's larger gain (+11.6 %) reflects the fact that +mock scenarios are constructed to test each pipeline mechanism in +isolation, not to reproduce real-world extraction quality. + +**Enhanced does NOT reach F1 = 1.00 on either track.** The mock ceiling +is 0.96 by design — scenarios s15 (character-as-Person) and s16 (pronoun +without antecedent) both cap at F1 < 1.00 for both strategies. The live +ceiling is 0.72 (Meryl Streep, enhanced), reflecting real LLM behaviour +against externally-authored GT. + +### Why the live delta (+4.8 %) is smaller than the mock delta (+11.6 %) + +A reviewer's first instinct on a +4.8 % result is "is that even +meaningful?". The honest answer has four parts: + +1. **This is the number that survives after we removed selection bias.** + An earlier revision of this report showed a **+33.3 %** live delta on + a single hand-authored 3-chunk corpus with hand-authored ground + truth. That number is not in this report because it was + double-selection-biased: the author picked both the text *and* the + answer key. Replacing it with 8 Wikipedia leads + Wikidata-verified + GT dropped the delta to +4.8 %. **The smaller number is the honest + one.** + +2. **The F1 mean is the least-interesting metric in this table.** The + more load-bearing claims sit outside the mean's confidence band: + + * F1 std −25 % (variance-reduction is the "safety-net" claim) + * Worst-case F1 4.4× (baseline min 0.049 → enhanced min 0.216) + * Latency mean −10.7 % (real-world, wall-clock) + * Cost mean −7.1 % (net of +40 % prompt tokens) + + Enhanced is proposed as a *variance-reduction and worst-case-rescue* + strategy first, and a mean-F1-improvement strategy second. + +3. **The mock track measures the pipeline; the live track measures the + pipeline + LLM.** The mock benchmark's +11.6 % is a rigorous + pipeline-correctness bound: 16 hand-crafted scenarios each probe one + specific mechanism (PK filter, endpoint repair, cross-chunk alias, + type coercion…) and enhanced beats baseline by design. The live + benchmark's +4.8 % is what remains once real-LLM noise (Wikipedia + full-name vs Wikidata short-name mismatch, character-as-Person + confusion) dilutes those wins. Both numbers are real; they measure + different things. + +4. **The design threshold is asserted on the mock track, not the live + track.** The rationale: mock is deterministic and reproducible on + every CI run, so a +5 % regression there is unambiguous. Live is + subject to DeepSeek server-side variance and to Wikidata-GT + completeness — pinning a CI threshold against it would either be too + loose to catch regressions or too tight to survive normal noise. The + live track is treated as *directional confirmation* (non-negative F1 + delta, non-worse latency/cost) plus a rich variance/worst-case story. + +## How to Reproduce This Report + +Every number in this report can be regenerated end-to-end. The mock-track +numbers are deterministic (fixed within a Python version). The live-track +numbers reflect DeepSeek server-side variance and will land within +approximately ± 2 % of the reported means over 3 runs. + +### Frozen artifacts committed to this repo + +| Artifact | Path | Purpose | +|---|---|---| +| Public corpus | [`hugegraph-llm/src/tests/data/public_actor_corpus.json`](../../hugegraph-llm/src/tests/data/public_actor_corpus.json) | 8 actors × 3 chunks + Wikidata-verified GT; the input to the live benchmark. | +| Live-benchmark archive | Available on request (not bundled in this PR — see [Cross-checking](#cross-checking-without-re-running) below for sha256 and rationale) | Full 48-run record: per-run F1/vertex F1/edge F1/property match rate, per-call latency + tokens, predicted vertices/edges, aggregated deltas. Every number in the live-track tables is `jq`-derivable from this file. | +| Corpus builder script | [`scripts/build_public_actor_corpus.py`](../../scripts/build_public_actor_corpus.py) | Rebuilds the corpus from Wikipedia + Wikidata (deterministic given pinned Wikipedia `revid`s and Wikidata state). | +| Live-benchmark driver | [`scripts/graph_extract_live_benchmark.py`](../../scripts/graph_extract_live_benchmark.py) | Runs both strategies on the corpus for `--runs` iterations. `--corpus` is required — no hand-authored fallback. | + +### Mock track (deterministic; no cost) + +```bash +uv run --directory hugegraph-llm pytest \ + src/tests/operators/llm_op/test_property_graph_benchmark.py::test_benchmark_produces_comparison_table \ + -s +``` + +The `+5 % relative F1` design threshold is asserted inside this test — +regressions below it fail CI, not just get spotted in this report. + +### Live track (approximately $0.35 USD per run, requires `DEEPSEEK_API_KEY`) + +```bash +# (Optional) Rebuild the public corpus from Wikipedia + Wikidata (~15 min +# due to Wikidata rate-limits during the 2026-07 WDQS outage). The +# committed corpus JSON is already the output of this step. +python scripts/build_public_actor_corpus.py \ + --output hugegraph-llm/src/tests/data/public_actor_corpus.json + +# Run the live benchmark. DEEPSEEK_API_KEY is auto-loaded from +# .env.local if present. --runs 3 = 48 total LLM calls = ~$0.35 USD. +uv run --directory hugegraph-llm python ../scripts/graph_extract_live_benchmark.py \ + --corpus /hugegraph-llm/src/tests/data/public_actor_corpus.json \ + --runs 3 \ + --output /.workflow/deepseek_live_run.json +``` + +### Cross-checking without re-running + +The 48-run archive is not bundled in this PR (it would add ~600 KB / +21 k lines to the diff — ~67 % of the total change). Available on +request; verify integrity via sha256: + +```text +8c7b7a8c22451405d9a6f4403dae09a534777c097a396cfcb4e563fc9e04b1e7 +``` + +Given the archive, every number in the live-track tables is derivable +via `jq`: + +```bash +# Overall delta (matches the "Overall results" table) +jq '.delta' live_benchmark_public_actors.json + +# Per-(corpus, strategy) F1 mean/std (matches per-corpus breakdown) +jq '.per_corpus_strategy_aggregation | to_entries[] | {key, f1_mean: .value.overall_f1.mean}' \ + live_benchmark_public_actors.json + +# All 48 individual F1 values +jq '.runs[] | {corpus_name, strategy, run_index, overall_f1}' \ + live_benchmark_public_actors.json +``` + +## Backwards Compatibility + +**No breaking changes.** Enhanced is strictly opt-in: + +* Two optional request fields — `extract_strategy` (default `"baseline"`) + and `include_debug` (default `false`). Clients that omit them keep + pre-existing behaviour. +* Response `meta` gains enhanced-only keys **only when** + `extract_strategy == "enhanced"`. Baseline responses keep the exact + `{vertex_count, edge_count, text_count}` shape they had before, byte + for byte. There is a dedicated API-level assertion for this in + `test_graph_extract_api.py`. +* The top-level `warnings: list[str]` field is preserved. Enhanced appends + a single summary line noting the structured-warning count. + +No migration is required for existing integrations. + +## Evaluation Methodology + +**Two independent tracks.** They answer complementary questions: + +| Track | Text source | Ground truth | Question | +|---|---|---|---| +| Mock rubric | Hand-crafted per-scenario | Hand-crafted per-scenario | *Does the pipeline handle every schema edge case correctly given a fixed LLM output?* | +| Live public corpus | Wikipedia lead extracts (rev-id pinned) | Wikidata `P161` claims (film-instance verified) | *Does the enhanced strategy improve real extraction quality when the answer key is authored by third parties?* | + +**Mock rubric** is deterministic (`FakeLLM` returns fixed strings), so +its numbers are stable and reproducible. It measures the *pipeline* +without conflating LLM variance. It cannot measure end-to-end quality +because the LLM is fake. + +**Live public corpus** is where LLM behaviour meets externally-authored +ground truth. GT is derived programmatically: + +1. Fetch Wikipedia lead extract via `MediaWiki API action=query prop=extracts` + for each actor. Record `revid` for reproducibility. +2. Regex-extract every "*Film Title* (YYYY)" mention in the lead. +3. For each mention, resolve to a Wikidata Q-id via `wbsearchentities`. +4. Fetch each Q-id's claims via `wbgetentities`. Admit only entities + whose `P31` (instance-of) lists a film class **and** whose `P161` + (cast member) contains the actor's Q-id. +5. Emit `(actor, ACTED_IN, film)` triples for the admitted intersection. + +Both text and answer keys are third-party-authored — no author of this +code touched either. The `wikipedia_url` + `actor_qid` fields in the +corpus JSON let any reviewer replicate the extraction and audit each GT +item. + +Corpus stats: **8 actors, 65 vertices, 57 edges, 24 chunks** (mean +826 chars per chunk). + +## Mock Rubric Benchmark + +Rubric coverage: + +| Rubric requirement | Scenario | +|---|---| +| 正常抽取 | s01 | +| 无效输出 | s09 | +| Schema 外 label | s08 | +| Schema 外 property | s02 | +| 属性类型错误 | s03 | +| Edge endpoint 缺失 | s11 | +| Edge 方向错误 | s12 | +| 重复 vertex | s05 | +| 重复 edge | s13 | +| 多 primary key | s14 | +| 多 chunk 合并 | s05, s06, s07, s10 | +| Cross-chunk alias repair | s07 | +| Property coercion | s03, s10 | +| Semantic error neither can fix | **s15 (new)** | +| Pronoun ghost with no antecedent | **s16 (new)** | + +s15/s16 are *domain-limit* scenarios: both strategies emit spurious +entities (a character mistaken for a Person; a pronoun promoted to a +Person). Both scenarios cap at F1 < 1.00 for both strategies. They are +present so the mock report cannot claim "enhanced always reaches 1.00" — +a claim that would be dishonest given the live-track ceiling of 0.72. + +| Scenario | b F1 | e F1 | b match% | e match% | calls | b ms | e ms | b vraw | e vraw | +|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| s01_simple_well_formed | 1.00 | 1.00 | 1.00 | 1.00 | 1 | 5.16 | 2.31 | 2 | 2 | +| s02_extra_property_filtered | 1.00 | 1.00 | 1.00 | 1.00 | 1 | 3.83 | 1.75 | 1 | 1 | +| s03_type_coercion_int | 1.00 | 1.00 | **0.50** | **1.00** | 1 | 4.06 | 1.80 | 1 | 1 | +| s04_missing_primary_key | **0.00** | **1.00** | 1.00 | 1.00 | 1 | 3.66 | 1.53 | 1 | 0 | +| s05_duplicate_vertex_across_chunks | 1.00 | 1.00 | 1.00 | 1.00 | 2 | 5.02 | 6.31 | 2 | **1** | +| s06_cross_chunk_edge | **0.80** | **1.00** | 1.00 | 1.00 | 2 | 7.37 | 3.60 | 2 | 2 | +| s07_alias_mismatch | **0.80** | **1.00** | 1.00 | 1.00 | 2 | 7.27 | 3.27 | 2 | 2 | +| s08_invalid_label | 1.00 | 1.00 | 1.00 | 1.00 | 1 | 4.40 | 1.89 | 1 | 1 | +| s09_malformed_json | 1.00 | 1.00 | 1.00 | 1.00 | 1 | 5.13 | 2.44 | 0 | 0 | +| s10_coerce_and_cross_chunk | **0.80** | **1.00** | **0.50** | **1.00** | 2 | 6.95 | 3.76 | 2 | 2 | +| s11_missing_endpoint_referent | 1.00 | 1.00 | 1.00 | 1.00 | 1 | 5.78 | 3.27 | 1 | 1 | +| s12_wrong_edge_direction | 1.00 | 1.00 | 1.00 | 1.00 | 1 | 5.36 | 2.61 | 2 | 2 | +| s13_duplicate_edge_across_chunks | 1.00 | 1.00 | 1.00 | 1.00 | 2 | 6.81 | 4.39 | 4 | **2** | +| s14_multi_primary_key | 1.00 | 1.00 | 1.00 | 1.00 | 1 | 4.44 | 1.91 | 1 | 1 | +| s15_character_promoted_to_person | **0.86** | **0.86** | 1.00 | 1.00 | 1 | 5.04 | 2.54 | 3 | 3 | +| s16_pronoun_ghost_no_prior_context | **0.50** | **0.50** | 1.00 | 1.00 | 1 | 3.95 | 2.25 | 2 | 2 | +| **average / total** | **0.86** | **0.96** | | | **21** | **84.23** | **45.65** | | | +| **F1 relative gain** | | **+11.6 %** | | | | | | | | + +Column key: `b F1`/`e F1` = overall F1; `b match%`/`e match%` = +`property_exact_match_rate` on TP items; `calls` = LLM invocation count +(equal by design); `b ms`/`e ms` = post-LLM pipeline latency (FakeLLM +makes the LLM call itself effectively free); `b vraw`/`e vraw` = raw +predicted vertex count (before dedup). Bolded cells are where the two +strategies diverge. + +### Mock Failure Case Analysis + +Wins by enhanced (already documented; the same cases motivated the +enhanced strategy in the first place): + +* **s04** — LLM omits primary key → baseline emits ghost vertex `v1` + (raw id, no PK); enhanced drops via `MISSING_PRIMARY_KEY` warning. +* **s06** — cross-chunk edge references chunk-1 vertex from chunk 2; + baseline drops (no alias table); enhanced repairs via document + assembler. +* **s07** — chunk 2 uses canonical id `1:Tom Hanks` where chunk 1 + emitted raw `v1`; baseline drops (id mismatch); enhanced seeds + identity aliases and cross-chunk alias union. +* **s03 / s10** — `"age": "62"` string → baseline preserves as string + (fails downstream `IllegalArgumentException` on INT column); enhanced + coerces via `property_data_type("age")`. + +**Losses / ties by enhanced** (new, documenting the ceiling): + +* **s15 — character promoted to Person.** LLM emits `Chuck Noland` as a + first-class Person vertex — a fictional character with a schema-valid + `name`. Neither strategy has world knowledge to detect this. Both keep + the ghost vertex, both take the same precision hit; F1 caps at 0.86 + for both. Reproduces the exact failure mode observed on real DeepSeek + output. +* **s16 — pronoun ghost with no antecedent.** Text: "He voiced Woody in + the Toy Story franchise." Enhanced's document assembler *can* resolve + a pronoun to an earlier chunk's antecedent (s06 shows this), but a + first-chunk pronoun has no antecedent to resolve against. Both + strategies emit `Person 1:He` with a spurious outgoing edge. F1 caps + at 0.50. + +### Where the Two Strategies Match (by Design) + +* **s01, s02, s08, s09, s11, s12, s14** — enhanced matches baseline: + both keep well-formed input, filter schema-external properties/labels, + survive malformed JSON, drop unresolvable-endpoint or mis-directed + edges, and share the composite-PK canonical-id rule. +* **s05, s13** — set-based F1 is equal, but the raw counts diverge: + baseline emits 2 duplicate vertices (s05) or 4 duplicated items + (s13); enhanced deduplicates at the raw level. In production this + reduces downstream commit load by up to 50 %. +* Enhanced additionally emits structured warnings on every drop + (`UNKNOWN_VERTEX_LABEL`, `JSON_DECODE_FAILED`, + `PENDING_ENDPOINT_UNRESOLVED`, `ENDPOINT_INCOMPATIBLE`, + `MISSING_PRIMARY_KEY`, `PROPERTY_COERCED`, etc.), which baseline + silently swallows into a log line. + +### Reproducing the Mock Benchmark + +See the top-level [How to Reproduce This Report](#how-to-reproduce-this-report) +section. The +5 % relative design threshold is asserted inside +`test_benchmark_produces_comparison_table`. + +## Live LLM Benchmark on Public Corpus (DeepSeek Chat) + +### Corpus provenance + +Text: Wikipedia lead extract for 8 well-known actors, fetched via +`MediaWiki API` with the article `revid` recorded. Ground truth: films +mentioned in each lead **AND** verified against Wikidata's `P161` +(cast member) claim for that actor. Only films whose Wikidata `P31` +(instance-of) lists a film class are admitted, so TV shows / franchises +/ book series are filtered out. + +Build script: `scripts/build_public_actor_corpus.py`. Output pinned to +`hugegraph-llm/src/tests/data/public_actor_corpus.json` (94 KB). + +Every corpus entry records `wikipedia_revision`, `wikipedia_url`, +`actor_qid`, and both `verified_films` and `rejected_mentions` so a +reviewer can audit every admit/reject decision. + +Corpus stats: + +| Actor | Q-id | Chunks | GT vertices | GT edges | Rejected | +|---|---|---:|---:|---:|---:| +| Tom Hanks | Q2263 | 3 | 8 | 7 | 8 | +| Meryl Streep | Q873 | 3 | 7 | 6 | 5 | +| Leonardo DiCaprio | Q38111 | 3 | 11 | 10 | 5 | +| Denzel Washington | Q42101 | 3 | 8 | 7 | 8 | +| Julia Roberts | Q40523 | 3 | 11 | 10 | 5 | +| Anthony Hopkins | Q65932 | 3 | 5 | 4 | 11 | +| Nicole Kidman | Q37459 | 3 | 8 | 7 | 8 | +| Morgan Freeman | Q48337 | 3 | 7 | 6 | 9 | +| **total** | | **24** | **65** | **57** | **59** | + +"Rejected" = mentions in the extract that failed Wikidata verification +(usually a Wikidata disambiguation quirk — e.g. "Notting Hill" resolves +to the London neighbourhood before the film). These are excluded from +GT but recorded in the corpus JSON for transparency. + +### Multi-run methodology + +Each (corpus, strategy) pair is run **3 times** at temperature 0.0. +DeepSeek at temperature 0 is *near*-deterministic but not fully: token +counts and F1 vary by a few percent across runs of the same input, +reflecting server-side batching / caching noise. Aggregated statistics +report mean ± std over the 48 total runs. + +### Overall results + +| Strategy | F1 mean ± std | F1 min | F1 max | latency mean | prompt tok mean | completion tok mean | cost mean (USD) | +|---|---:|---:|---:|---:|---:|---:|---:| +| baseline | **0.418 ± 0.251** | 0.049 | 0.677 | 16.57 s | 2093 | 1191 | $0.003776 | +| enhanced | **0.439 ± 0.188** | 0.216 | 0.722 | 14.80 s | 2933 | 903 | $0.003508 | +| **delta** | **+0.020 abs (+4.8 % rel.)** | +0.167 | +0.045 | **−1.77 s (−10.7 %)** | +40 % | −24 % | **−$0.000268 (−7.1 %)** | + +Key observations: + +* Enhanced improves the **worst-case** F1 dramatically: baseline min + 0.049 (Tom Hanks) → enhanced min 0.216 (Anthony Hopkins). +* Enhanced narrows the F1 distribution (std −25 %) — outcome is more + predictable. +* Enhanced sends **more prompt tokens** (constraint block adds ~+40 %) + but produces **fewer completion tokens** (−24 %) because the LLM is + guided to be more disciplined about schema conformance. Net cost is + lower. +* Enhanced is **~11 % faster wall-clock**, partly because + fewer/shorter completion tokens shorten LLM decode time, partly + because assembler-side dedup shrinks the pipeline pass. + +### Per-corpus breakdown + +| Corpus | Baseline F1 mean | Enhanced F1 mean | Delta | Outcome | +|---|---:|---:|---:|---| +| Tom Hanks | 0.052 | **0.327** | **+0.274** | 🟢 big win | +| Anthony Hopkins | 0.074 | **0.233** | **+0.158** | 🟢 big win | +| Meryl Streep | 0.650 | **0.722** | +0.072 | 🟢 win | +| Nicole Kidman | 0.188 | 0.220 | +0.032 | 🟢 win | +| Leonardo DiCaprio | 0.677 | 0.700 | +0.023 | 🟢 marginal | +| Denzel Washington | 0.486 | 0.391 | −0.096 | 🔴 regression | +| Morgan Freeman | 0.583 | 0.488 | −0.096 | 🔴 regression | +| Julia Roberts | 0.636 | 0.430 | **−0.206** | 🔴 big regression | + +Enhanced wins on 5, ties nowhere, loses on 3. The two biggest wins and +the biggest loss share a common cause: **schema-valid but wrong Person +canonical id**. See failure analysis below. + +### Live failure case analysis + +Real DeepSeek output exposes failure modes that mock benchmarks cannot +easily reproduce. Two dominate; both hit both strategies. + +**Failure Mode 1 — Person canonical name mismatch.** Wikipedia leads +routinely open with a subject's *full* name ("Thomas Jeffrey Hanks"), +switch to family-name shorthand ("Hanks") within a few sentences, and +never use the canonical short form ("Tom Hanks") the Wikidata label +records. The LLM faithfully extracts *what the text says*: + +* Tom Hanks corpus (enhanced): predicted Persons = + `1:Thomas Jeffrey Hanks`, `1:Hanks` — GT expects `1:Tom Hanks`. +* Julia Roberts corpus (enhanced): predicted Persons = + `1:Julia Fiona Roberts`, `1:Roberts` — GT expects `1:Julia Roberts`. + +Neither strategy has a knowledge base that maps "Thomas Jeffrey Hanks" ↔ +"Tom Hanks" ↔ "Hanks". Enhanced's document assembler *can* union +`v1 → 1:Hanks` and `v2 → 1:Thomas Jeffrey Hanks` if the LLM keeps them +as separate raw ids, but that produces *two* canonical vertices, not +one. This is a legitimate limitation of a schema-only pipeline — solving +it needs either LLM alignment (system prompt: "always emit shortest +canonical name") or a real entity-resolution step (e.g., string +similarity + external gazetteer). Neither is in scope for Issue #74. + +**Failure Mode 2 — LLM extracts films the GT does not verify.** The +extract mentions ~15 films; only ~7 pass Wikidata's `P161` filter (rest +are false-positive disambiguations, uncredited appearances, or films +Wikidata simply hasn't tagged). The LLM extracts all ~15; both +strategies pass ~15 through. This inflates predicted count and tanks +precision. Enhanced can filter schema violations but cannot fact-check +against Wikidata. + +**Where enhanced still helps despite these two problems.** + +*Tom Hanks (baseline F1 0.052 → enhanced 0.327):* The baseline +extraction diverged on chunk 3 (TV work) and lost most of the GT films. +Enhanced's constraint block appears to have kept the LLM focused on +film-instance extraction across all chunks, so more of the correct films +made it through. This is the constraint block's largest observed win — +a probabilistic effect via LLM cooperation, not a hard guarantee. + +*Anthony Hopkins (baseline F1 0.074 → enhanced 0.233):* Similar pattern. +Baseline predicted many long-form names + role names as Persons ("Sir +Anthony Philip Hopkins", "King Lear", etc.); enhanced kept the number of +Persons lower and the film precision higher. + +**Why enhanced regresses on Julia Roberts / Denzel Washington / Morgan +Freeman.** These are the corpora where baseline was already relatively +strong (0.49–0.68 F1). On those, enhanced's schema-guided constraint +block did not add much information (baseline was already extracting +mostly clean output), but enhanced's stricter deduplication and cross- +chunk alias union sometimes *merged* films that had slight title +variations across chunks (e.g. `Ocean's Eleven` vs `Ocean's 11`), +knocking F1 down. This is the classic ceiling effect: safety-net +strategies help when the baseline fails, but can hurt when the baseline +is already good. + +### Cost / quality trade-off + +Enhanced is **not** cost-neutral: it sends +40 % prompt tokens (the +constraint block). But it produces **−24 % completion tokens** because +the LLM is guided to output tighter JSON. Net cost per document is +**−7.1 %** and net latency is **−10.7 %**. + +The trade-off is favourable when quality is the bottleneck (worst-case +corpora) and roughly neutral or slightly worse when baseline is already +good. A production deployment could route to enhanced conditionally +based on a first-pass quality-gate signal, but that adds complexity; +running enhanced unconditionally is defensible given the small negative +side. + +### Reproducing the live benchmark + +See the top-level [How to Reproduce This Report](#how-to-reproduce-this-report) +section for the `build_public_actor_corpus.py` and +`graph_extract_live_benchmark.py` invocations, plus the sha256 for +verifying an archive obtained on request. The archive (~600 KB, 48-run +record including every LLM call's prompt/completion tokens, latency, +and raw predicted output) makes every number in this section +independently recomputable without re-running the benchmark. + +## Metric Definitions + +* **Precision** = |predicted ∩ expected| / |predicted_unique|, matched by + `(label, id)` for vertices and `(label, outV, inV)` for edges. Empty + predicted → 1.0. +* **Recall** = |predicted ∩ expected| / |expected|. Empty expected → 1.0. +* **F1** = harmonic mean of precision and recall. Both empty → 1.0. +* **Overall F1** = combined vertex + edge precision/recall. +* **Property valid ratio** = fraction of predicted `(label, key)` pairs + whose key is declared on that label in the schema. +* **Property exact match rate** = fraction of expected properties on TP + items whose predicted `(key, value)` matches exactly (Python `==`). + +## Scope and Caveats + +* **The mock benchmark is not effect evidence.** Its 16 scenarios probe + pipeline correctness, not end-to-end LLM quality. Its +11.6 % gain is + a lower-bound consistency check for the pipeline. The live benchmark + is the effect-evidence track. + +* **The live benchmark's Person canonical id is a real limit** neither + strategy overcomes. Wikipedia openings use "Thomas Jeffrey Hanks"; + Wikidata canonicals use "Tom Hanks". Both strategies fail on the + Person axis in most corpora, dragging edge F1 with them. A future + entity-resolution step could plausibly recover the majority of this + gap; it is out of scope for Issue #74. + +* **Live-corpus GT is strict.** Only films whose Wikidata entity + passes both instance-of-film and cast-member-verified is admitted. + The LLM extracts many *real* films that Wikidata hasn't tagged as + cast-linked (recent releases, uncredited work); these are counted as + false positives even though they are factually correct. This + systematically depresses precision. It is a limit of Wikidata's + completeness, not the extraction pipeline's; live F1 is a lower bound + on true quality. + +* **8 corpora × 3 runs = 48 samples** is a modest sample size. The F1 + gap of +0.020 is not statistically dramatic (baseline std 0.251; + enhanced std 0.188). The stronger claims are on *variance reduction* + (−25 %), *worst-case improvement* (0.049 → 0.216), *latency* (−11 %), + and *cost* (−7 %) — all of which are outside the F1 mean's confidence + band. Larger corpora would tighten the mean's confidence. + +* **Live latency is DeepSeek-side, not enhanced-pipeline-side.** + Enhanced's post-LLM processing adds < 100 ms per chunk (see mock + latency column); the observed −11 % wall-clock savings come from the + LLM decoding fewer completion tokens. + +* **Wikidata WDQS outage during data collection (2026-07).** SPARQL was + 1-req/min throttled at time of writing; the corpus builder uses the + entity API instead, which was less affected but still needed 5 s + inter-request pacing and generous 429 backoff. + +* **Deterministic FakeLLM in mock.** The mock benchmark's LLM + invocations are dict pops, so the mock's millisecond figures reflect + post-processing overhead only. The live section's wall-clock latency + is the honest end-to-end number. + +* **Ground truth uses canonical ids.** When a schema entry omits `id` + (as inline user schemas often do), neither strategy can compute + canonical ids and the evaluator falls back to `(label, raw_id)` for + both, keeping the comparison fair. diff --git a/docs/quality/schema-based-graph-extract-usage.md b/docs/quality/schema-based-graph-extract-usage.md new file mode 100644 index 000000000..3a2b33fc0 --- /dev/null +++ b/docs/quality/schema-based-graph-extract-usage.md @@ -0,0 +1,208 @@ +# Schema-based Graph Extraction — Usage Guide + +The enhanced graph extraction strategy is an **opt-in** upgrade to the +`/graph/extract` API and the `GraphExtractFlow` pipeline. This document +covers how to enable it, when it helps, and its known limitations. + +## Quickstart + +### HTTP API + +Existing callers keep working unchanged (baseline is still the default): + +```bash +curl -X POST http://localhost:8001/graph/extract \ + -H "Content-Type: application/json" \ + -d '{ + "texts": ["Tom Hanks starred in Forrest Gump."], + "graph_schema": "movie_graph" + }' +``` + +To enable the enhanced strategy, add `extract_strategy: "enhanced"`: + +```bash +curl -X POST http://localhost:8001/graph/extract \ + -H "Content-Type: application/json" \ + -d '{ + "texts": ["Tom Hanks starred in Forrest Gump."], + "graph_schema": "movie_graph", + "extract_strategy": "enhanced", + "include_meta": true + }' +``` + +Enhanced-only fields in the response `meta` block: + +| Field | Type | Description | +| --- | --- | --- | +| `extract_strategy` | `"enhanced"` | Echoes the strategy in effect. Absent under baseline. | +| `chunk_count` | int | Number of chunks the input was split into. | +| `call_count` | int | Number of LLM invocations (== chunk_count). | +| `token_usage` | str | Placeholder `"unavailable"` until the LLM adapter surfaces token metadata. | +| `structured_warnings` | list | Machine-readable warning records — see below. | +| `quality_metrics` | dict | Per-run quality aggregate — see below. | + +Passing `include_debug: true` additionally surfaces `meta.debug_info` +with per-chunk raw LLM output (truncated to 2 KB) plus normalized item +counts. Use this in staging for prompt tuning; keep it off in production +to avoid response bloat. + +### Python SDK + +```python +from hugegraph_llm.flows import FlowName +from hugegraph_llm.flows.scheduler import SchedulerSingleton + +result = SchedulerSingleton.get_instance().schedule_flow( + FlowName.GRAPH_EXTRACT, + schema=my_schema, + texts=["…"], + example_prompt="…", + extract_type="…", + extract_strategy="enhanced", # opt-in + include_debug=False, # optional, default False +) +``` + +The `result` string is a JSON payload compatible with the baseline +schema; enhanced-only fields appear as additional top-level keys. + +## Structured Warnings + +Each warning has this shape: + +```json +{ + "code": "MISSING_PRIMARY_KEY", + "item_type": "vertex", + "reason": "vertex 'Person' has no value for primary key 'name'", + "chunk_id": 0 +} +``` + +Warning codes are grouped by their producer stage (parser, normalizer, +assembler, quality gate). Full list in +[`property_graph_extract_enhanced/warnings.py`](../../hugegraph-llm/src/hugegraph_llm/operators/llm_op/property_graph_extract_enhanced/warnings.py). + +Common codes: + +* `MISSING_PRIMARY_KEY` — vertex is dropped because a PK value is missing. +* `PROPERTY_COERCED` — a property value was converted to its schema type + (e.g. `"62"` → `62`). Non-fatal. +* `UNKNOWN_VERTEX_LABEL` / `UNKNOWN_EDGE_LABEL` — item dropped for using a + label not in the schema. +* `ENDPOINT_INCOMPATIBLE` — edge direction violates the schema + `source_label` / `target_label`. +* `PENDING_ENDPOINT_UNRESOLVED` — edge references a vertex that no chunk + ever defined. +* `JSON_DECODE_FAILED` — the LLM's chunk output could not be parsed. + +## Quality Metrics + +Each enhanced run reports a `quality_metrics` block with 11 fields +including `parse_success_rate`, `endpoint_repair_rate`, +`property_coerce_rate`, `dropped_item_count`, and +`warning_code_distribution`. Full field list in +[`property_graph_extract_enhanced/quality_gate.py`](../../hugegraph-llm/src/hugegraph_llm/operators/llm_op/property_graph_extract_enhanced/quality_gate.py). + +Use these as live signals in staging: a sudden spike in +`property_coerce_rate` usually means the prompt drifted and started +emitting stringified numerics; a spike in `endpoint_repair_rate` means +chunking may be too aggressive. + +## Applicable Scenarios + +The enhanced strategy helps most when **any** of these hold: + +* **Multi-chunk documents.** The strategy's `DocumentGraphAssembler` + merges vertices across chunks and repairs cross-chunk edge endpoints — + the biggest F1 win in the benchmark comes from here. +* **Strict schema with canonical id requirements.** If HugeGraph's + `PRIMARY_KEY` id strategy is in use, enhanced ensures every emitted + vertex has a valid canonical id and drops PK-missing candidates before + they reach the commit stage. +* **Typed properties beyond TEXT.** INT/LONG/FLOAT/BOOLEAN/DATE columns + benefit from enhanced's best-effort coercion. Baseline's `filter_item` + is key-only and does not repair types. +* **Prompts under drift.** Structured warnings + quality metrics give + operators a signal to detect when a previously-good prompt starts + regressing. + +Baseline remains a fine default when: + +* Every schema property is TEXT. +* Documents fit in a single chunk. +* No canonical id is needed downstream (e.g., `id_strategy = "AUTOMATIC"`). +* You want to minimize LLM prompt tokens (see limitation below). + +## Known Limitations + +* **Prompt-token overhead.** The enhanced strategy appends a + `constraint_block` listing every schema label to the LLM prompt on + every chunk (~500-800 extra tokens for a small schema, more for large + ones). This is real cost on paid APIs. For a 100-chunk document this + is 50k-80k extra tokens; at DeepSeek Chat pricing (2026-07 rates) that + is single-digit cents but non-zero. + +* **Coercion is best-effort, not lossless.** `"1994"` → `1994` (INT) is + safe. `"1.5"` → `1` (INT) drops information — enhanced emits + `PROPERTY_COERCED` with the reason string. Review warnings before + concluding the graph is clean. + +* **No cross-request state.** The alias table is scoped to a single + request. Two entities with the same schema-canonical id extracted from + different `/graph/extract` calls still land as two records — this is + the storage layer's job, not the extractor's. + +* **Set-based F1.** The offline evaluator scores against deduplicated + predictions. Baseline's raw-count duplication cost does not surface in + F1; if you care about commit load, read the raw counts from the + benchmark's per-scenario `predicted_count_raw`. + +* **No streaming.** Enhanced processes all chunks synchronously and + produces the assembled graph at the end. For very large documents, + consider chunking upstream and calling `/graph/extract` per batch. + +* **No knowledge-base entity resolution.** Enhanced can resolve aliases + *within* a single request (e.g. `"He"` → `"Tom Hanks"` when both + appear in different chunks) via its `DocumentGraphAssembler`. It + cannot resolve *out-of-corpus* aliases: if the LLM extracts + `"Thomas Jeffrey Hanks"` from an article's opening line and the + downstream canonical id is `"Tom Hanks"`, both records land as + separate vertices. Solving this needs either LLM-side alignment + (system prompt telling the model to emit shortest canonical names) or + an external entity-resolution step (string similarity + gazetteer / + Wikidata / customer-supplied alias table). Both are out of scope for + Issue #74. See the effect report's live-benchmark failure analysis + for concrete numbers. + +## When enhanced does **not** help + +The live benchmark on 8 public Wikipedia corpora showed enhanced +regressing F1 on 3 of the 8 corpora (baseline already strong; enhanced's +stricter deduplication merged near-duplicate film titles that were +actually different films). Consider running baseline first and switching +to enhanced only if `warnings` count is high or if quality is +consistently under target. A production deployment can gate this on +first-pass baseline warning counts. + +## Migration Notes + +None required. Existing baseline callers keep byte-compatible responses +as long as they do not send `extract_strategy` or `include_debug`. + +If you are the first team to opt-in on a graph, run one enhanced request +with `include_meta=true` and inspect `meta.structured_warnings` / +`meta.quality_metrics` — the warning codes indicate any places where the +prompt is drifting from the schema, and you should tighten those before +switching all traffic. + +## References + +* [Effect Report](./schema-based-graph-extract-report.md) — quality and + latency benchmark numbers. +* [Test Taxonomy](./test-taxonomy.md) — where the benchmark fits in the + overall test pyramid. +* Source: + [`hugegraph_llm/operators/llm_op/property_graph_extract_enhanced/`](../../hugegraph-llm/src/hugegraph_llm/operators/llm_op/property_graph_extract_enhanced/) diff --git a/hugegraph-llm/src/hugegraph_llm/api/graph_extract_api.py b/hugegraph-llm/src/hugegraph_llm/api/graph_extract_api.py index 6884412d7..565affd95 100644 --- a/hugegraph-llm/src/hugegraph_llm/api/graph_extract_api.py +++ b/hugegraph-llm/src/hugegraph_llm/api/graph_extract_api.py @@ -41,10 +41,28 @@ def extract_sync(req: GraphExtractRequest) -> GraphExtractResponse: language=req.language, split_type=req.split_type, client_config=req.client_config, + extract_strategy=req.extract_strategy, + include_debug=req.include_debug, ) raw = json.loads(result_str) warnings = [raw.pop("warning")] if "warning" in raw else [] + # Enhanced-strategy fields — the flow layer only emits these when + # extract_strategy == "enhanced", so their absence signals baseline. + extract_strategy = raw.pop("extract_strategy", "baseline") + chunk_count = raw.pop("chunk_count", None) + call_count = raw.pop("call_count", None) + structured_warnings = raw.pop("structured_warnings", None) + quality_metrics = raw.pop("quality_metrics", None) + debug_info = raw.pop("debug_info", None) + result = {"vertices": raw.get("vertices", []), "edges": raw.get("edges", [])} + + # Surface a short top-level summary of the structured warning count + # so callers reading the legacy warnings[] can still notice enhanced + # activity without parsing meta. + if structured_warnings: + warnings.append(f"enhanced graph extraction generated {len(structured_warnings)} structured warning(s)") + meta = {} if req.include_meta: meta = { @@ -52,6 +70,21 @@ def extract_sync(req: GraphExtractRequest) -> GraphExtractResponse: "edge_count": len(result["edges"]), "text_count": len(req.texts), } + if extract_strategy == "enhanced": + meta["extract_strategy"] = extract_strategy + if chunk_count is not None: + meta["chunk_count"] = chunk_count + if call_count is not None: + meta["call_count"] = call_count + meta["token_usage"] = "unavailable" + if structured_warnings is not None: + meta["structured_warnings"] = structured_warnings + if quality_metrics is not None: + meta["quality_metrics"] = quality_metrics + # include_debug adds debug_info regardless of include_meta so callers + # can grab the debug payload without opting in to counts. + if req.include_debug and debug_info is not None: + meta["debug_info"] = debug_info return GraphExtractResponse(result=result, warnings=warnings, meta=meta) except HTTPException: raise diff --git a/hugegraph-llm/src/hugegraph_llm/api/models/graph_extract_requests.py b/hugegraph-llm/src/hugegraph_llm/api/models/graph_extract_requests.py index d3e2654a4..b33c9f3ba 100644 --- a/hugegraph-llm/src/hugegraph_llm/api/models/graph_extract_requests.py +++ b/hugegraph-llm/src/hugegraph_llm/api/models/graph_extract_requests.py @@ -45,6 +45,20 @@ class GraphExtractRequest(BaseModel): language: Literal["zh", "en"] = Query("zh", description="Language for chunk splitting.") split_type: Literal["document", "paragraph", "sentence"] = Query("document", description="Chunk split granularity.") include_meta: bool = Query(False, description="Include vertex/edge/text counts in the response.") + extract_strategy: Literal["baseline", "enhanced"] = Query( + "baseline", + description=( + "Which extraction strategy to use. 'baseline' preserves the existing behavior; " + "'enhanced' opts into the schema-aware graph quality layer." + ), + ) + include_debug: bool = Query( + False, + description=( + "When true, the enhanced strategy attaches per-chunk debug details under 'meta.debug_info'. " + "Ignored by the baseline strategy." + ), + ) client_config: Optional[GraphExtractClientConfig] = Field(None, description="Request-scoped HugeGraph connection.") @field_validator("texts") diff --git a/hugegraph-llm/src/hugegraph_llm/flows/graph_extract.py b/hugegraph-llm/src/hugegraph_llm/flows/graph_extract.py index 4c96434a6..989b29afa 100644 --- a/hugegraph-llm/src/hugegraph_llm/flows/graph_extract.py +++ b/hugegraph-llm/src/hugegraph_llm/flows/graph_extract.py @@ -14,6 +14,7 @@ # limitations under the License. import json +from typing import Any, Dict from pycgraph import GPipeline @@ -56,6 +57,9 @@ def prepare( prepared_input.example_prompt = example_prompt prepared_input.schema = schema prepared_input.extract_type = extract_type + # Enhanced graph extraction opt-in flags (baseline behavior stays unchanged when defaults hold). + prepared_input.extract_strategy = kwargs.get("extract_strategy", "baseline") + prepared_input.include_debug = bool(kwargs.get("include_debug", False)) client_config = kwargs.get("client_config") if client_config: # URL stays server-controlled; only identity/graphspace are request-scoped. @@ -109,20 +113,30 @@ def post_deal(self, pipeline=None, **kwargs): vertices = res.get("vertices", []) edges = res.get("edges", []) chunk_count = len(res.get("chunks", [])) + extract_strategy = res.get("extract_strategy", "baseline") log.info("Graph extraction chunk_count: %s", chunk_count) + if extract_strategy == "enhanced": + log.info("Enhanced graph extraction strategy in use.") + payload: Dict[str, Any] = {"vertices": vertices, "edges": edges} + # Enhanced-strategy fields flow to the API layer, which decides which + # of them to surface under `meta` based on include_meta / include_debug. + # Baseline stays byte-compatible: none of these keys appear. + if extract_strategy == "enhanced": + payload["extract_strategy"] = extract_strategy + payload["chunk_count"] = chunk_count + call_count = res.get("call_count") + if call_count is not None: + payload["call_count"] = call_count + structured_warnings = res.get("structured_warnings") + if structured_warnings is not None: + payload["structured_warnings"] = structured_warnings + quality_metrics = res.get("quality_metrics") + if quality_metrics is not None: + payload["quality_metrics"] = quality_metrics + debug_info = res.get("debug_info") + if debug_info is not None: + payload["debug_info"] = debug_info if not vertices and not edges: log.info("Please check the schema.(The schema may not match the Doc)") - return json.dumps( - { - "vertices": vertices, - "edges": edges, - "warning": "The schema may not match the Doc", - }, - ensure_ascii=False, - indent=2, - ) - return json.dumps( - {"vertices": vertices, "edges": edges}, - ensure_ascii=False, - indent=2, - ) + payload["warning"] = "The schema may not match the Doc" + return json.dumps(payload, ensure_ascii=False, indent=2) diff --git a/hugegraph-llm/src/hugegraph_llm/nodes/llm_node/extract_info.py b/hugegraph-llm/src/hugegraph_llm/nodes/llm_node/extract_info.py index 9ac26970c..a5341e15b 100644 --- a/hugegraph-llm/src/hugegraph_llm/nodes/llm_node/extract_info.py +++ b/hugegraph-llm/src/hugegraph_llm/nodes/llm_node/extract_info.py @@ -38,10 +38,17 @@ def node_init(self): example_prompt = self.wk_input.example_prompt extract_type = self.wk_input.extract_type self.extract_type = extract_type + extract_strategy = self.wk_input.extract_strategy or "baseline" + include_debug = bool(self.wk_input.include_debug) if extract_type == "triples": self.info_extract = InfoExtract(llm, example_prompt) elif extract_type == "property_graph": - self.property_graph_extract = PropertyGraphExtract(llm, example_prompt) + self.property_graph_extract = PropertyGraphExtract( + llm, + example_prompt, + extract_strategy=extract_strategy, + include_debug=include_debug, + ) else: return CStatus(-1, f"Unsupported extract_type: {extract_type}") return super().node_init() diff --git a/hugegraph-llm/src/hugegraph_llm/operators/llm_op/property_graph_extract.py b/hugegraph-llm/src/hugegraph_llm/operators/llm_op/property_graph_extract.py index 3e3974746..956725fd3 100644 --- a/hugegraph-llm/src/hugegraph_llm/operators/llm_op/property_graph_extract.py +++ b/hugegraph-llm/src/hugegraph_llm/operators/llm_op/property_graph_extract.py @@ -19,11 +19,19 @@ import json import re -from typing import Any, Dict, List +from typing import Any, Dict, List, Mapping from hugegraph_llm.config import prompt from hugegraph_llm.document.chunk_split import ChunkSplitter from hugegraph_llm.models.llms.base import BaseLLM +from hugegraph_llm.operators.llm_op.property_graph_extract_enhanced import ( + CandidateGraphParser, + DocumentGraphAssembler, + GraphQualityGate, + GraphSchemaIndex, + SchemaAwareNormalizer, + build_prompt_contract, +) from hugegraph_llm.utils.log import log # TODO: It is not clear whether there is any other dependence on the SCHEMA_EXAMPLE_PROMPT variable. @@ -31,6 +39,18 @@ # prompt.extract_graph_prompt changes after the system loads, this does not seem to meet expectations. SCHEMA_EXAMPLE_PROMPT = prompt.extract_graph_prompt +# Enhanced-strategy debug payload keeps the raw LLM output trimmed to a bounded +# size so the API response cannot grow unbounded on large chunks. +_DEBUG_RAW_OUTPUT_LIMIT = 2000 + + +def _truncate(text: str, limit: int) -> str: + if text is None: + return "" + if len(text) <= limit: + return text + return text[:limit] + f"... [truncated {len(text) - limit} chars]" + def generate_extract_property_graph_prompt(text, schema=None) -> str: return f"""--- @@ -78,9 +98,19 @@ def filter_item(schema, items) -> List[Dict[str, Any]]: class PropertyGraphExtract: - def __init__(self, llm: BaseLLM, example_prompt: str = prompt.extract_graph_prompt) -> None: + def __init__( + self, + llm: BaseLLM, + example_prompt: str = prompt.extract_graph_prompt, + extract_strategy: str = "baseline", + include_debug: bool = False, + ) -> None: self.llm = llm self.example_prompt = example_prompt + # extract_strategy is captured now; the schema-aware quality layer wired in a later + # commit inspects it to branch behavior. Baseline is byte-compatible with the pre-flag path. + self.extract_strategy = extract_strategy if extract_strategy in ("baseline", "enhanced") else "baseline" + self.include_debug = bool(include_debug) self.NECESSARY_ITEM_KEYS = {"label", "type", "properties"} # pylint: disable=invalid-name def run(self, context: Dict[str, Any]) -> Dict[str, List[Any]]: @@ -90,6 +120,13 @@ def run(self, context: Dict[str, Any]) -> Dict[str, List[Any]]: context["vertices"] = [] if "edges" not in context: context["edges"] = [] + # Dispatch to the schema-aware quality layer when explicitly requested. + # Baseline stays byte-compatible; the enhanced path never runs unless + # the caller opted in via extract_strategy="enhanced". + if self.extract_strategy == "enhanced": + return self._run_enhanced(context, schema, chunks) + # Mirror the resolved strategy into the shared context so downstream API/meta layers see it. + context["extract_strategy"] = self.extract_strategy items = [] for chunk in chunks: proceeded_chunk = self.extract_property_graph_by_llm(schema, chunk) @@ -110,6 +147,96 @@ def run(self, context: Dict[str, Any]) -> Dict[str, List[Any]]: context["call_count"] = context.get("call_count", 0) + len(chunks) return context + # ------------------------------------------------------- enhanced strategy + def _run_enhanced( + self, + context: Dict[str, Any], + schema: Any, + chunks: List[str], + ) -> Dict[str, List[Any]]: + """Enhanced extraction: LLM → parser → normalizer → assembler → quality gate. + + The schema is threaded through the pipeline twice: once as text embedded + in the LLM prompt (via ``generate_extract_property_graph_prompt``), and + once as a compiled ``GraphSchemaIndex`` used by every post-LLM stage. + Warnings from every stage accumulate into a single list surfaced via + ``context["structured_warnings"]``; the assembled document graph plus + the quality-gate aggregate populate the remaining meta fields. + """ + schema_dict = schema if isinstance(schema, Mapping) else json.loads(schema) + schema_index = GraphSchemaIndex(schema_dict) + + parser = CandidateGraphParser() + normalizer = SchemaAwareNormalizer(schema_index) + assembler = DocumentGraphAssembler(schema_index) + constraint_block = build_prompt_contract(schema_index) + + warnings = [] + chunk_graphs = [] + candidate_vertex_total = 0 + candidate_edge_total = 0 + debug_records: List[Dict[str, Any]] = [] + + for chunk_id, chunk in enumerate(chunks): + raw = self._extract_property_graph_by_llm_enhanced(schema, chunk, constraint_block) + log.debug( + "[LLM enhanced] %s chunk_id=%s output: %s", + self.__class__.__name__, + chunk_id, + raw, + ) + candidate, parse_warnings = parser.parse(raw, chunk_id=chunk_id) + warnings.extend(parse_warnings) + candidate_vertex_total += len(candidate.vertices) + candidate_edge_total += len(candidate.edges) + + normalized, norm_warnings = normalizer.normalize(candidate, chunk_id=chunk_id) + warnings.extend(norm_warnings) + chunk_graphs.append(normalized) + + if self.include_debug: + debug_records.append( + { + "chunk_id": chunk_id, + "raw_output": _truncate(raw, _DEBUG_RAW_OUTPUT_LIMIT), + "candidate_vertex_count": len(candidate.vertices), + "candidate_edge_count": len(candidate.edges), + "normalized_vertex_count": len(normalized.vertices), + "normalized_edge_count": len(normalized.edges), + "chunk_warning_count": len(parse_warnings) + len(norm_warnings), + } + ) + + doc_graph, asm_warnings = assembler.assemble(chunk_graphs) + warnings.extend(asm_warnings) + + metrics = GraphQualityGate.compute( + doc_graph, + warnings=warnings, + candidate_vertex_count=candidate_vertex_total, + candidate_edge_count=candidate_edge_total, + ) + + context["vertices"] = doc_graph.vertices + context["edges"] = doc_graph.edges + context["call_count"] = context.get("call_count", 0) + len(chunks) + context["extract_strategy"] = "enhanced" + context["chunk_count"] = len(chunks) + context["structured_warnings"] = [w.to_dict() for w in warnings] + context["quality_metrics"] = metrics.to_dict() + if self.include_debug: + context["debug_info"] = { + "chunks": debug_records, + "warning_code_distribution": metrics.warning_code_distribution, + } + return context + + def _extract_property_graph_by_llm_enhanced(self, schema: Any, chunk: str, constraint_block: str) -> str: + """Same LLM call as baseline, with the schema-aware constraint block appended.""" + prompt_body = generate_extract_property_graph_prompt(chunk, schema) + header = (self.example_prompt or "") + constraint_block + return self.llm.generate(prompt=header + prompt_body) + def extract_property_graph_by_llm(self, schema, chunk): prompt = generate_extract_property_graph_prompt(chunk, schema) if self.example_prompt is not None: diff --git a/hugegraph-llm/src/hugegraph_llm/operators/llm_op/property_graph_extract_enhanced/__init__.py b/hugegraph-llm/src/hugegraph_llm/operators/llm_op/property_graph_extract_enhanced/__init__.py new file mode 100644 index 000000000..3cb94535c --- /dev/null +++ b/hugegraph-llm/src/hugegraph_llm/operators/llm_op/property_graph_extract_enhanced/__init__.py @@ -0,0 +1,86 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +"""Schema-aware graph extraction quality layer. + +The enhanced extraction strategy compiles the request-provided graph schema into a +runtime constraint index (``GraphSchemaIndex``) and layers a candidate parser, +schema-aware normalizer, document-level assembler, and quality gate on top of the +baseline extraction pipeline. All modules in this package are pure Python and +require no external services; they operate only on data already handed in by +existing baseline nodes (schema, chunks, LLM outputs). +""" + +from hugegraph_llm.operators.llm_op.property_graph_extract_enhanced.candidate_parser import ( + CandidateGraphParser, +) +from hugegraph_llm.operators.llm_op.property_graph_extract_enhanced.document_assembler import ( + DocumentGraphAssembler, +) +from hugegraph_llm.operators.llm_op.property_graph_extract_enhanced.evaluator import ( + EvaluationReport, + GraphExtractionEvaluator, + ItemMetrics, + PropertyMetrics, +) +from hugegraph_llm.operators.llm_op.property_graph_extract_enhanced.normalizer import ( + SchemaAwareNormalizer, +) +from hugegraph_llm.operators.llm_op.property_graph_extract_enhanced.prompt_contract import ( + build_prompt_contract, +) +from hugegraph_llm.operators.llm_op.property_graph_extract_enhanced.quality_gate import ( + GraphQualityGate, + QualityMetrics, +) +from hugegraph_llm.operators.llm_op.property_graph_extract_enhanced.schema_index import ( + GraphSchemaIndex, +) +from hugegraph_llm.operators.llm_op.property_graph_extract_enhanced.types import ( + PENDING_IN_KEY, + PENDING_OUT_KEY, + CandidateGraph, + DocumentGraph, + NormalizedChunkGraph, +) +from hugegraph_llm.operators.llm_op.property_graph_extract_enhanced.warnings import ( + StructuredWarning, + WarningCode, + warning_code_distribution, +) + +__all__ = [ + "PENDING_IN_KEY", + "PENDING_OUT_KEY", + "CandidateGraph", + "CandidateGraphParser", + "DocumentGraph", + "DocumentGraphAssembler", + "EvaluationReport", + "GraphExtractionEvaluator", + "GraphQualityGate", + "GraphSchemaIndex", + "ItemMetrics", + "NormalizedChunkGraph", + "PropertyMetrics", + "QualityMetrics", + "SchemaAwareNormalizer", + "StructuredWarning", + "WarningCode", + "build_prompt_contract", + "warning_code_distribution", +] diff --git a/hugegraph-llm/src/hugegraph_llm/operators/llm_op/property_graph_extract_enhanced/candidate_parser.py b/hugegraph-llm/src/hugegraph_llm/operators/llm_op/property_graph_extract_enhanced/candidate_parser.py new file mode 100644 index 000000000..4f7802fad --- /dev/null +++ b/hugegraph-llm/src/hugegraph_llm/operators/llm_op/property_graph_extract_enhanced/candidate_parser.py @@ -0,0 +1,312 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +"""Candidate graph parser for the enhanced graph extraction strategy. + +Reads a single chunk's raw LLM output (a string) and produces a +``CandidateGraph`` plus a list of ``StructuredWarning``s describing every +recoverable or unrecoverable defect encountered along the way. Downstream +stages (normalizer, assembler) rely on the parser to have already: + +* stripped Markdown code fences, +* located and parsed a JSON payload, +* separated grouped ``{"vertices": [...], "edges": [...]}`` and flat-array + ``[{"type": "vertex", ...}, ...]`` shapes into two arrays, +* dropped items that are not dicts or that carry an explicit ``type`` in + conflict with their containing array. + +The parser does NOT perform any schema validation. That is the normalizer's +responsibility. This keeps parser failure modes JSON-shaped rather than +schema-shaped and makes the warning surface easier to reason about. +""" + +from __future__ import annotations + +import json +import re +from typing import Any, Dict, List, Optional, Tuple + +from hugegraph_llm.operators.llm_op.property_graph_extract_enhanced.types import CandidateGraph +from hugegraph_llm.operators.llm_op.property_graph_extract_enhanced.warnings import ( + StructuredWarning, + WarningCode, +) + +_FENCE_TAG_RE = re.compile(r"```[^\n]*\n?", re.IGNORECASE) +_FENCE_CLOSE_RE = re.compile(r"```") +# Greedy match from the first '{' to the last '}' (or first '[' to last ']'). +# The greedy .* in DOTALL mode lets us pick up JSON that spans newlines and is +# wrapped in prose. If the extracted substring is not valid JSON the caller +# emits JSON_DECODE_FAILED instead of hallucinating a smaller match. +_JSON_OBJECT_RE = re.compile(r"\{.*\}", re.DOTALL) +_JSON_ARRAY_RE = re.compile(r"\[.*\]", re.DOTALL) + + +class CandidateGraphParser: + """Turn raw LLM output into a ``CandidateGraph`` with structured warnings. + + The parser is stateless — a single instance can be reused across chunks + and threads. Pass ``chunk_id`` on each call so emitted warnings can be + correlated back to the chunk they came from. + """ + + def parse( + self, + raw_text: str, + *, + chunk_id: Optional[int] = None, + ) -> Tuple[CandidateGraph, List[StructuredWarning]]: + """Parse a single chunk's raw LLM output. + + Returns: + A ``(CandidateGraph, warnings)`` tuple. On complete parse failure, + the returned graph has empty vertices and edges lists. + """ + warnings: List[StructuredWarning] = [] + text = self._strip_fences(raw_text or "") + + payload, decode_warning = self._locate_and_parse_json(text, chunk_id=chunk_id) + if decode_warning is not None: + warnings.append(decode_warning) + if payload is None: + return CandidateGraph(), warnings + + vertices_section, edges_section, section_warnings = self._extract_sections(payload, chunk_id=chunk_id) + warnings.extend(section_warnings) + + vertices, v_warnings = self._collect_items(vertices_section, expected_type="vertex", chunk_id=chunk_id) + edges, e_warnings = self._collect_items(edges_section, expected_type="edge", chunk_id=chunk_id) + warnings.extend(v_warnings) + warnings.extend(e_warnings) + + return CandidateGraph(vertices=vertices, edges=edges), warnings + + # ---------------------------------------------------------------- steps + @staticmethod + def _strip_fences(text: str) -> str: + """Remove Markdown code fences (```, ```json, ```JSON) from the text.""" + stripped = _FENCE_TAG_RE.sub("", text) + stripped = _FENCE_CLOSE_RE.sub("", stripped) + return stripped.strip() + + @staticmethod + def _locate_and_parse_json( + text: str, *, chunk_id: Optional[int] + ) -> Tuple[Optional[Any], Optional[StructuredWarning]]: + """Best-effort JSON extraction. + + Order of attempts: + + 1. ``json.loads`` of the entire (stripped) text — succeeds when the + LLM emits well-formed JSON with no surrounding prose. + 2. Greedy ``{...}`` extraction — for grouped format wrapped in prose. + 3. Greedy ``[...]`` extraction — for flat-array format wrapped in prose. + + Returns ``(payload, None)`` on success, ``(None, warning)`` on failure. + A failure is classified as ``JSON_DECODE_FAILED`` whenever the text + contains an opening brace or bracket (evidence that JSON was attempted + but is malformed), and ``JSON_NOT_FOUND`` otherwise. + """ + if not text: + return None, StructuredWarning( + code=WarningCode.JSON_NOT_FOUND, + item_type="graph", + reason="LLM output is empty after stripping code fences", + chunk_id=chunk_id, + ) + + try: + return json.loads(text), None + except json.JSONDecodeError as full_exc: + full_error = str(full_exc) + + obj_match = _JSON_OBJECT_RE.search(text) + arr_match = _JSON_ARRAY_RE.search(text) + + candidates: List[str] = [] + if obj_match: + candidates.append(obj_match.group(0)) + if arr_match: + candidates.append(arr_match.group(0)) + + last_error: Optional[str] = full_error + for candidate in candidates: + try: + return json.loads(candidate), None + except json.JSONDecodeError as exc: + last_error = str(exc) + + # Discriminate: text with an opening brace/bracket is a broken JSON + # attempt; text without either bracket really has no JSON at all. + contains_json_start = ("{" in text) or ("[" in text) + if contains_json_start: + return None, StructuredWarning( + code=WarningCode.JSON_DECODE_FAILED, + item_type="graph", + reason=f"located a JSON-like substring but failed to decode: {last_error}", + chunk_id=chunk_id, + ) + return None, StructuredWarning( + code=WarningCode.JSON_NOT_FOUND, + item_type="graph", + reason="no JSON object or array found in the LLM output", + chunk_id=chunk_id, + ) + + @staticmethod + def _extract_sections( + payload: Any, *, chunk_id: Optional[int] + ) -> Tuple[List[Any], List[Any], List[StructuredWarning]]: + """Route a parsed JSON payload into (vertices_section, edges_section). + + Handles three top-level shapes: + + * ``list`` — flat array; items are partitioned by their ``type`` field. + Items without ``type`` are dropped later by ``_collect_items`` via + ``ITEM_TYPE_MISMATCH``. + * ``dict`` with ``vertices``/``edges`` keys — grouped format. + * Anything else — treated as empty; both sections yield warnings. + """ + warnings: List[StructuredWarning] = [] + + if isinstance(payload, list): + vertices: List[Any] = [] + edges: List[Any] = [] + for item in payload: + if isinstance(item, dict): + t = item.get("type") + if t == "vertex": + vertices.append(item) + elif t == "edge": + edges.append(item) + else: + # Route once (to vertices) with a marker; the collector + # surfaces ITEM_TYPE_MISMATCH for the missing type. + vertices.append({"__unroutable__": True, **item}) + else: + # Non-dict items are routed to vertices as a single site + # where the collector will log ITEM_NOT_OBJECT. + vertices.append(item) + return vertices, edges, warnings + + if isinstance(payload, dict): + vertices_section = payload.get("vertices") + edges_section = payload.get("edges") + if "vertices" not in payload: + warnings.append( + StructuredWarning( + code=WarningCode.GRAPH_SECTION_MISSING, + item_type="graph", + reason="'vertices' section missing from parsed payload", + chunk_id=chunk_id, + ) + ) + vertices_section = [] + if "edges" not in payload: + warnings.append( + StructuredWarning( + code=WarningCode.GRAPH_SECTION_MISSING, + item_type="graph", + reason="'edges' section missing from parsed payload", + chunk_id=chunk_id, + ) + ) + edges_section = [] + # Normalize non-list section values so the collector still runs. + if not isinstance(vertices_section, list): + vertices_section = [] + if not isinstance(edges_section, list): + edges_section = [] + return vertices_section, edges_section, warnings + + # Payload is a scalar (str/number/bool/None) — surface both missing + # sections; the collector will produce two empty lists. + warnings.append( + StructuredWarning( + code=WarningCode.GRAPH_SECTION_MISSING, + item_type="graph", + reason=f"parsed payload is not an object or array: {type(payload).__name__}", + chunk_id=chunk_id, + ) + ) + return [], [], warnings + + @staticmethod + def _collect_items( + items: List[Any], + *, + expected_type: str, + chunk_id: Optional[int], + ) -> Tuple[List[Dict[str, Any]], List[StructuredWarning]]: + """Validate each candidate item and drop obvious defects. + + Rules: + + * Non-dict → ``ITEM_NOT_OBJECT``. + * Item bore the ``__unroutable__`` marker (from a flat-array item that + had no ``type`` field) → ``ITEM_TYPE_MISMATCH``. + * Item has an explicit ``type`` disagreeing with ``expected_type`` → + ``ITEM_TYPE_MISMATCH``. + * Otherwise the item is kept as-is with its ``type`` field + normalized to ``expected_type`` (this frees the normalizer from + repeating the same defensive check). + """ + collected: List[Dict[str, Any]] = [] + warnings: List[StructuredWarning] = [] + for item in items: + if not isinstance(item, dict): + warnings.append( + StructuredWarning( + code=WarningCode.ITEM_NOT_OBJECT, + item_type="graph", + reason=f"candidate {expected_type} is not an object: {type(item).__name__}", + chunk_id=chunk_id, + ) + ) + continue + + if item.get("__unroutable__") is True: + warnings.append( + StructuredWarning( + code=WarningCode.ITEM_TYPE_MISMATCH, + item_type="graph", + reason="flat-array item has no 'type' field to route it to vertex or edge", + chunk_id=chunk_id, + context={"label": item.get("label")}, + ) + ) + continue + + declared_type = item.get("type") + if declared_type is not None and declared_type != expected_type: + warnings.append( + StructuredWarning( + code=WarningCode.ITEM_TYPE_MISMATCH, + item_type="graph", + reason=(f"candidate has type={declared_type!r} but was placed in the {expected_type} section"), + label=item.get("label"), + chunk_id=chunk_id, + ) + ) + continue + + # Normalize the type for downstream consumers so they can always + # rely on item["type"] being present. + normalized = dict(item) + normalized["type"] = expected_type + collected.append(normalized) + return collected, warnings diff --git a/hugegraph-llm/src/hugegraph_llm/operators/llm_op/property_graph_extract_enhanced/document_assembler.py b/hugegraph-llm/src/hugegraph_llm/operators/llm_op/property_graph_extract_enhanced/document_assembler.py new file mode 100644 index 000000000..e6a00414e --- /dev/null +++ b/hugegraph-llm/src/hugegraph_llm/operators/llm_op/property_graph_extract_enhanced/document_assembler.py @@ -0,0 +1,368 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +"""Document-level assembler for the enhanced graph extraction strategy. + +Takes the list of ``NormalizedChunkGraph`` outputs from every chunk and +produces the final ``DocumentGraph`` plus the warnings that only arise once +we can see the whole document at once: + +* cross-chunk vertex merge by ``(label, id)``, with first-wins property + conflict handling and ``PROPERTY_CONFLICT`` warnings on disagreement; +* endpoint repair for edges the normalizer left pending — the assembler + unions every chunk's alias table into a document-level alias index and + services the ``explicit_id_alias`` repair tier; +* ambiguity detection: an LLM raw id used across chunks for different + canonical vertices yields ``ENDPOINT_AMBIGUOUS`` on the edges that + depended on it; +* edge deduplication by ``(label, outVLabel, outV, inVLabel, inV, + properties_signature)``, preserving first-appearance order. + +The assembler performs no schema I/O and does not modify any chunk graph in +place. Chunks flow through in their given order so the emitted graph is +deterministic given the same LLM outputs. +""" + +from __future__ import annotations + +import json +from typing import Any, Dict, List, Mapping, Optional, Sequence, Set, Tuple + +from hugegraph_llm.operators.llm_op.property_graph_extract_enhanced.schema_index import ( + GraphSchemaIndex, +) +from hugegraph_llm.operators.llm_op.property_graph_extract_enhanced.types import ( + PENDING_IN_KEY, + PENDING_OUT_KEY, + DocumentGraph, + NormalizedChunkGraph, +) +from hugegraph_llm.operators.llm_op.property_graph_extract_enhanced.warnings import ( + StructuredWarning, + WarningCode, +) + + +class DocumentGraphAssembler: + """Assemble chunk-level normalized graphs into a document-level graph.""" + + def __init__(self, schema_index: GraphSchemaIndex) -> None: + self._schema = schema_index + + # ---------------------------------------------------------------- public + def assemble( + self, + chunk_graphs: Sequence[NormalizedChunkGraph], + ) -> Tuple[DocumentGraph, List[StructuredWarning]]: + warnings: List[StructuredWarning] = [] + + doc_aliases, ambiguous_keys = self._union_aliases(chunk_graphs) + + merged_vertices, vertex_warnings, pre_v = self._merge_vertices(chunk_graphs) + warnings.extend(vertex_warnings) + + repaired_edges, edge_warnings, pre_e, repair_count = self._repair_endpoints( + chunk_graphs, doc_aliases, ambiguous_keys + ) + warnings.extend(edge_warnings) + + deduped_edges, dedup_warnings = self._dedupe_edges(repaired_edges) + warnings.extend(dedup_warnings) + + return ( + DocumentGraph( + vertices=merged_vertices, + edges=deduped_edges, + pre_merge_vertex_count=pre_v, + pre_merge_edge_count=pre_e, + endpoint_repair_count=repair_count, + ), + warnings, + ) + + # ---------------------------------------------------------------- alias + @staticmethod + def _union_aliases( + chunk_graphs: Sequence[NormalizedChunkGraph], + ) -> Tuple[Dict[Tuple[str, str], str], Set[Tuple[str, str]]]: + """Union every chunk's alias table into a document-level index. + + When the same ``(label, key)`` maps to different canonical ids across + chunks, the key is marked ambiguous — resolving through it yields + ``ENDPOINT_AMBIGUOUS`` for the affected edges instead of silently + picking one of the candidates. + """ + doc_aliases: Dict[Tuple[str, str], str] = {} + ambiguous: Set[Tuple[str, str]] = set() + for cg in chunk_graphs: + for key, value in cg.aliases.items(): + if key in doc_aliases and doc_aliases[key] != value: + ambiguous.add(key) + # Keep the first mapping in the table; ambiguity is + # tracked separately so callers can distinguish it from + # "not found at all". + continue + doc_aliases[key] = value + return doc_aliases, ambiguous + + # -------------------------------------------------------------- vertices + @staticmethod + def _merge_vertices( + chunk_graphs: Sequence[NormalizedChunkGraph], + ) -> Tuple[List[Dict[str, Any]], List[StructuredWarning], int]: + """Merge vertices by ``(label, id)``. Returns (merged, warnings, pre_count). + + Rules per design section 6.5: + + * First-appearance wins for merge target and property values. + * Non-conflicting properties from later occurrences are added. + * Property conflicts emit ``PROPERTY_CONFLICT`` (soft — first value kept). + * Every merge event emits ``DUPLICATE_VERTEX_MERGED`` so the + quality gate can count exactly how many duplicates were folded in. + * Vertices without an ``id`` are kept as-is (they have no way to + merge with any other) and each occupies a distinct output slot. + """ + merged: Dict[Tuple[str, str], Dict[str, Any]] = {} + order: List[Tuple[str, str]] = [] + no_id_vertices: List[Dict[str, Any]] = [] + warnings: List[StructuredWarning] = [] + pre_count = 0 + + for cg in chunk_graphs: + for vertex in cg.vertices: + pre_count += 1 + vid = vertex.get("id") + label = vertex.get("label") + if vid is None or label is None: + # No key to merge on — keep the vertex verbatim. + no_id_vertices.append(dict(vertex)) + continue + key = (str(label), str(vid)) + if key not in merged: + merged[key] = _copy_vertex(vertex) + order.append(key) + continue + # Fold this occurrence into the existing merged vertex. + existing_props = merged[key].setdefault("properties", {}) + new_props = vertex.get("properties") or {} + for pk, pv in new_props.items(): + if pk not in existing_props: + existing_props[pk] = pv + elif existing_props[pk] != pv: + warnings.append( + StructuredWarning( + code=WarningCode.PROPERTY_CONFLICT, + item_type="vertex", + reason=( + f"property {pk!r} on {label} vertex conflicts across chunks; " + f"kept {existing_props[pk]!r}, discarded {pv!r}" + ), + label=str(label), + context={ + "property": pk, + "kept": existing_props[pk], + "discarded": pv, + }, + ) + ) + warnings.append( + StructuredWarning( + code=WarningCode.DUPLICATE_VERTEX_MERGED, + item_type="vertex", + reason=f"duplicate vertex merged by (label, id) = ({label}, {vid})", + label=str(label), + ) + ) + + ordered = [merged[k] for k in order] + ordered.extend(no_id_vertices) + return ordered, warnings, pre_count + + # ----------------------------------------------------------------- edges + def _repair_endpoints( + self, + chunk_graphs: Sequence[NormalizedChunkGraph], + doc_aliases: Mapping[Tuple[str, str], str], + ambiguous: Set[Tuple[str, str]], + ) -> Tuple[List[Dict[str, Any]], List[StructuredWarning], int, int]: + """Attempt document-level endpoint repair for edges left pending. + + Returns ``(edges, warnings, pre_count, repair_count)``: + + * ``edges`` — edges with both endpoints resolved, ready for dedup. + * ``pre_count`` — total edges seen across all chunks (pre-dedup). + * ``repair_count`` — number of pending endpoints the assembler resolved. + """ + warnings: List[StructuredWarning] = [] + edges: List[Dict[str, Any]] = [] + pre_count = 0 + repair_count = 0 + + for cg in chunk_graphs: + for edge in cg.edges: + pre_count += 1 + cleaned = {k: v for k, v in edge.items() if k not in (PENDING_OUT_KEY, PENDING_IN_KEY)} + + out_hint = edge.get(PENDING_OUT_KEY) + in_hint = edge.get(PENDING_IN_KEY) + edge_label = edge.get("label") + + out_result = self._resolve_pending(out_hint, edge.get("outVLabel"), doc_aliases, ambiguous) + in_result = self._resolve_pending(in_hint, edge.get("inVLabel"), doc_aliases, ambiguous) + + # Populate resolved endpoint ids where we succeeded. + if out_hint is not None and out_result[0] is not None: + cleaned["outV"] = out_result[0] + repair_count += 1 + if in_hint is not None and in_result[0] is not None: + cleaned["inV"] = in_result[0] + repair_count += 1 + + # Categorize a definitive failure so the reader can distinguish + # "no candidate" from "multiple candidates". + out_status = out_result[1] if out_hint is not None else "resolved" + in_status = in_result[1] if in_hint is not None else "resolved" + if "outV" not in cleaned: + out_status = "unresolved" if out_status == "resolved" else out_status + if "inV" not in cleaned: + in_status = "unresolved" if in_status == "resolved" else in_status + + if "outV" in cleaned and "inV" in cleaned: + edges.append(cleaned) + continue + + # Drop with a categorized warning. Ambiguous beats unresolved + # when both endpoints failed — it points to a real data issue. + if out_status == "ambiguous" or in_status == "ambiguous": + warnings.append( + StructuredWarning( + code=WarningCode.ENDPOINT_AMBIGUOUS, + item_type="edge", + reason=( + f"edge {edge_label!r} endpoint alias resolves to multiple canonical ids " + f"across chunks; edge dropped" + ), + label=str(edge_label) if edge_label else None, + ) + ) + else: + warnings.append( + StructuredWarning( + code=WarningCode.ENDPOINT_UNRESOLVED, + item_type="edge", + reason=( + f"edge {edge_label!r} has unresolved endpoint(s) after document-level " + f"repair; edge dropped" + ), + label=str(edge_label) if edge_label else None, + ) + ) + + return edges, warnings, pre_count, repair_count + + def _resolve_pending( + self, + hint: Optional[Mapping[str, Any]], + endpoint_label: Optional[str], + doc_aliases: Mapping[Tuple[str, str], str], + ambiguous: Set[Tuple[str, str]], + ) -> Tuple[Optional[str], str]: + """Resolve a single pending-endpoint hint against the doc-level index. + + Returns ``(canonical_id_or_None, status)`` where status is + ``"resolved"``, ``"unresolved"``, or ``"ambiguous"``. + """ + if hint is None or endpoint_label is None: + return None, "unresolved" + + if "original_id" in hint: + key = (endpoint_label, str(hint["original_id"])) + if key in ambiguous: + return None, "ambiguous" + resolved = doc_aliases.get(key) + return (resolved, "resolved") if resolved is not None else (None, "unresolved") + + if "legacy" in hint: + legacy = hint["legacy"] + # Re-attempt schema-based canonical id computation at doc level. + # Rarely more productive than the normalizer's earlier attempt, + # but harmless when the schema now has more property keys or + # the caller has supplied a richer schema. + if isinstance(legacy, Mapping): + canonical = self._schema.canonical_vertex_id(legacy.get("label"), legacy.get("properties") or {}) + if canonical is not None: + return canonical, "resolved" + return None, "unresolved" + + return None, "unresolved" + + # ------------------------------------------------------------- dedupe + @staticmethod + def _dedupe_edges( + edges: List[Dict[str, Any]], + ) -> Tuple[List[Dict[str, Any]], List[StructuredWarning]]: + """Deduplicate edges by ``(label, endpoints, properties_signature)``. + + Edges with the same endpoints but different property signatures are + preserved — the reader should not lose facts even when duplicates + share endpoints. First-appearance order is retained. + """ + seen: Dict[Tuple[Any, ...], Dict[str, Any]] = {} + order: List[Tuple[Any, ...]] = [] + warnings: List[StructuredWarning] = [] + for edge in edges: + props = edge.get("properties") or {} + # json.dumps with sort_keys gives a stable string signature that + # handles nested lists/dicts (e.g. LIST cardinality property values) + # which a plain tuple() would refuse to hash. + try: + prop_sig = json.dumps(props, sort_keys=True, ensure_ascii=False, default=str) + except TypeError: + prop_sig = repr(sorted(props.items())) + key = ( + edge.get("label"), + edge.get("outVLabel"), + edge.get("outV"), + edge.get("inVLabel"), + edge.get("inV"), + prop_sig, + ) + if key not in seen: + seen[key] = edge + order.append(key) + continue + warnings.append( + StructuredWarning( + code=WarningCode.DUPLICATE_EDGE_MERGED, + item_type="edge", + reason=( + f"duplicate edge merged by (label, endpoints, properties_signature) — " + f"label={edge.get('label')!r}" + ), + label=edge.get("label"), + ) + ) + return [seen[k] for k in order], warnings + + +def _copy_vertex(vertex: Mapping[str, Any]) -> Dict[str, Any]: + """Shallow-copy a vertex, deep-copying the properties dict so first-wins + conflict handling can mutate it safely.""" + out = dict(vertex) + if "properties" in out and isinstance(out["properties"], Mapping): + out["properties"] = dict(out["properties"]) + return out diff --git a/hugegraph-llm/src/hugegraph_llm/operators/llm_op/property_graph_extract_enhanced/evaluator.py b/hugegraph-llm/src/hugegraph_llm/operators/llm_op/property_graph_extract_enhanced/evaluator.py new file mode 100644 index 000000000..f1d6e08df --- /dev/null +++ b/hugegraph-llm/src/hugegraph_llm/operators/llm_op/property_graph_extract_enhanced/evaluator.py @@ -0,0 +1,304 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +"""Offline evaluator for schema-aware graph extraction. + +Given a schema plus a predicted graph and a ground-truth graph, the evaluator +computes: + +* Vertex / edge / overall precision, recall, F1 (set-based; matched by + ``(label, id)`` for vertices and ``(label, outV, inV)`` for edges). +* Property fidelity: ``property_valid_ratio`` over all predicted properties + (schema-allowed keys / total predicted keys) and ``property_exact_match_rate`` + over true-positive-item properties (matched (key, value) / total expected + (key, value) on TP items). + +The evaluator is strategy-agnostic: it accepts any extractor result whose shape +is ``{"vertices": [...], "edges": [...]}``. It powers the offline benchmark +that quantifies the enhanced strategy's quality gains over baseline. + +Edge cases: + +* Both sides empty → every ratio is 1.0 (nothing to find, nothing predicted). +* Predicted empty, expected non-empty → precision=1.0, recall=0.0, F1=0.0. +* Predicted non-empty, expected empty → precision=0.0, recall=1.0, F1=0.0. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from typing import Any, Callable, Dict, List, Mapping, Optional, Sequence, Tuple + +from hugegraph_llm.operators.llm_op.property_graph_extract_enhanced.schema_index import ( + GraphSchemaIndex, +) + + +@dataclass(frozen=True) +class ItemMetrics: + """Precision/recall/F1 for one item type (vertex or edge).""" + + predicted_count_raw: int + predicted_count_unique: int + expected_count: int + true_positive_count: int + precision: float + recall: float + f1: float + + def to_dict(self) -> Dict[str, Any]: + return { + "predicted_count_raw": self.predicted_count_raw, + "predicted_count_unique": self.predicted_count_unique, + "expected_count": self.expected_count, + "true_positive_count": self.true_positive_count, + "precision": self.precision, + "recall": self.recall, + "f1": self.f1, + } + + +@dataclass(frozen=True) +class PropertyMetrics: + """Property fidelity aggregated across all predicted items.""" + + predicted_property_count: int + valid_property_count: int + property_valid_ratio: float + expected_tp_property_count: int + predicted_tp_property_count: int + exact_match_property_count: int + property_exact_match_rate: float + + def to_dict(self) -> Dict[str, Any]: + return { + "predicted_property_count": self.predicted_property_count, + "valid_property_count": self.valid_property_count, + "property_valid_ratio": self.property_valid_ratio, + "expected_tp_property_count": self.expected_tp_property_count, + "predicted_tp_property_count": self.predicted_tp_property_count, + "exact_match_property_count": self.exact_match_property_count, + "property_exact_match_rate": self.property_exact_match_rate, + } + + +@dataclass(frozen=True) +class EvaluationReport: + """Aggregate evaluation of one predicted graph against ground truth.""" + + vertex_metrics: ItemMetrics + edge_metrics: ItemMetrics + property_metrics: PropertyMetrics + overall_precision: float + overall_recall: float + overall_f1: float + + def to_dict(self) -> Dict[str, Any]: + return { + "vertex_metrics": self.vertex_metrics.to_dict(), + "edge_metrics": self.edge_metrics.to_dict(), + "property_metrics": self.property_metrics.to_dict(), + "overall_precision": self.overall_precision, + "overall_recall": self.overall_recall, + "overall_f1": self.overall_f1, + } + + +def _safe_ratio(numerator: int, denominator: int) -> float: + """Empty-graph safe ratio: 0/0 → 1.0, else numerator/denominator.""" + if denominator == 0: + return 1.0 if numerator == 0 else 0.0 + return numerator / denominator + + +def _f1(precision: float, recall: float) -> float: + if precision + recall == 0.0: + return 0.0 + return 2.0 * precision * recall / (precision + recall) + + +KeyFn = Callable[[Mapping[str, Any]], Optional[Tuple[Any, ...]]] + + +class GraphExtractionEvaluator: + """Compare a predicted graph against a ground-truth graph under a schema. + + ``schema_index`` is used only to classify which predicted property keys are + schema-allowed and to reconstruct canonical vertex ids when the predicted + item omits ``id`` (rare, but happens when an extractor emits a legal vertex + without an id field). Structural matching itself is schema-agnostic. + """ + + def __init__(self, schema_index: GraphSchemaIndex) -> None: + self._schema_index = schema_index + + def evaluate( + self, + predicted: Mapping[str, Sequence[Mapping[str, Any]]], + expected: Mapping[str, Sequence[Mapping[str, Any]]], + ) -> EvaluationReport: + vertex_metrics, vertex_tp_pairs = self._score_items( + list(predicted.get("vertices", [])), + list(expected.get("vertices", [])), + key_fn=self._vertex_key, + ) + edge_metrics, edge_tp_pairs = self._score_items( + list(predicted.get("edges", [])), + list(expected.get("edges", [])), + key_fn=self._edge_key, + ) + property_metrics = self._score_properties( + all_predicted=list(predicted.get("vertices", [])) + list(predicted.get("edges", [])), + tp_pairs=vertex_tp_pairs + edge_tp_pairs, + ) + overall_precision = _safe_ratio( + vertex_metrics.true_positive_count + edge_metrics.true_positive_count, + vertex_metrics.predicted_count_unique + edge_metrics.predicted_count_unique, + ) + overall_recall = _safe_ratio( + vertex_metrics.true_positive_count + edge_metrics.true_positive_count, + vertex_metrics.expected_count + edge_metrics.expected_count, + ) + overall_f1 = _f1(overall_precision, overall_recall) + return EvaluationReport( + vertex_metrics=vertex_metrics, + edge_metrics=edge_metrics, + property_metrics=property_metrics, + overall_precision=overall_precision, + overall_recall=overall_recall, + overall_f1=overall_f1, + ) + + # -------------------------------------------------------------- match keys + def _vertex_key(self, vertex: Mapping[str, Any]) -> Optional[Tuple[str, str]]: + label = vertex.get("label") + if not label: + return None + raw_id = vertex.get("id") + if raw_id: + return (str(label), str(raw_id)) + canonical = self._schema_index.canonical_vertex_id(str(label), dict(vertex.get("properties", {}) or {})) + if canonical is None: + return None + return (str(label), canonical) + + def _edge_key(self, edge: Mapping[str, Any]) -> Optional[Tuple[str, str, str]]: + label = edge.get("label") + out_v = edge.get("outV") + in_v = edge.get("inV") + if not label or not out_v or not in_v: + return None + return (str(label), str(out_v), str(in_v)) + + # ------------------------------------------------------------ score helpers + def _score_items( + self, + predicted: Sequence[Mapping[str, Any]], + expected: Sequence[Mapping[str, Any]], + key_fn: KeyFn, + ) -> Tuple[ItemMetrics, List[Tuple[Mapping[str, Any], Mapping[str, Any]]]]: + pred_raw_keys: List[Tuple[Any, ...]] = [] + pred_by_key: Dict[Tuple[Any, ...], Mapping[str, Any]] = {} + for item in predicted: + key = key_fn(item) + if key is None: + continue + pred_raw_keys.append(key) + pred_by_key.setdefault(key, item) + + exp_by_key: Dict[Tuple[Any, ...], Mapping[str, Any]] = {} + for item in expected: + key = key_fn(item) + if key is None: + continue + exp_by_key.setdefault(key, item) + + pred_unique_set = set(pred_by_key) + exp_set = set(exp_by_key) + tp_set = pred_unique_set & exp_set + + precision = _safe_ratio(len(tp_set), len(pred_unique_set)) + recall = _safe_ratio(len(tp_set), len(exp_set)) + f1 = _f1(precision, recall) + + tp_pairs: List[Tuple[Mapping[str, Any], Mapping[str, Any]]] = [(pred_by_key[k], exp_by_key[k]) for k in tp_set] + + return ( + ItemMetrics( + predicted_count_raw=len(pred_raw_keys), + predicted_count_unique=len(pred_unique_set), + expected_count=len(exp_set), + true_positive_count=len(tp_set), + precision=precision, + recall=recall, + f1=f1, + ), + tp_pairs, + ) + + def _score_properties( + self, + all_predicted: Sequence[Mapping[str, Any]], + tp_pairs: Sequence[Tuple[Mapping[str, Any], Mapping[str, Any]]], + ) -> PropertyMetrics: + predicted_property_count = 0 + valid_property_count = 0 + for item in all_predicted: + label = item.get("label") + if not label: + continue + props = item.get("properties", {}) or {} + item_type = item.get("type") + for key in props: + predicted_property_count += 1 + if self._is_valid_property(str(label), item_type, key): + valid_property_count += 1 + + expected_tp_property_count = 0 + predicted_tp_property_count = 0 + exact_match_property_count = 0 + for pred, exp in tp_pairs: + pred_props = pred.get("properties", {}) or {} + exp_props = exp.get("properties", {}) or {} + predicted_tp_property_count += len(pred_props) + expected_tp_property_count += len(exp_props) + for key, value in pred_props.items(): + if key in exp_props and exp_props[key] == value: + exact_match_property_count += 1 + + return PropertyMetrics( + predicted_property_count=predicted_property_count, + valid_property_count=valid_property_count, + property_valid_ratio=_safe_ratio(valid_property_count, predicted_property_count), + expected_tp_property_count=expected_tp_property_count, + predicted_tp_property_count=predicted_tp_property_count, + exact_match_property_count=exact_match_property_count, + property_exact_match_rate=_safe_ratio(exact_match_property_count, expected_tp_property_count), + ) + + def _is_valid_property(self, label: str, item_type: Optional[Any], key: str) -> bool: + if item_type == "vertex": + return key in self._schema_index.allowed_properties("vertex", label) + if item_type == "edge": + return key in self._schema_index.allowed_properties("edge", label) + # Unknown type: allow the property if it's declared on either the vertex + # label or edge label of the same name (item_type is a soft hint). + if self._schema_index.is_vertex_label(label): + return key in self._schema_index.allowed_properties("vertex", label) + if self._schema_index.is_edge_label(label): + return key in self._schema_index.allowed_properties("edge", label) + return False diff --git a/hugegraph-llm/src/hugegraph_llm/operators/llm_op/property_graph_extract_enhanced/normalizer.py b/hugegraph-llm/src/hugegraph_llm/operators/llm_op/property_graph_extract_enhanced/normalizer.py new file mode 100644 index 000000000..64b8c99e6 --- /dev/null +++ b/hugegraph-llm/src/hugegraph_llm/operators/llm_op/property_graph_extract_enhanced/normalizer.py @@ -0,0 +1,437 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +"""Schema-aware chunk-level normalizer for the enhanced extraction strategy. + +Takes a ``CandidateGraph`` (already parsed from raw LLM output) and a compiled +``GraphSchemaIndex``, then produces a ``NormalizedChunkGraph`` in which: + +* every vertex has a schema-valid label and only schema-declared properties; +* property values have been safely coerced to the schema's declared types; +* every vertex whose primary keys resolve gets a canonical id + (``{vertex_label.id}:{pk1}!{pk2}``); vertices whose canonical rule doesn't + apply keep the LLM-provided id as a fallback (mirrors baseline); +* an alias table maps ``(label, llm_original_id) → canonical_id`` so the + document-level assembler can resolve cross-chunk endpoint references; +* every edge has a schema-valid label, only schema-declared properties, and + either both endpoints resolved to canonical ids (ready to emit) or explicit + ``_pending_out`` / ``_pending_in`` hints for the assembler. + +Vertex/edge processing order is: label → property filter → coerce → primary +key check → canonical id. Design section 6.4 pins this order so that a +schema-invalid property never causes a false primary-key miss. +""" + +from __future__ import annotations + +from typing import Any, Dict, List, Mapping, Optional, Tuple + +from hugegraph_llm.operators.llm_op.property_graph_extract_enhanced.schema_index import ( + GraphSchemaIndex, +) +from hugegraph_llm.operators.llm_op.property_graph_extract_enhanced.types import ( + PENDING_IN_KEY, + PENDING_OUT_KEY, + CandidateGraph, + NormalizedChunkGraph, +) +from hugegraph_llm.operators.llm_op.property_graph_extract_enhanced.warnings import ( + StructuredWarning, + WarningCode, +) + + +class SchemaAwareNormalizer: + """Normalize a chunk-level ``CandidateGraph`` against a ``GraphSchemaIndex``. + + Stateless — one instance can be shared across chunks and threads. The + schema index passed to the constructor is not modified. + """ + + def __init__(self, schema_index: GraphSchemaIndex) -> None: + self._schema = schema_index + + # ---------------------------------------------------------------- public + def normalize( + self, + candidate: CandidateGraph, + *, + chunk_id: Optional[int] = None, + ) -> Tuple[NormalizedChunkGraph, List[StructuredWarning]]: + """Normalize a chunk's candidate graph. Returns (graph, warnings).""" + warnings: List[StructuredWarning] = [] + aliases: Dict[Tuple[str, str], str] = {} + + vertices: List[Dict[str, Any]] = [] + for item in candidate.vertices: + normalized, item_warnings, alias = self._normalize_vertex(item, chunk_id) + warnings.extend(item_warnings) + if normalized is not None: + vertices.append(normalized) + # Seed identity aliases so edges can resolve endpoints regardless of + # whether the LLM referenced the vertex by its raw id or its canonical id. + vid = normalized.get("id") + if vid is not None: + aliases[(normalized["label"], str(vid))] = str(vid) + if alias is not None: + aliases[(alias[0], alias[1])] = alias[2] + + edges: List[Dict[str, Any]] = [] + for item in candidate.edges: + normalized, item_warnings = self._normalize_edge(item, aliases, chunk_id) + warnings.extend(item_warnings) + if normalized is not None: + edges.append(normalized) + + return ( + NormalizedChunkGraph(vertices=vertices, edges=edges, aliases=aliases), + warnings, + ) + + # ---------------------------------------------------------------- vertex + def _normalize_vertex( + self, item: Mapping[str, Any], chunk_id: Optional[int] + ) -> Tuple[Optional[Dict[str, Any]], List[StructuredWarning], Optional[Tuple[str, str, str]]]: + """Normalize a single candidate vertex. + + Returns a triple ``(vertex_or_None, warnings, alias_or_None)``: + + * ``vertex_or_None`` is the normalized vertex dict (with keys + ``type``, ``label``, ``properties``, and optionally ``id``), or + ``None`` when the vertex was dropped. + * ``alias_or_None`` is ``(label, llm_original_id, canonical_id)`` when + the LLM-provided id differs from the canonical one; the caller adds + it to the alias table. + """ + warnings: List[StructuredWarning] = [] + label = item.get("label") + + if not isinstance(label, str) or not self._schema.is_vertex_label(label): + warnings.append( + StructuredWarning( + code=WarningCode.VERTEX_LABEL_NOT_IN_SCHEMA, + item_type="vertex", + reason=f"vertex label {label!r} is not in the schema", + label=label if isinstance(label, str) else None, + chunk_id=chunk_id, + ) + ) + return None, warnings, None + + # Property filter + coerce (order pinned by design §6.4). + raw_properties = item.get("properties") or {} + if not isinstance(raw_properties, Mapping): + raw_properties = {} + filtered, prop_warnings, primary_key_fatal = self._filter_and_coerce_properties( + raw_properties=raw_properties, + item_type="vertex", + label=label, + primary_keys=set(self._schema.primary_keys(label)), + chunk_id=chunk_id, + ) + warnings.extend(prop_warnings) + if primary_key_fatal: + return None, warnings, None + + # Primary key completeness check. + pk_names = self._schema.primary_keys(label) + missing_pks = [k for k in pk_names if k not in filtered or filtered[k] in (None, "")] + if pk_names and missing_pks: + warnings.append( + StructuredWarning( + code=WarningCode.VERTEX_PRIMARY_KEY_MISSING, + item_type="vertex", + reason=f"primary key(s) {missing_pks} missing or empty on {label} vertex", + label=label, + chunk_id=chunk_id, + context={"missing_primary_keys": missing_pks}, + ) + ) + return None, warnings, None + + canonical_id = self._schema.canonical_vertex_id(label, filtered) + original_id = item.get("id") if isinstance(item.get("id"), (str, int)) else None + if isinstance(original_id, int): + original_id = str(original_id) + + normalized: Dict[str, Any] = { + "type": "vertex", + "label": label, + "properties": filtered, + } + # Prefer canonical id; fall back to the LLM's original id for schemas + # that lack vertex_label.id (baseline-compatible degrade path). + resolved_id = canonical_id if canonical_id is not None else original_id + if resolved_id is not None: + normalized["id"] = resolved_id + + alias: Optional[Tuple[str, str, str]] = None + if canonical_id is not None and original_id is not None and original_id != canonical_id: + warnings.append( + StructuredWarning( + code=WarningCode.VERTEX_ALIAS_RECORDED, + item_type="vertex", + reason=f"LLM original id {original_id!r} mapped to canonical id {canonical_id!r}", + label=label, + chunk_id=chunk_id, + context={"original_id": original_id, "canonical_id": canonical_id}, + ) + ) + alias = (label, original_id, canonical_id) + + return normalized, warnings, alias + + # ---------------------------------------------------------------- edge + def _normalize_edge( + self, + item: Mapping[str, Any], + aliases: Mapping[Tuple[str, str], str], + chunk_id: Optional[int], + ) -> Tuple[Optional[Dict[str, Any]], List[StructuredWarning]]: + """Normalize a single candidate edge.""" + warnings: List[StructuredWarning] = [] + label = item.get("label") + + if not isinstance(label, str) or not self._schema.is_edge_label(label): + warnings.append( + StructuredWarning( + code=WarningCode.EDGE_LABEL_NOT_IN_SCHEMA, + item_type="edge", + reason=f"edge label {label!r} is not in the schema", + label=label if isinstance(label, str) else None, + chunk_id=chunk_id, + ) + ) + return None, warnings + + # Property filter + coerce (edges have no primary key concept). + raw_properties = item.get("properties") or {} + if not isinstance(raw_properties, Mapping): + raw_properties = {} + filtered, prop_warnings, _fatal = self._filter_and_coerce_properties( + raw_properties=raw_properties, + item_type="edge", + label=label, + primary_keys=set(), + chunk_id=chunk_id, + ) + warnings.extend(prop_warnings) + + # Endpoint label discovery: prefer explicit outVLabel/inVLabel, then + # fall back to legacy source/target dicts. Missing labels we can still + # infer from the schema's edge spec (there is exactly one legal pair). + endpoint_spec = self._schema.edge_endpoint_spec(label) + assert endpoint_spec is not None # is_edge_label already verified + schema_out_label, schema_in_label = endpoint_spec + + legacy_source = item.get("source") if isinstance(item.get("source"), Mapping) else None + legacy_target = item.get("target") if isinstance(item.get("target"), Mapping) else None + out_label = item.get("outVLabel") or (legacy_source.get("label") if legacy_source else None) + in_label = item.get("inVLabel") or (legacy_target.get("label") if legacy_target else None) + + # If labels are absent, fall back to the schema-required labels rather + # than dropping — the LLM may have skipped these fields when they are + # redundant with the edge label. + if out_label is None: + out_label = schema_out_label + if in_label is None: + in_label = schema_in_label + + if not self._schema.is_endpoint_compatible(label, out_label, in_label): + warnings.append( + StructuredWarning( + code=WarningCode.EDGE_ENDPOINT_MISMATCH, + item_type="edge", + reason=( + f"edge {label!r} endpoints ({out_label!r} → {in_label!r}) do not match " + f"the schema spec ({schema_out_label!r} → {schema_in_label!r})" + ), + label=label, + chunk_id=chunk_id, + context={"out_label": out_label, "in_label": in_label}, + ) + ) + return None, warnings + + # Endpoint id resolution: try legacy source/target dicts first (they + # carry the primary key values needed for a schema-only canonical id), + # then the chunk's own alias table (LLM raw id → canonical id), then + # leave a pending marker for the document-level assembler. + out_v, out_pending = self._resolve_endpoint( + explicit_id=item.get("outV"), + legacy=legacy_source, + endpoint_label=out_label, + aliases=aliases, + ) + in_v, in_pending = self._resolve_endpoint( + explicit_id=item.get("inV"), + legacy=legacy_target, + endpoint_label=in_label, + aliases=aliases, + ) + + normalized: Dict[str, Any] = { + "type": "edge", + "label": label, + "outVLabel": out_label, + "inVLabel": in_label, + "properties": filtered, + } + if out_v is not None: + normalized["outV"] = out_v + if in_v is not None: + normalized["inV"] = in_v + + if out_pending is not None: + normalized[PENDING_OUT_KEY] = out_pending + if in_pending is not None: + normalized[PENDING_IN_KEY] = in_pending + + if out_v is None or in_v is None: + warnings.append( + StructuredWarning( + code=WarningCode.ENDPOINT_PENDING_REPAIR, + item_type="edge", + reason=( + f"edge {label!r} has {'out' if out_v is None else 'in'}-endpoint pending document-level repair" + ), + label=label, + chunk_id=chunk_id, + ) + ) + + return normalized, warnings + + def _resolve_endpoint( + self, + *, + explicit_id: Any, + legacy: Optional[Mapping[str, Any]], + endpoint_label: str, + aliases: Mapping[Tuple[str, str], str], + ) -> Tuple[Optional[str], Optional[Dict[str, Any]]]: + """Try three sources for a resolved canonical id, in order. + + Returns ``(canonical_id_or_None, pending_hint_or_None)``. When both + components are None, no endpoint info was provided at all — the caller + emits ENDPOINT_PENDING_REPAIR anyway (an endpoint that we can neither + resolve nor hint at will be dropped by the assembler as UNRESOLVED). + """ + # Tier 1 & 2 combined: legacy source/target dict has enough for the + # schema-only canonical id, which is stateless w.r.t. the chunk. + if isinstance(legacy, Mapping): + legacy_label = legacy.get("label") + legacy_props = legacy.get("properties") + if isinstance(legacy_label, str) and isinstance(legacy_props, Mapping): + canonical = self._schema.canonical_vertex_id(legacy_label, legacy_props) + if canonical is not None: + return canonical, None + # No canonical possible — keep the legacy dict as a hint so + # the assembler can decide. + return None, {"legacy": {"label": legacy_label, "properties": dict(legacy_props)}} + + # Tier 3: explicit outV / inV as an LLM raw id, resolved via aliases. + if isinstance(explicit_id, (str, int)): + key = str(explicit_id) + resolved = aliases.get((endpoint_label, key)) + if resolved is not None: + return resolved, None + return None, {"original_id": key} + + # Nothing to work with. + return None, None + + # ------------------------------------------------------------- helpers + def _filter_and_coerce_properties( + self, + *, + raw_properties: Mapping[str, Any], + item_type: str, + label: str, + primary_keys: set, + chunk_id: Optional[int], + ) -> Tuple[Dict[str, Any], List[StructuredWarning], bool]: + """Filter properties against the schema and coerce their values. + + Returns ``(filtered, warnings, primary_key_fatal)``. The fatal flag is + set when a primary-key property failed coercion — vertex callers use + it to drop the whole vertex; edges ignore it (edges have no PKs). + """ + allowed = self._schema.allowed_properties(item_type, label) + warnings: List[StructuredWarning] = [] + filtered: Dict[str, Any] = {} + pk_fatal = False + + for key, value in raw_properties.items(): + if key not in allowed: + warnings.append( + StructuredWarning( + code=WarningCode.PROPERTY_NOT_IN_SCHEMA, + item_type=item_type, + reason=f"property {key!r} is not allowed on {label} {item_type}", + label=label, + chunk_id=chunk_id, + context={"property": key}, + ) + ) + continue + coerced, reason = self._schema.coerce_property_value(key, value) + if reason is not None: + if key in primary_keys: + warnings.append( + StructuredWarning( + code=WarningCode.VERTEX_PRIMARY_KEY_INVALID, + item_type=item_type, + reason=reason, + label=label, + chunk_id=chunk_id, + context={"property": key}, + ) + ) + pk_fatal = True + # Return early so the caller drops the whole vertex. + return {}, warnings, True + warnings.append( + StructuredWarning( + code=WarningCode.PROPERTY_COERCION_FAILED, + item_type=item_type, + reason=reason, + label=label, + chunk_id=chunk_id, + context={"property": key}, + ) + ) + continue + # Successful coerce — emit soft PROPERTY_COERCED only when the + # value type actually changed. Same-type coerce (e.g. "Tom" → "Tom" + # for TEXT) is silent to avoid drowning downstream consumers. + if type(coerced) is not type(value): + warnings.append( + StructuredWarning( + code=WarningCode.PROPERTY_COERCED, + item_type=item_type, + reason=( + f"property {key!r} coerced from {type(value).__name__} to " + f"{self._schema.property_data_type(key)}" + ), + label=label, + chunk_id=chunk_id, + context={"property": key}, + ) + ) + filtered[key] = coerced + return filtered, warnings, pk_fatal diff --git a/hugegraph-llm/src/hugegraph_llm/operators/llm_op/property_graph_extract_enhanced/prompt_contract.py b/hugegraph-llm/src/hugegraph_llm/operators/llm_op/property_graph_extract_enhanced/prompt_contract.py new file mode 100644 index 000000000..6efaa058c --- /dev/null +++ b/hugegraph-llm/src/hugegraph_llm/operators/llm_op/property_graph_extract_enhanced/prompt_contract.py @@ -0,0 +1,70 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +"""Schema-aware prompt contract for the enhanced graph extraction strategy. + +The contract is a short structural-constraint block appended after the +caller's ``example_prompt`` and before the per-chunk input section. It does +NOT try to replace the downstream quality layer — parser, normalizer, +assembler, and quality gate remain the effective guarantors. The contract +only nudges the LLM toward a schema-clean first candidate so those stages +have less to fix up. + +Scope is intentionally narrow (design section 6.1): + +* structural constraints only — no chain-of-thought scaffolding, no + hallucination-avoidance instructions beyond the design's "omit uncertain + entities" clause, no anti-jailbreak text; +* no per-property or per-edge examples — the block stays language-agnostic + and short enough to leave headroom for the caller's own prompt content; +* no attempt to fix the ``example_prompt`` — the block is *appended*, so + a user-supplied prompt keeps whatever framing it already has. +""" + +from __future__ import annotations + +from hugegraph_llm.operators.llm_op.property_graph_extract_enhanced.schema_index import ( + GraphSchemaIndex, +) + + +def build_prompt_contract(schema_index: GraphSchemaIndex) -> str: + """Return the constraint block to append after the caller's example_prompt.""" + vertex_labels = ", ".join(sorted(schema_index.vertex_label_names())) or "(none)" + edge_labels = ", ".join(sorted(schema_index.edge_label_names())) or "(none)" + + return ( + "\n" + "# Enhanced-strategy constraints\n" + "The extraction below is post-processed by a schema-aware quality layer. To " + "maximize the fraction of your output that survives that layer:\n" + "\n" + f"1. Vertex `label` MUST be one of: {vertex_labels}.\n" + f"2. Edge `label` MUST be one of: {edge_labels}.\n" + "3. Every property key you emit on a vertex or edge MUST be declared in the " + "schema for that label. Omit properties whose keys are not declared.\n" + "4. Every vertex MUST include the primary-key properties declared by its " + "schema label; without them the vertex will be dropped.\n" + "5. Edge endpoints MAY be given either as `outV`/`inV` (referring to a " + "vertex `id` you emit in the same output) OR as `source`/`target` objects " + "containing `label` and `properties` (with the referenced vertex's primary " + "keys). Either form is accepted.\n" + "6. When the input text does not clearly state a fact, OMIT it. Do not " + "fabricate entities, edges, or property values.\n" + "7. Output ONLY JSON — no prose before or after the JSON block.\n" + "\n" + ) diff --git a/hugegraph-llm/src/hugegraph_llm/operators/llm_op/property_graph_extract_enhanced/quality_gate.py b/hugegraph-llm/src/hugegraph_llm/operators/llm_op/property_graph_extract_enhanced/quality_gate.py new file mode 100644 index 000000000..88f615ef8 --- /dev/null +++ b/hugegraph-llm/src/hugegraph_llm/operators/llm_op/property_graph_extract_enhanced/quality_gate.py @@ -0,0 +1,222 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +"""Graph quality gate for the enhanced graph extraction strategy. + +Aggregates the warnings emitted throughout the pipeline plus the counts +tracked by ``DocumentGraph`` into a single ``QualityMetrics`` bundle that: + +* powers ``meta.quality_metrics`` in the API response, +* feeds the baseline-vs-enhanced comparison report in ``docs/quality/``, +* stays stable across zero-input cases (empty documents, zero-candidate + chunks) — no NaN, no divide-by-zero. + +The gate is stateless: one call to ``compute`` gives a snapshot for the +inputs at hand. Nothing here mutates the passed-in graph or warning list. +""" + +from __future__ import annotations + +from collections import Counter +from dataclasses import dataclass +from typing import Any, Dict, FrozenSet, Sequence + +from hugegraph_llm.operators.llm_op.property_graph_extract_enhanced.types import DocumentGraph +from hugegraph_llm.operators.llm_op.property_graph_extract_enhanced.warnings import ( + StructuredWarning, + WarningCode, + warning_code_distribution, +) + +# Codes that cause a candidate item to disappear from the emitted graph. +# Excludes DUPLICATE_*_MERGED (item consolidated, not dropped) and +# PROPERTY_COERCED (value changed, item survived). +_VERTEX_DROP_CODES: FrozenSet[WarningCode] = frozenset( + { + WarningCode.VERTEX_LABEL_NOT_IN_SCHEMA, + WarningCode.VERTEX_PRIMARY_KEY_MISSING, + WarningCode.VERTEX_PRIMARY_KEY_INVALID, + } +) +_EDGE_DROP_CODES: FrozenSet[WarningCode] = frozenset( + { + WarningCode.EDGE_LABEL_NOT_IN_SCHEMA, + WarningCode.EDGE_ENDPOINT_MISMATCH, + WarningCode.ENDPOINT_UNRESOLVED, + WarningCode.ENDPOINT_AMBIGUOUS, + } +) +_ITEM_DROP_CODES: FrozenSet[WarningCode] = frozenset( + { + WarningCode.ITEM_NOT_OBJECT, + WarningCode.ITEM_TYPE_MISMATCH, + } +) +_PROPERTY_DROP_CODES: FrozenSet[WarningCode] = frozenset( + { + WarningCode.PROPERTY_NOT_IN_SCHEMA, + WarningCode.PROPERTY_COERCION_FAILED, + } +) + + +@dataclass(frozen=True) +class QualityMetrics: + """Aggregate quality metrics for one enhanced-strategy invocation. + + Ratios live in ``[0.0, 1.0]``. Zero-candidate cases record ``1.0`` + (no candidates, no problems) — this matches the design contract that + ratios never surface ``NaN``. + """ + + # Must metrics per design section 6.6. + schema_valid_vertex_ratio: float + schema_valid_edge_ratio: float + endpoint_resolution_rate: float + duplicate_vertex_reduction: float + duplicate_edge_reduction: float + property_valid_ratio: float + dropped_item_count: int + coerced_property_count: int + endpoint_repair_count: int + # Should metrics we can produce cheaply from the same inputs. + property_conflict_count: int + warning_code_distribution: Dict[str, int] + + def to_dict(self) -> Dict[str, Any]: + """Serialize to a JSON-safe dict suitable for the API response. + + Ratios are rounded to 4 decimals to keep the response compact and + stable across floating-point noise; integer counters flow through + unmodified. + """ + return { + "schema_valid_vertex_ratio": round(self.schema_valid_vertex_ratio, 4), + "schema_valid_edge_ratio": round(self.schema_valid_edge_ratio, 4), + "endpoint_resolution_rate": round(self.endpoint_resolution_rate, 4), + "duplicate_vertex_reduction": round(self.duplicate_vertex_reduction, 4), + "duplicate_edge_reduction": round(self.duplicate_edge_reduction, 4), + "property_valid_ratio": round(self.property_valid_ratio, 4), + "dropped_item_count": self.dropped_item_count, + "coerced_property_count": self.coerced_property_count, + "endpoint_repair_count": self.endpoint_repair_count, + "property_conflict_count": self.property_conflict_count, + "warning_code_distribution": dict(self.warning_code_distribution), + } + + +class GraphQualityGate: + """Compute ``QualityMetrics`` from a document graph and its warnings. + + Callers supply the pre-normalize candidate counts (typically the parser's + output totals) so the gate can compute schema-validity ratios without + re-scanning the raw LLM output. + """ + + @staticmethod + def compute( + document_graph: DocumentGraph, + warnings: Sequence[StructuredWarning], + *, + candidate_vertex_count: int, + candidate_edge_count: int, + ) -> QualityMetrics: + counter: Counter[WarningCode] = Counter() + for w in warnings: + if isinstance(w, StructuredWarning): + counter[w.code] += 1 + + post_merge_vertex = len(document_graph.vertices) + post_merge_edge = len(document_graph.edges) + pre_merge_vertex = document_graph.pre_merge_vertex_count + pre_merge_edge = document_graph.pre_merge_edge_count + + # Schema-validity ratios: fraction of candidate items that survived + # normalization. Duplicates count as valid (they made it through + # normalization); the merge step reduces them separately. + vertex_drops = sum(counter.get(c, 0) for c in _VERTEX_DROP_CODES) + edge_drops = sum(counter.get(c, 0) for c in _EDGE_DROP_CODES) + schema_valid_vertex_ratio = _safe_ratio(pre_merge_vertex, candidate_vertex_count) + schema_valid_edge_ratio = _safe_ratio(pre_merge_edge, candidate_edge_count) + + # Endpoint resolution: edges the normalizer left pending vs. how + # many the assembler either resolved or dropped as unresolved/ + # ambiguous. Using code counts ties the metric to observable + # warning surface rather than internal state. + pending_edges = counter.get(WarningCode.ENDPOINT_PENDING_REPAIR, 0) + unresolved_edges = counter.get(WarningCode.ENDPOINT_UNRESOLVED, 0) + counter.get( + WarningCode.ENDPOINT_AMBIGUOUS, 0 + ) + resolved_edges = max(0, pending_edges - unresolved_edges) + endpoint_resolution_rate = _safe_ratio(resolved_edges, pending_edges) + + # Duplicate reduction from the assembler's merge step. + duplicate_vertex_reduction = _safe_ratio( + max(0, pre_merge_vertex - post_merge_vertex), pre_merge_vertex, default=0.0 + ) + duplicate_edge_reduction = _safe_ratio(max(0, pre_merge_edge - post_merge_edge), pre_merge_edge, default=0.0) + + # Property validity: kept properties over kept-plus-invalidated ones. + # Merge-time conflicts do not count as invalid (the first value stays + # in the emitted graph). + kept_properties = _count_emitted_properties(document_graph) + property_drops = sum(counter.get(c, 0) for c in _PROPERTY_DROP_CODES) + property_valid_ratio = _safe_ratio(kept_properties, kept_properties + property_drops) + + item_drops = sum(counter.get(c, 0) for c in _ITEM_DROP_CODES) + dropped_item_count = vertex_drops + edge_drops + item_drops + property_drops + coerced_property_count = counter.get(WarningCode.PROPERTY_COERCED, 0) + property_conflict_count = counter.get(WarningCode.PROPERTY_CONFLICT, 0) + + return QualityMetrics( + schema_valid_vertex_ratio=schema_valid_vertex_ratio, + schema_valid_edge_ratio=schema_valid_edge_ratio, + endpoint_resolution_rate=endpoint_resolution_rate, + duplicate_vertex_reduction=duplicate_vertex_reduction, + duplicate_edge_reduction=duplicate_edge_reduction, + property_valid_ratio=property_valid_ratio, + dropped_item_count=dropped_item_count, + coerced_property_count=coerced_property_count, + endpoint_repair_count=document_graph.endpoint_repair_count, + property_conflict_count=property_conflict_count, + warning_code_distribution=warning_code_distribution(warnings), + ) + + +def _safe_ratio(numerator: int, denominator: int, *, default: float = 1.0) -> float: + """Return ``numerator / denominator`` with a deterministic zero fallback. + + Design contract: ratios never surface NaN. When both numerator and + denominator would be zero (no candidates, no problems) the ratio is + ``1.0`` by default. Callers can override to ``0.0`` for + reduction-style metrics where "nothing to reduce" should read as + "no reduction achieved". + """ + if denominator <= 0: + return default + return float(numerator) / float(denominator) + + +def _count_emitted_properties(graph: DocumentGraph) -> int: + total = 0 + for v in graph.vertices: + props = v.get("properties") or {} + total += len(props) + for e in graph.edges: + props = e.get("properties") or {} + total += len(props) + return total diff --git a/hugegraph-llm/src/hugegraph_llm/operators/llm_op/property_graph_extract_enhanced/schema_index.py b/hugegraph-llm/src/hugegraph_llm/operators/llm_op/property_graph_extract_enhanced/schema_index.py new file mode 100644 index 000000000..86dc5d3ff --- /dev/null +++ b/hugegraph-llm/src/hugegraph_llm/operators/llm_op/property_graph_extract_enhanced/schema_index.py @@ -0,0 +1,303 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +"""Runtime schema index for the enhanced graph extraction strategy. + +``GraphSchemaIndex`` compiles a HugeGraph schema (already validated by +``CheckSchema`` or fetched via ``SchemaManager``) into fast lookups used by the +schema-aware quality layer. It performs no HugeGraph I/O and does not modify the +input schema. +""" + +from __future__ import annotations + +import json +import re +from typing import Any, FrozenSet, List, Mapping, Optional, Tuple, Union + +_INT_TYPES = frozenset({"INT", "LONG", "BYTE"}) +_FLOAT_TYPES = frozenset({"FLOAT", "DOUBLE"}) +_TEXT_TYPES = frozenset({"TEXT", "UUID", "BLOB"}) + +_BOOL_TRUE = frozenset({"true", "yes", "1"}) +_BOOL_FALSE = frozenset({"false", "no", "0"}) + +_DATE_RE = re.compile(r"^\d{4}-\d{2}-\d{2}$") + + +class GraphSchemaIndex: + """Immutable lookup over a HugeGraph property-graph schema. + + Callers pass a schema object (already parsed) or a JSON string. Constructor + validates only the outer envelope shape; deeper validation belongs to + ``CheckSchema`` upstream. + """ + + def __init__(self, schema: Mapping[str, Any]) -> None: + if not isinstance(schema, Mapping): + raise TypeError("schema must be a mapping (dict-like).") + if "vertexlabels" not in schema or "edgelabels" not in schema: + raise ValueError("schema must contain 'vertexlabels' and 'edgelabels'.") + + self._raw: Mapping[str, Any] = schema + self._vertex_labels: dict[str, Mapping[str, Any]] = { + v["name"]: v for v in schema["vertexlabels"] if isinstance(v, Mapping) and "name" in v + } + self._edge_labels: dict[str, Mapping[str, Any]] = { + e["name"]: e for e in schema["edgelabels"] if isinstance(e, Mapping) and "name" in e + } + self._property_keys: dict[str, Mapping[str, Any]] = { + p["name"]: p for p in schema.get("propertykeys", []) if isinstance(p, Mapping) and "name" in p + } + + @classmethod + def from_schema(cls, schema: Union[str, Mapping[str, Any]]) -> "GraphSchemaIndex": + """Build an index from either a dict schema or a JSON-string schema. + + Named-graph strings (schema names without a leading '{') are not accepted: + callers must fetch the concrete schema first (see ``SchemaNode``). + """ + if isinstance(schema, str): + trimmed = schema.strip() + if not trimmed.startswith("{"): + raise ValueError( + "GraphSchemaIndex.from_schema does not resolve named graphs; " + "hand in the fetched schema object or JSON string." + ) + try: + parsed = json.loads(trimmed) + except json.JSONDecodeError as exc: + raise ValueError(f"schema JSON is not parseable: {exc}") from exc + return cls(parsed) + return cls(schema) + + # ------------------------------------------------------------------ labels + def is_vertex_label(self, label: str) -> bool: + return label in self._vertex_labels + + def is_edge_label(self, label: str) -> bool: + return label in self._edge_labels + + def vertex_label(self, label: str) -> Optional[Mapping[str, Any]]: + return self._vertex_labels.get(label) + + def edge_label(self, label: str) -> Optional[Mapping[str, Any]]: + return self._edge_labels.get(label) + + def vertex_label_names(self) -> FrozenSet[str]: + return frozenset(self._vertex_labels) + + def edge_label_names(self) -> FrozenSet[str]: + return frozenset(self._edge_labels) + + # -------------------------------------------------------------- properties + def is_property_key(self, key: str) -> bool: + return key in self._property_keys + + def property_data_type(self, key: str) -> str: + pk = self._property_keys.get(key) + if pk is None: + return "TEXT" + return str(pk.get("data_type", "TEXT")).upper() + + def property_cardinality(self, key: str) -> str: + pk = self._property_keys.get(key) + if pk is None: + return "SINGLE" + return str(pk.get("cardinality", "SINGLE")).upper() + + def allowed_properties(self, item_type: str, label: str) -> FrozenSet[str]: + """Return the set of property keys a given vertex/edge label may carry.""" + if item_type == "vertex": + v = self._vertex_labels.get(label) + if v is None: + return frozenset() + return frozenset(v.get("properties", []) or []) + if item_type == "edge": + e = self._edge_labels.get(label) + if e is None: + return frozenset() + return frozenset(e.get("properties", []) or []) + return frozenset() + + # --------------------------------------------------------- primary key id + def primary_keys(self, vertex_label: str) -> Tuple[str, ...]: + v = self._vertex_labels.get(vertex_label) + if v is None: + return () + pks = v.get("primary_keys") or () + return tuple(pks) + + def canonical_vertex_id(self, label: str, properties: Mapping[str, Any]) -> Optional[str]: + """Compute the canonical id ``{vertex_label.id}:{pk1}!{pk2}`` for a vertex. + + Returns ``None`` when the canonical rule cannot apply. This mirrors the + baseline behavior in ``PropertyGraphExtract._primary_key_id`` — callers + fall back to the LLM-provided raw id in that case. Preconditions: + + * ``vertex_label.id_strategy`` is ``PRIMARY_KEY`` (or absent, treated as + PRIMARY_KEY); + * ``vertex_label.id`` is present in the schema entry (HugeGraph server + populates this; inline user schemas may not); + * every primary key resolves to a non-empty property value in the input. + """ + v = self._vertex_labels.get(label) + if v is None: + return None + id_strategy = v.get("id_strategy") + if id_strategy and str(id_strategy).upper() != "PRIMARY_KEY": + return None + if "id" not in v: + return None + pks = v.get("primary_keys") or () + if not pks: + return None + values: List[str] = [] + for key in pks: + value = properties.get(key) + if value is None or value == "": + return None + values.append(str(value)) + return f"{v['id']}:{'!'.join(values)}" + + # ---------------------------------------------------------------- edges + def edge_endpoint_spec(self, edge_label: str) -> Optional[Tuple[str, str]]: + """Return ``(source_label, target_label)`` for a schema edge, or None.""" + e = self._edge_labels.get(edge_label) + if e is None: + return None + source = e.get("source_label") + target = e.get("target_label") + if not isinstance(source, str) or not isinstance(target, str): + return None + return source, target + + def is_endpoint_compatible(self, edge_label: str, out_label: str, in_label: str) -> bool: + spec = self.edge_endpoint_spec(edge_label) + if spec is None: + return False + return spec == (out_label, in_label) + + # ------------------------------------------------------------- coercion + def coerce_property_value(self, key: str, value: Any) -> Tuple[Any, Optional[str]]: + """Best-effort safe conversion of ``value`` to the schema's declared type. + + Returns ``(coerced_value, warning_reason)``. ``warning_reason`` is + ``None`` on clean success. Failures return ``(None, reason)``. For LIST + and SET cardinalities, partial success returns the surviving elements + plus a summary reason listing the dropped ones. + + Design notes: + + * INT/LONG/BYTE never accept booleans or lossy floats; DATE accepts only + ``YYYY-MM-DD`` (no fuzzy parsing) — see design doc §6.4. + * BLOB is treated as TEXT-like passthrough (unspecified in the design + doc, kept forgiving so downstream commit-to-graph handles the actual + bytes-vs-text boundary). + """ + cardinality = self.property_cardinality(key) + if cardinality in ("LIST", "SET"): + if not isinstance(value, list): + return ( + None, + f"property '{key}' expects a list for {cardinality} cardinality, got {type(value).__name__}", + ) + coerced: List[Any] = [] + dropped: List[str] = [] + for idx, item in enumerate(value): + item_coerced, item_reason = self._coerce_scalar(key, item) + if item_reason is not None: + dropped.append(f"[{idx}]: {item_reason}") + continue + coerced.append(item_coerced) + if cardinality == "SET": + seen: List[Any] = [] + for item in coerced: + if item not in seen: + seen.append(item) + coerced = seen + if dropped: + return coerced, f"property '{key}' dropped {len(dropped)} items: {'; '.join(dropped)}" + return coerced, None + return self._coerce_scalar(key, value) + + def _coerce_scalar(self, key: str, value: Any) -> Tuple[Any, Optional[str]]: + data_type = self.property_data_type(key) + if value is None: + return None, f"property '{key}' value is None" + + if data_type in _TEXT_TYPES: + if isinstance(value, str): + return value, None + if isinstance(value, bool): + # str(True) → "True" is rarely what an integrator wants; keep it + # explicit rather than silently converting. + return str(value), None + try: + return str(value), None + except Exception as exc: # pragma: no cover - defensive + return None, f"property '{key}' failed to stringify: {exc}" + + if data_type in _INT_TYPES: + if isinstance(value, bool): + return None, f"property '{key}' bool cannot be coerced to {data_type}" + if isinstance(value, int): + return value, None + if isinstance(value, float): + if value.is_integer(): + return int(value), None + return None, f"property '{key}' float {value} is not a lossless {data_type}" + if isinstance(value, str): + stripped = value.strip() + try: + return int(stripped), None + except ValueError: + return None, f"property '{key}' string '{value}' is not a {data_type}" + return None, f"property '{key}' cannot coerce {type(value).__name__} to {data_type}" + + if data_type in _FLOAT_TYPES: + if isinstance(value, bool): + return None, f"property '{key}' bool cannot be coerced to {data_type}" + if isinstance(value, (int, float)): + return float(value), None + if isinstance(value, str): + try: + return float(value.strip()), None + except ValueError: + return None, f"property '{key}' string '{value}' is not a {data_type}" + return None, f"property '{key}' cannot coerce {type(value).__name__} to {data_type}" + + if data_type == "BOOLEAN": + if isinstance(value, bool): + return value, None + if isinstance(value, int) and value in (0, 1): + return bool(value), None + if isinstance(value, str): + lowered = value.strip().lower() + if lowered in _BOOL_TRUE: + return True, None + if lowered in _BOOL_FALSE: + return False, None + return None, f"property '{key}' value {value!r} is not a BOOLEAN" + + if data_type == "DATE": + if isinstance(value, str) and _DATE_RE.match(value.strip()): + return value.strip(), None + return None, f"property '{key}' value {value!r} is not a YYYY-MM-DD DATE" + + # Unknown type — pass through as-is with a soft warning so the writer can decide. + return value, f"property '{key}' has unrecognized data_type {data_type!r}; passing through" diff --git a/hugegraph-llm/src/hugegraph_llm/operators/llm_op/property_graph_extract_enhanced/types.py b/hugegraph-llm/src/hugegraph_llm/operators/llm_op/property_graph_extract_enhanced/types.py new file mode 100644 index 000000000..184b8f25f --- /dev/null +++ b/hugegraph-llm/src/hugegraph_llm/operators/llm_op/property_graph_extract_enhanced/types.py @@ -0,0 +1,111 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +"""Shared data structures for the enhanced graph extraction quality layer.""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from typing import Any, Dict, List, Tuple + + +@dataclass(frozen=True) +class CandidateGraph: + """Chunk-level candidate graph produced by ``CandidateGraphParser``. + + The candidate graph is intentionally schema-agnostic — items are handed on + to the schema-aware normalizer in whatever shape the LLM produced them. + The parser guarantees only that: + + * ``vertices`` and ``edges`` are lists (possibly empty); + * every element is a ``dict`` (non-dict candidates are dropped by the + parser with an ``ITEM_NOT_OBJECT`` warning); + * items with an explicit ``type`` field agree with the array they live in + (mismatches are dropped with ``ITEM_TYPE_MISMATCH``). + + The dataclass is frozen so a consumer cannot rebind the ``vertices``/ + ``edges`` lists, but the lists themselves are ordinary mutable lists that + downstream stages read but do not modify in place. + """ + + vertices: List[Dict[str, Any]] = field(default_factory=list) + edges: List[Dict[str, Any]] = field(default_factory=list) + + @property + def is_empty(self) -> bool: + return not self.vertices and not self.edges + + +# Pending-endpoint marker keys used on normalized edges. They live under a +# leading underscore so JSON serialization at the API boundary can strip them +# without special casing individual fields. +PENDING_OUT_KEY = "_pending_out" +PENDING_IN_KEY = "_pending_in" + + +@dataclass(frozen=True) +class NormalizedChunkGraph: + """Chunk-level schema-normalized graph produced by ``SchemaAwareNormalizer``. + + Vertices are schema-valid and have canonical ids when the schema permits; + otherwise the LLM-provided id (or none) is kept for baseline-compatible + fallback. Edges either have both endpoints resolved to canonical ids + (ready to emit) or carry ``PENDING_OUT_KEY`` / ``PENDING_IN_KEY`` hints + describing what the document-level assembler can still try. Edges whose + endpoints have already been ruled out by the schema are dropped by the + normalizer and never appear here. + + ``aliases`` maps ``(label, llm_original_id)`` → ``canonical_id`` for every + vertex whose LLM-generated id differed from the canonical one. The + assembler unions these tables across chunks to service the third-tier + ``explicit_id_alias`` endpoint repair pass. + """ + + vertices: List[Dict[str, Any]] = field(default_factory=list) + edges: List[Dict[str, Any]] = field(default_factory=list) + aliases: Dict[Tuple[str, str], str] = field(default_factory=dict) + + @property + def is_empty(self) -> bool: + return not self.vertices and not self.edges + + +@dataclass(frozen=True) +class DocumentGraph: + """Final document-level graph ready for API response or write-to-graph. + + Emitted by ``DocumentGraphAssembler`` after cross-chunk vertex merge, + document-level endpoint repair, and edge deduplication. Vertices and + edges are plain dicts in the shape the existing baseline post-deal + already produces — the enhanced path stays wire-compatible. + + ``pre_merge_vertex_count`` / ``pre_merge_edge_count`` capture the totals + across all input chunks before deduplication; the quality gate uses them + to compute the duplicate-reduction ratios. ``endpoint_repair_count`` is + the number of edges the assembler resolved via the cross-chunk alias + table (edges resolved at chunk level do not count). + """ + + vertices: List[Dict[str, Any]] = field(default_factory=list) + edges: List[Dict[str, Any]] = field(default_factory=list) + pre_merge_vertex_count: int = 0 + pre_merge_edge_count: int = 0 + endpoint_repair_count: int = 0 + + @property + def is_empty(self) -> bool: + return not self.vertices and not self.edges diff --git a/hugegraph-llm/src/hugegraph_llm/operators/llm_op/property_graph_extract_enhanced/warnings.py b/hugegraph-llm/src/hugegraph_llm/operators/llm_op/property_graph_extract_enhanced/warnings.py new file mode 100644 index 000000000..c613811a3 --- /dev/null +++ b/hugegraph-llm/src/hugegraph_llm/operators/llm_op/property_graph_extract_enhanced/warnings.py @@ -0,0 +1,190 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +"""Structured warning types for the enhanced graph extraction strategy. + +The enhanced strategy attaches a structured warning to every deviation from the +schema-clean happy path (a dropped item, a coerced value, a merged duplicate, +an unresolvable endpoint). These warnings surface in ``meta.structured_warnings`` +alongside the resulting graph and drive both the effect report and downstream +debugging. + +A ``WarningCode`` is a ``str, Enum`` so it JSON-serializes as its bare code name +(``"ENDPOINT_UNRESOLVED"``) without needing custom encoders. ``StructuredWarning`` +is a frozen dataclass, cheap to hash and deduplicate. +""" + +from __future__ import annotations + +from collections import Counter +from dataclasses import dataclass, field +from enum import Enum +from typing import Any, Dict, Iterable, Mapping, Optional + + +class WarningCode(str, Enum): + """Stable identifiers for every warning the enhanced strategy may emit. + + Codes are grouped by their producer to make the surface easy to audit. + A single code may originate from multiple producers when the semantic + ("this property does not belong to the label") is the same regardless of + caller. + """ + + # -- Candidate parser (raw LLM output → chunk-level candidate graph) ------ + JSON_NOT_FOUND = "JSON_NOT_FOUND" + JSON_DECODE_FAILED = "JSON_DECODE_FAILED" + GRAPH_SECTION_MISSING = "GRAPH_SECTION_MISSING" + ITEM_NOT_OBJECT = "ITEM_NOT_OBJECT" + ITEM_TYPE_MISMATCH = "ITEM_TYPE_MISMATCH" + + # -- Schema-aware normalizer, vertex --------------------------------------- + VERTEX_LABEL_NOT_IN_SCHEMA = "VERTEX_LABEL_NOT_IN_SCHEMA" + VERTEX_PRIMARY_KEY_MISSING = "VERTEX_PRIMARY_KEY_MISSING" + VERTEX_PRIMARY_KEY_INVALID = "VERTEX_PRIMARY_KEY_INVALID" + VERTEX_ALIAS_RECORDED = "VERTEX_ALIAS_RECORDED" + + # -- Schema-aware normalizer, edge ----------------------------------------- + EDGE_LABEL_NOT_IN_SCHEMA = "EDGE_LABEL_NOT_IN_SCHEMA" + EDGE_ENDPOINT_MISMATCH = "EDGE_ENDPOINT_MISMATCH" + + # -- Property handling (vertex or edge) ------------------------------------ + PROPERTY_NOT_IN_SCHEMA = "PROPERTY_NOT_IN_SCHEMA" + PROPERTY_COERCED = "PROPERTY_COERCED" + PROPERTY_COERCION_FAILED = "PROPERTY_COERCION_FAILED" + + # -- Document-level assembler (cross-chunk repair and merge) --------------- + ENDPOINT_PENDING_REPAIR = "ENDPOINT_PENDING_REPAIR" + ENDPOINT_UNRESOLVED = "ENDPOINT_UNRESOLVED" + ENDPOINT_AMBIGUOUS = "ENDPOINT_AMBIGUOUS" + DUPLICATE_VERTEX_MERGED = "DUPLICATE_VERTEX_MERGED" + DUPLICATE_EDGE_MERGED = "DUPLICATE_EDGE_MERGED" + PROPERTY_CONFLICT = "PROPERTY_CONFLICT" + + +# Warnings that materially changed the emitted graph — reader should expect +# a dropped item, a coerced value, or a merged duplicate. Non-affecting +# warnings (e.g. VERTEX_ALIAS_RECORDED) are recorded for observability but do +# not cause the reader to lose or gain output items. +_SURFACE_AFFECTING = frozenset( + { + WarningCode.JSON_NOT_FOUND, + WarningCode.JSON_DECODE_FAILED, + WarningCode.ITEM_NOT_OBJECT, + WarningCode.ITEM_TYPE_MISMATCH, + WarningCode.VERTEX_LABEL_NOT_IN_SCHEMA, + WarningCode.VERTEX_PRIMARY_KEY_MISSING, + WarningCode.VERTEX_PRIMARY_KEY_INVALID, + WarningCode.EDGE_LABEL_NOT_IN_SCHEMA, + WarningCode.EDGE_ENDPOINT_MISMATCH, + WarningCode.PROPERTY_NOT_IN_SCHEMA, + WarningCode.PROPERTY_COERCED, + WarningCode.PROPERTY_COERCION_FAILED, + WarningCode.ENDPOINT_UNRESOLVED, + WarningCode.ENDPOINT_AMBIGUOUS, + WarningCode.DUPLICATE_VERTEX_MERGED, + WarningCode.DUPLICATE_EDGE_MERGED, + } +) + +_ITEM_TYPES = frozenset({"graph", "vertex", "edge"}) + + +@dataclass(frozen=True) +class StructuredWarning: + """A single, immutable warning produced by the enhanced strategy. + + Attributes: + code: The stable ``WarningCode`` identifying the reason. + item_type: ``"graph"``, ``"vertex"``, or ``"edge"``. ``"graph"`` covers + parser-level issues that are not attributable to a specific item. + reason: A short human-readable explanation. Format is not stable — + consumers should key off ``code`` for programmatic decisions. + label: The vertex or edge label the warning refers to, when known. + chunk_id: The chunk index (0-based) the warning originated from, when + the warning is attributable to a single chunk. + strategy: The extract strategy that produced the warning. Currently + always ``"enhanced"`` since baseline emits no structured warnings, + but kept explicit so future strategies can share the same envelope. + context: Optional free-form metadata (e.g. the offending property key) + that a debug consumer may find useful. Must be JSON-serializable. + """ + + code: WarningCode + item_type: str + reason: str + label: Optional[str] = None + chunk_id: Optional[int] = None + strategy: str = "enhanced" + context: Optional[Mapping[str, Any]] = field(default=None) + + def __post_init__(self) -> None: + # Accept raw string codes so callers can write + # StructuredWarning(code="JSON_NOT_FOUND", ...) without importing the enum. + if not isinstance(self.code, WarningCode): + object.__setattr__(self, "code", WarningCode(self.code)) + if self.item_type not in _ITEM_TYPES: + raise ValueError( + f"StructuredWarning.item_type must be one of {sorted(_ITEM_TYPES)}, got {self.item_type!r}" + ) + if self.chunk_id is not None and self.chunk_id < 0: + raise ValueError(f"StructuredWarning.chunk_id must be non-negative, got {self.chunk_id}") + if self.context is not None: + # Freeze the context so the dataclass stays truly immutable and + # dedup/hashing works via to_hashable_tuple below. + object.__setattr__(self, "context", dict(self.context)) + + @property + def is_surface_affecting(self) -> bool: + """Whether this warning represents a material change to the emitted graph.""" + return self.code in _SURFACE_AFFECTING + + def to_dict(self) -> Dict[str, Any]: + """Return a JSON-serializable dict for ``meta.structured_warnings``. + + Fields with a value of ``None`` are omitted so the response stays + compact. ``context`` is copied so downstream mutation cannot leak + back into the frozen warning. + """ + out: Dict[str, Any] = { + "code": self.code.value, + "item_type": self.item_type, + "reason": self.reason, + "strategy": self.strategy, + } + if self.label is not None: + out["label"] = self.label + if self.chunk_id is not None: + out["chunk_id"] = self.chunk_id + if self.context: + out["context"] = dict(self.context) + return out + + +def warning_code_distribution(warnings: Iterable[StructuredWarning]) -> Dict[str, int]: + """Aggregate a warning list into a ``code_name -> count`` histogram. + + Used by the effect report and the ``meta.debug_info.warning_code_distribution`` + field. Non-``StructuredWarning`` entries are silently skipped rather than + raising — the effect report should never fail because of a malformed entry + at the very end of a long pipeline. + """ + counter: Counter[str] = Counter() + for w in warnings: + if isinstance(w, StructuredWarning): + counter[w.code.value] += 1 + return dict(counter) diff --git a/hugegraph-llm/src/hugegraph_llm/state/ai_state.py b/hugegraph-llm/src/hugegraph_llm/state/ai_state.py index 739588c56..8d974f7af 100644 --- a/hugegraph-llm/src/hugegraph_llm/state/ai_state.py +++ b/hugegraph-llm/src/hugegraph_llm/state/ai_state.py @@ -30,6 +30,8 @@ class WkFlowInput(GParam): graph_client_config: Optional[Dict[str, Any]] = None data_json: Optional[Dict[str, Any]] = None extract_type: Optional[str] = None + extract_strategy: Optional[str] = None # "baseline" or "enhanced"; None falls back to baseline + include_debug: Optional[bool] = None # opt-in debug payload for enhanced strategy query_examples: Optional[Any] = None few_shot_schema: Optional[Any] = None # Fields related to PromptGenerate @@ -91,6 +93,8 @@ def reset(self, _: CStatus) -> None: self.graph_client_config = None self.data_json = None self.extract_type = None + self.extract_strategy = None + self.include_debug = None self.query_examples = None self.few_shot_schema = None # PromptGenerate related configuration @@ -145,6 +149,12 @@ class WkFlowState(GParam): vertices: Optional[List[Any]] = None triples: Optional[List[Any]] = None call_count: Optional[int] = None + extract_strategy: Optional[str] = None # mirrors WkFlowInput.extract_strategy for downstream nodes + # Enhanced-strategy outputs. Populated only when extract_strategy == "enhanced". + chunk_count: Optional[int] = None + structured_warnings: Optional[List[Dict[str, Any]]] = None + quality_metrics: Optional[Dict[str, Any]] = None + debug_info: Optional[Dict[str, Any]] = None keywords: Optional[List[str]] = None vector_result: Optional[Any] = None @@ -203,6 +213,11 @@ def setup(self) -> CStatus: self.vertices = None self.triples = None self.call_count = None + self.extract_strategy = None + self.chunk_count = None + self.structured_warnings = None + self.quality_metrics = None + self.debug_info = None self.keywords = None self.vector_result = None diff --git a/hugegraph-llm/src/tests/api/test_graph_extract_api.py b/hugegraph-llm/src/tests/api/test_graph_extract_api.py index 2f7cdc3f5..54096b7af 100644 --- a/hugegraph-llm/src/tests/api/test_graph_extract_api.py +++ b/hugegraph-llm/src/tests/api/test_graph_extract_api.py @@ -324,6 +324,263 @@ def test_flow_prepare_keeps_omitted_graphspace_none(): assert prepared_input.graph_client_config["graphspace"] is None +def test_request_defaults_extract_strategy_to_baseline(): + req = GraphExtractRequest(texts="hello", schema=INLINE_SCHEMA) + assert req.extract_strategy == "baseline" + assert req.include_debug is False + + +def test_request_accepts_explicit_baseline_and_enhanced_strategy(): + baseline_req = GraphExtractRequest(texts="hello", schema=INLINE_SCHEMA, extract_strategy="baseline") + enhanced_req = GraphExtractRequest( + texts="hello", + schema=INLINE_SCHEMA, + extract_strategy="enhanced", + include_debug=True, + ) + assert baseline_req.extract_strategy == "baseline" + assert baseline_req.include_debug is False + assert enhanced_req.extract_strategy == "enhanced" + assert enhanced_req.include_debug is True + + +def test_graph_extract_rejects_unknown_extract_strategy(): + response = _graph_client().post( + "/graph/extract", + json={"texts": "x", "schema": INLINE_SCHEMA, "extract_strategy": "aggressive"}, + ) + assert response.status_code == status.HTTP_422_UNPROCESSABLE_ENTITY + + +@patch("hugegraph_llm.api.graph_extract_api.SchedulerSingleton") +def test_graph_extract_threads_strategy_and_debug_into_scheduler_kwargs(mock_singleton): + scheduler = MagicMock() + scheduler.schedule_flow.return_value = json.dumps({"vertices": [], "edges": []}) + mock_singleton.get_instance.return_value = scheduler + + response = _graph_client().post( + "/graph/extract", + json={ + "texts": "x", + "schema": INLINE_SCHEMA, + "extract_strategy": "enhanced", + "include_debug": True, + }, + ) + + assert response.status_code == status.HTTP_200_OK + kwargs = scheduler.schedule_flow.call_args.kwargs + assert kwargs["extract_strategy"] == "enhanced" + assert kwargs["include_debug"] is True + + +@patch("hugegraph_llm.api.graph_extract_api.SchedulerSingleton") +def test_graph_extract_defaults_baseline_strategy_when_field_omitted(mock_singleton): + scheduler = MagicMock() + scheduler.schedule_flow.return_value = json.dumps({"vertices": [], "edges": []}) + mock_singleton.get_instance.return_value = scheduler + + response = _graph_client().post("/graph/extract", json={"texts": "x", "schema": INLINE_SCHEMA}) + + assert response.status_code == status.HTTP_200_OK + kwargs = scheduler.schedule_flow.call_args.kwargs + assert kwargs["extract_strategy"] == "baseline" + assert kwargs["include_debug"] is False + + +def test_flow_prepare_captures_extract_strategy_and_include_debug(): + flow = GraphExtractFlow() + prepared_input = WkFlowInput() + + flow.prepare( + prepared_input, + json.dumps(INLINE_SCHEMA), + ["text"], + "prompt", + "property_graph", + extract_strategy="enhanced", + include_debug=True, + ) + + assert prepared_input.extract_strategy == "enhanced" + assert prepared_input.include_debug is True + + +def test_flow_prepare_defaults_extract_strategy_and_include_debug(): + flow = GraphExtractFlow() + prepared_input = WkFlowInput() + + flow.prepare( + prepared_input, + json.dumps(INLINE_SCHEMA), + ["text"], + "prompt", + "property_graph", + ) + + assert prepared_input.extract_strategy == "baseline" + assert prepared_input.include_debug is False + + +def test_wkflow_input_reset_clears_enhanced_strategy_fields(): + prepared_input = WkFlowInput() + prepared_input.extract_strategy = "enhanced" + prepared_input.include_debug = True + + from pycgraph import CStatus # local import to avoid polluting module namespace + + prepared_input.reset(CStatus()) + + assert prepared_input.extract_strategy is None + assert prepared_input.include_debug is None + + +# ============================================================================== +# Enhanced-strategy envelope +# ============================================================================== + + +def _enhanced_scheduler_payload(*, structured_warnings=None, quality_metrics=None, debug_info=None): + """Mimic what GraphExtractFlow.post_deal emits under enhanced strategy.""" + payload = { + "vertices": [{"id": "1:Tom", "label": "person", "properties": {"name": "Tom"}}], + "edges": [], + "extract_strategy": "enhanced", + "chunk_count": 2, + "call_count": 2, + } + if structured_warnings is not None: + payload["structured_warnings"] = structured_warnings + if quality_metrics is not None: + payload["quality_metrics"] = quality_metrics + if debug_info is not None: + payload["debug_info"] = debug_info + return json.dumps(payload) + + +@patch("hugegraph_llm.api.graph_extract_api.SchedulerSingleton") +def test_enhanced_meta_carries_strategy_chunk_call_token_usage(mock_singleton): + scheduler = MagicMock() + scheduler.schedule_flow.return_value = _enhanced_scheduler_payload( + structured_warnings=[], + quality_metrics={"schema_valid_vertex_ratio": 1.0}, + ) + mock_singleton.get_instance.return_value = scheduler + + response = _graph_client().post( + "/graph/extract", + json={ + "texts": "some text", + "schema": INLINE_SCHEMA, + "extract_strategy": "enhanced", + "include_meta": True, + }, + ) + + body = response.json() + assert response.status_code == status.HTTP_200_OK + assert body["meta"]["extract_strategy"] == "enhanced" + assert body["meta"]["chunk_count"] == 2 + assert body["meta"]["call_count"] == 2 + assert body["meta"]["token_usage"] == "unavailable" + + +@patch("hugegraph_llm.api.graph_extract_api.SchedulerSingleton") +def test_enhanced_structured_warnings_and_quality_metrics_in_meta(mock_singleton): + warnings_payload = [ + {"code": "PROPERTY_COERCED", "item_type": "vertex", "reason": "x", "strategy": "enhanced"}, + {"code": "DUPLICATE_VERTEX_MERGED", "item_type": "vertex", "reason": "y", "strategy": "enhanced"}, + ] + metrics_payload = { + "schema_valid_vertex_ratio": 1.0, + "duplicate_vertex_reduction": 0.5, + } + scheduler = MagicMock() + scheduler.schedule_flow.return_value = _enhanced_scheduler_payload( + structured_warnings=warnings_payload, + quality_metrics=metrics_payload, + ) + mock_singleton.get_instance.return_value = scheduler + + response = _graph_client().post( + "/graph/extract", + json={ + "texts": "t", + "schema": INLINE_SCHEMA, + "extract_strategy": "enhanced", + "include_meta": True, + }, + ) + + body = response.json() + assert body["meta"]["structured_warnings"] == warnings_payload + assert body["meta"]["quality_metrics"] == metrics_payload + # Top-level warnings surface a short summary of the structured warning count. + assert any("structured warning" in w for w in body["warnings"]) + + +@patch("hugegraph_llm.api.graph_extract_api.SchedulerSingleton") +def test_enhanced_debug_info_only_when_include_debug(mock_singleton): + debug_payload = { + "chunks": [{"chunk_id": 0, "raw_output": "..."}], + "warning_code_distribution": {"PROPERTY_COERCED": 1}, + } + scheduler = MagicMock() + scheduler.schedule_flow.return_value = _enhanced_scheduler_payload( + structured_warnings=[], + quality_metrics={}, + debug_info=debug_payload, + ) + mock_singleton.get_instance.return_value = scheduler + + # include_debug=false → no debug_info in meta. + response = _graph_client().post( + "/graph/extract", + json={ + "texts": "t", + "schema": INLINE_SCHEMA, + "extract_strategy": "enhanced", + "include_meta": True, + }, + ) + assert "debug_info" not in response.json()["meta"] + + # include_debug=true → debug_info populated even without include_meta. + scheduler.schedule_flow.return_value = _enhanced_scheduler_payload( + structured_warnings=[], + quality_metrics={}, + debug_info=debug_payload, + ) + response = _graph_client().post( + "/graph/extract", + json={ + "texts": "t", + "schema": INLINE_SCHEMA, + "extract_strategy": "enhanced", + "include_debug": True, + }, + ) + body = response.json() + assert body["meta"]["debug_info"] == debug_payload + + +@patch("hugegraph_llm.api.graph_extract_api.SchedulerSingleton") +def test_baseline_response_untouched_when_extract_strategy_is_baseline(mock_singleton): + """Regression: baseline meta must stay byte-identical to the pre-enhanced shape.""" + scheduler = MagicMock() + scheduler.schedule_flow.return_value = json.dumps({"vertices": [{"id": "v1"}], "edges": []}) + mock_singleton.get_instance.return_value = scheduler + + response = _graph_client().post( + "/graph/extract", + json={"texts": "t", "schema": INLINE_SCHEMA, "include_meta": True}, + ) + body = response.json() + # Only the three baseline counts, nothing else. + assert body["meta"] == {"vertex_count": 1, "edge_count": 0, "text_count": 1} + assert body["warnings"] == [] + + def test_flow_prepare_does_not_leak_config_across_runs(): # A pooled pipeline is reused across requests, so prepare() must clear config # when a later request omits client_config. diff --git a/hugegraph-llm/src/tests/data/public_actor_corpus.json b/hugegraph-llm/src/tests/data/public_actor_corpus.json new file mode 100644 index 000000000..de967e4a7 --- /dev/null +++ b/hugegraph-llm/src/tests/data/public_actor_corpus.json @@ -0,0 +1,1976 @@ +{ + "meta": { + "schema_source": "Person + Movie + ACTED_IN (matches live_benchmark schema)", + "text_source": "Wikipedia lead extract (via /w/api.php action=query prop=extracts exintro)", + "ground_truth_source": "Wikidata P161 (cast member) verified per film via wbgetentities. Only films whose Wikidata entity is an instance-of (P31) a film class AND whose P161 claims include the actor's Q-id are admitted.", + "built_at_utc": "2026-07-05T10:13:18.577372+00:00", + "actor_count": 8, + "total_vertices": 65, + "total_edges": 57 + }, + "corpora": [ + { + "name": "Tom Hanks", + "actor_qid": "Q2263", + "wikipedia_title": "Tom Hanks", + "wikipedia_revision": 1362528109, + "wikipedia_url": "https://en.wikipedia.org/w/index.php?oldid=1362528109", + "text": "Thomas Jeffrey Hanks (born July 9, 1956) is an American actor and filmmaker. Known for both his comedic and dramatic roles, he is one of the most popular and recognizable film stars worldwide, and is regarded as an American cultural icon. In 2020, Hanks was ranked as the fourth-highest-grossing American film actor of all time. His numerous awards include two Academy Awards, seven Emmy Awards, and four Golden Globe Awards; he has also been nominated for five BAFTA Awards and a Tony Award. He received the AFI Life Achievement Award in 2002, the Kennedy Center Honor in 2014, the Presidential Medal of Freedom in 2016, and the Golden Globe Cecil B. DeMille Award in 2020.\nHanks rose to fame with leading roles in the comedies Splash (1984), The Money Pit (1986), Big (1988), and A League of Their Own (1992). He won two consecutive Academy Awards for Best Actor, playing a gay lawyer suffering from AIDS in Philadelphia (1993), then the title character in Forrest Gump (1994). Hanks has collaborated with Steven Spielberg on five films—Saving Private Ryan (1998), Catch Me If You Can (2002), The Terminal (2004), Bridge of Spies (2015) and The Post (2017)—and three World War II-themed miniseries: Band of Brothers (2001), The Pacific (2010) and Masters of the Air (2024). He has also frequently collaborated with directors Ron Howard, Nora Ephron and Robert Zemeckis.\nHanks cemented his film stardom with lead roles in the romantic comedies Sleepless in Seattle (1993) and You've Got Mail (1998); the dramas Apollo 13 (1995), The Green Mile (1999), Cast Away (2000), Road to Perdition (2002), Cloud Atlas (2012) and News of the World (2020); and the biographical dramas Charlie Wilson's War (2007), Captain Phillips (2013), Saving Mr. Banks (2013), Sully (2016), A Beautiful Day in the Neighborhood (2019) and Elvis (2022). He played the title character in the Robert Langdon series (2006–2016), voiced Sheriff Woody in the Toy Story franchise (1995–present) and multiple roles in The Polar Express (2004). Hanks directed and acted in That Thing You Do! (1996) and Larry Crowne (2011).\nHis breakthrough television role was a co-lead in the ABC sitcom Bosom Buddies (1980–1982). He has hosted Saturday Night Live ten times and launched a production company, Playtone, which has produced various limited series and television movies, including From the Earth to the Moon (1998), Band of Brothers, John Adams (2008), The Pacific, Game Change (2012) and Olive Kitteridge (2015). He made his Broadway debut in Nora Ephron's Lucky Guy (2013), earning a nomination for the Tony Award for Best Actor in a Play.", + "chunks": [ + "Thomas Jeffrey Hanks (born July 9, 1956) is an American actor and filmmaker. Known for both his comedic and dramatic roles, he is one of the most popular and recognizable film stars worldwide, and is regarded as an American cultural icon. In 2020, Hanks was ranked as the fourth-highest-grossing American film actor of all time. His numerous awards include two Academy Awards, seven Emmy Awards, and four Golden Globe Awards; he has also been nominated for five BAFTA Awards and a Tony Award. He received the AFI Life Achievement Award in 2002, the Kennedy Center Honor in 2014, the Presidential Medal of Freedom in 2016, and the Golden Globe Cecil B. DeMille Award in 2020.", + "Hanks rose to fame with leading roles in the comedies Splash (1984), The Money Pit (1986), Big (1988), and A League of Their Own (1992). He won two consecutive Academy Awards for Best Actor, playing a gay lawyer suffering from AIDS in Philadelphia (1993), then the title character in Forrest Gump (1994). Hanks has collaborated with Steven Spielberg on five films—Saving Private Ryan (1998), Catch Me If You Can (2002), The Terminal (2004), Bridge of Spies (2015) and The Post (2017)—and three World War II-themed miniseries: Band of Brothers (2001), The Pacific (2010) and Masters of the Air (2024). He has also frequently collaborated with directors Ron Howard, Nora Ephron and Robert Zemeckis. Hanks cemented his film stardom with lead roles in the romantic comedies Sleepless in Seattle (1993) and You've Got Mail (1998); the dramas Apollo 13 (1995), The Green Mile (1999), Cast Away (2000), Road to Perdition (2002), Cloud Atlas (2012) and News of the World (2020); and the biographical dramas Charlie Wilson's War (2007), Captain Phillips (2013), Saving Mr. Banks (2013), Sully (2016), A Beautiful Day in the Neighborhood (2019) and Elvis (2022).", + "He played the title character in the Robert Langdon series (2006–2016), voiced Sheriff Woody in the Toy Story franchise (1995–present) and multiple roles in The Polar Express (2004). Hanks directed and acted in That Thing You Do! (1996) and Larry Crowne (2011). His breakthrough television role was a co-lead in the ABC sitcom Bosom Buddies (1980–1982). He has hosted Saturday Night Live ten times and launched a production company, Playtone, which has produced various limited series and television movies, including From the Earth to the Moon (1998), Band of Brothers, John Adams (2008), The Pacific, Game Change (2012) and Olive Kitteridge (2015). He made his Broadway debut in Nora Ephron's Lucky Guy (2013), earning a nomination for the Tony Award for Best Actor in a Play." + ], + "ground_truth": { + "vertices": [ + { + "label": "Person", + "type": "vertex", + "id": "1:Tom Hanks", + "properties": { + "name": "Tom Hanks" + } + }, + { + "label": "Movie", + "type": "vertex", + "id": "2:Splash", + "properties": { + "title": "Splash", + "year": 1984 + } + }, + { + "label": "Movie", + "type": "vertex", + "id": "2:A League of Their Own", + "properties": { + "title": "A League of Their Own", + "year": 1992 + } + }, + { + "label": "Movie", + "type": "vertex", + "id": "2:Catch Me If You Can", + "properties": { + "title": "Catch Me If You Can", + "year": 2002 + } + }, + { + "label": "Movie", + "type": "vertex", + "id": "2:The Terminal", + "properties": { + "title": "The Terminal", + "year": 2004 + } + }, + { + "label": "Movie", + "type": "vertex", + "id": "2:Bridge of Spies", + "properties": { + "title": "Bridge of Spies", + "year": 2015 + } + }, + { + "label": "Movie", + "type": "vertex", + "id": "2:The Post", + "properties": { + "title": "The Post", + "year": 2017 + } + }, + { + "label": "Movie", + "type": "vertex", + "id": "2:Sleepless in Seattle", + "properties": { + "title": "Sleepless in Seattle", + "year": 1993 + } + } + ], + "edges": [ + { + "label": "ACTED_IN", + "type": "edge", + "outV": "1:Tom Hanks", + "inV": "2:Splash", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {} + }, + { + "label": "ACTED_IN", + "type": "edge", + "outV": "1:Tom Hanks", + "inV": "2:A League of Their Own", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {} + }, + { + "label": "ACTED_IN", + "type": "edge", + "outV": "1:Tom Hanks", + "inV": "2:Catch Me If You Can", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {} + }, + { + "label": "ACTED_IN", + "type": "edge", + "outV": "1:Tom Hanks", + "inV": "2:The Terminal", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {} + }, + { + "label": "ACTED_IN", + "type": "edge", + "outV": "1:Tom Hanks", + "inV": "2:Bridge of Spies", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {} + }, + { + "label": "ACTED_IN", + "type": "edge", + "outV": "1:Tom Hanks", + "inV": "2:The Post", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {} + }, + { + "label": "ACTED_IN", + "type": "edge", + "outV": "1:Tom Hanks", + "inV": "2:Sleepless in Seattle", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {} + } + ] + }, + "verified_films": [ + { + "title": "Splash", + "mentioned_title": "Splash", + "year": 1984, + "wikidata_qid": "Q1127184" + }, + { + "title": "A League of Their Own", + "mentioned_title": "A League of Their Own", + "year": 1992, + "wikidata_qid": "Q1306472" + }, + { + "title": "Catch Me If You Can", + "mentioned_title": "Catch Me If You Can", + "year": 2002, + "wikidata_qid": "Q208108" + }, + { + "title": "The Terminal", + "mentioned_title": "The Terminal", + "year": 2004, + "wikidata_qid": "Q318766" + }, + { + "title": "Bridge of Spies", + "mentioned_title": "Bridge of Spies", + "year": 2015, + "wikidata_qid": "Q18067135" + }, + { + "title": "The Post", + "mentioned_title": "The Post", + "year": 2017, + "wikidata_qid": "Q30203425" + }, + { + "title": "Sleepless in Seattle", + "mentioned_title": "Sleepless in Seattle", + "year": 1993, + "wikidata_qid": "Q107167" + } + ], + "rejected_mentions": [ + { + "title": "The Money Pit", + "year": "1986", + "reason": "not_a_film", + "qid": "Q116897480" + }, + { + "title": "Big", + "year": "1988", + "reason": "not_a_film", + "qid": "Q84" + }, + { + "title": "AIDS in Philadelphia", + "year": "1993", + "reason": "no_wikidata_match" + }, + { + "title": "Forrest Gump", + "year": "1994", + "reason": "not_a_film", + "qid": "Q552213" + }, + { + "title": "Private Ryan", + "year": "1998", + "reason": "no_wikidata_match" + }, + { + "title": "Band of Brothers", + "year": "2001", + "reason": "not_a_film", + "qid": "Q16950158" + }, + { + "title": "The Pacific", + "year": "2010", + "reason": "not_a_film", + "qid": "Q98" + }, + { + "title": "Air", + "year": "2024", + "reason": "not_a_film", + "qid": "Q318452" + } + ] + }, + { + "name": "Meryl Streep", + "actor_qid": "Q873", + "wikipedia_title": "Meryl Streep", + "wikipedia_revision": 1361813527, + "wikipedia_url": "https://en.wikipedia.org/w/index.php?oldid=1361813527", + "text": "Mary Louise \"Meryl\" Streep (born June 22, 1949) is an American actress. Widely regarded by peers and critics as the greatest actress of her generation, and by many as the greatest of all time, she is recognized as one of the most versatile performers in cinema, noted for her technical precision, nuanced emotional depth, command of dialects, and professional longevity. Streep is an alumna of Vassar College and the Yale School of Drama, holding a Bachelor of Arts and a Master of Fine Arts. Her artistic process often includes refining her characters' dialogue so that their motivations possess a psychological depth and agency that transcend traditional archetypes. Beyond her creative work, she is a prominent advocate for gender parity, labor protections, and a challenge to the influence of the male gaze in film criticism and production. \nStreep began her professional stage career in 1975 in Trelawny of the Wells and earned a Tony Award nomination the following year for 27 Wagons Full of Cotton. She transitioned to film in Julia (1977), followed by her first Academy Award nomination for The Deer Hunter (1978). For Kramer vs. Kramer (1979), she won her first Oscar after advocating to rewrite her character's courtroom testimony, shifting the role from a secondary antagonist to a portrayal focused on a woman's autonomy. She became a leading actress of the 1980s with performances in The French Lieutenant's Woman (1981), Sophie's Choice (1982), Out of Africa (1985), and A Cry in the Dark (1988), for which she received the Best Actress Award at Cannes.\nAfter expanding into various genres in the 1990s, Streep achieved a new level of commercial stardom in the 2000s. She transitioned from her established dramatic niche to lead high-grossing mainstream successes, portraying a formidable antagonist in The Devil Wears Prada (2006) and a singing lead in the musical Mamma Mia! (2008). During this period, she continued to receive critical acclaim for her dramatic turns in Doubt (2008) and The Iron Lady (2011), the latter earning her a third Academy Award. \nStreep's career honors include three Academy Awards, eight Golden Globe Awards, two British Academy Film Awards, three Primetime Emmy Awards, one Children's and Family Emmy Award, and two Screen Actors Guild Awards. She is a recipient of the AFI Life Achievement Award, the Honorary Palme d'Or, the Kennedy Center Honor, the Golden Globe Cecil B. DeMille Award, and an Honorary César. In 2014, she was awarded the Presidential Medal of Freedom. She holds the record for the most Academy Award and Globe nominations of any performer, with 21 and 34 respectively. A prolific artist, her career spans over 64 films and 18 television projects, alongside more than 20 musical recordings and 30 audiobooks.\n\n", + "chunks": [ + "Mary Louise \"Meryl\" Streep (born June 22, 1949) is an American actress. Widely regarded by peers and critics as the greatest actress of her generation, and by many as the greatest of all time, she is recognized as one of the most versatile performers in cinema, noted for her technical precision, nuanced emotional depth, command of dialects, and professional longevity. Streep is an alumna of Vassar College and the Yale School of Drama, holding a Bachelor of Arts and a Master of Fine Arts. Her artistic process often includes refining her characters' dialogue so that their motivations possess a psychological depth and agency that transcend traditional archetypes. Beyond her creative work, she is a prominent advocate for gender parity, labor protections, and a challenge to the influence of the male gaze in film criticism and production. Streep began her professional stage career in 1975 in Trelawny of the Wells and earned a Tony Award nomination the following year for 27 Wagons Full of Cotton.", + "She transitioned to film in Julia (1977), followed by her first Academy Award nomination for The Deer Hunter (1978). For Kramer vs. Kramer (1979), she won her first Oscar after advocating to rewrite her character's courtroom testimony, shifting the role from a secondary antagonist to a portrayal focused on a woman's autonomy. She became a leading actress of the 1980s with performances in The French Lieutenant's Woman (1981), Sophie's Choice (1982), Out of Africa (1985), and A Cry in the Dark (1988), for which she received the Best Actress Award at Cannes. After expanding into various genres in the 1990s, Streep achieved a new level of commercial stardom in the 2000s. She transitioned from her established dramatic niche to lead high-grossing mainstream successes, portraying a formidable antagonist in The Devil Wears Prada (2006) and a singing lead in the musical Mamma Mia!", + "(2008). During this period, she continued to receive critical acclaim for her dramatic turns in Doubt (2008) and The Iron Lady (2011), the latter earning her a third Academy Award. Streep's career honors include three Academy Awards, eight Golden Globe Awards, two British Academy Film Awards, three Primetime Emmy Awards, one Children's and Family Emmy Award, and two Screen Actors Guild Awards. She is a recipient of the AFI Life Achievement Award, the Honorary Palme d'Or, the Kennedy Center Honor, the Golden Globe Cecil B. DeMille Award, and an Honorary César. In 2014, she was awarded the Presidential Medal of Freedom. She holds the record for the most Academy Award and Globe nominations of any performer, with 21 and 34 respectively. A prolific artist, her career spans over 64 films and 18 television projects, alongside more than 20 musical recordings and 30 audiobooks." + ], + "ground_truth": { + "vertices": [ + { + "label": "Person", + "type": "vertex", + "id": "1:Meryl Streep", + "properties": { + "name": "Meryl Streep" + } + }, + { + "label": "Movie", + "type": "vertex", + "id": "2:The Deer Hunter", + "properties": { + "title": "The Deer Hunter", + "year": 1978 + } + }, + { + "label": "Movie", + "type": "vertex", + "id": "2:Sophie's Choice", + "properties": { + "title": "Sophie's Choice", + "year": 1982 + } + }, + { + "label": "Movie", + "type": "vertex", + "id": "2:Out of Africa", + "properties": { + "title": "Out of Africa", + "year": 1985 + } + }, + { + "label": "Movie", + "type": "vertex", + "id": "2:The Devil Wears Prada", + "properties": { + "title": "The Devil Wears Prada", + "year": 2006 + } + }, + { + "label": "Movie", + "type": "vertex", + "id": "2:Mamma Mia!", + "properties": { + "title": "Mamma Mia!", + "year": 2008 + } + }, + { + "label": "Movie", + "type": "vertex", + "id": "2:The Iron Lady", + "properties": { + "title": "The Iron Lady", + "year": 2011 + } + } + ], + "edges": [ + { + "label": "ACTED_IN", + "type": "edge", + "outV": "1:Meryl Streep", + "inV": "2:The Deer Hunter", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {} + }, + { + "label": "ACTED_IN", + "type": "edge", + "outV": "1:Meryl Streep", + "inV": "2:Sophie's Choice", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {} + }, + { + "label": "ACTED_IN", + "type": "edge", + "outV": "1:Meryl Streep", + "inV": "2:Out of Africa", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {} + }, + { + "label": "ACTED_IN", + "type": "edge", + "outV": "1:Meryl Streep", + "inV": "2:The Devil Wears Prada", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {} + }, + { + "label": "ACTED_IN", + "type": "edge", + "outV": "1:Meryl Streep", + "inV": "2:Mamma Mia!", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {} + }, + { + "label": "ACTED_IN", + "type": "edge", + "outV": "1:Meryl Streep", + "inV": "2:The Iron Lady", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {} + } + ] + }, + "verified_films": [ + { + "title": "The Deer Hunter", + "mentioned_title": "The Deer Hunter", + "year": 1978, + "wikidata_qid": "Q201674" + }, + { + "title": "Sophie's Choice", + "mentioned_title": "Sophie's Choice", + "year": 1982, + "wikidata_qid": "Q165627" + }, + { + "title": "Out of Africa", + "mentioned_title": "Out of Africa", + "year": 1985, + "wikidata_qid": "Q208266" + }, + { + "title": "The Devil Wears Prada", + "mentioned_title": "The Devil Wears Prada", + "year": 2006, + "wikidata_qid": "Q223316" + }, + { + "title": "Mamma Mia!", + "mentioned_title": "Mamma Mia!", + "year": 2008, + "wikidata_qid": "Q188850" + }, + { + "title": "The Iron Lady", + "mentioned_title": "The Iron Lady", + "year": 2011, + "wikidata_qid": "Q269810" + } + ], + "rejected_mentions": [ + { + "title": "Julia", + "year": "1977", + "reason": "not_a_film", + "qid": "Q2737173" + }, + { + "title": "For Kramer vs. Kramer", + "year": "1979", + "reason": "no_wikidata_match" + }, + { + "title": "The French Lieutenant's Woman", + "year": "1981", + "reason": "not_a_film", + "qid": "Q1268075" + }, + { + "title": "Dark", + "year": "1988", + "reason": "not_a_film", + "qid": "Q28443710" + }, + { + "title": "Doubt", + "year": "2008", + "reason": "not_a_film", + "qid": "Q37529041" + } + ] + }, + { + "name": "Leonardo DiCaprio", + "actor_qid": "Q38111", + "wikipedia_title": "Leonardo DiCaprio", + "wikipedia_revision": 1360720429, + "wikipedia_url": "https://en.wikipedia.org/w/index.php?oldid=1360720429", + "text": "Leonardo Wilhelm DiCaprio ( ; Italian: [diˈkaːprjo]; born November 11, 1974) is an American actor and film producer. Known for his work in biographical and period films, he is the recipient of numerous accolades, including an Academy Award, an Actor Award, a British Academy Film Award, an Emmy Award, a Silver Bear and three Golden Globes. His films as a leading actor have grossed $7 billion worldwide, and he has been placed eight times in annual rankings of the world's highest-paid actors.\nBorn in Los Angeles, DiCaprio began his career in the late 1980s by appearing in television commercials. He had a recurring role in the sitcom Parenthood (1990–1991), and had his first major film part as author Tobias Wolff in This Boy's Life (1993). He received critical acclaim and his first Academy Award nomination for playing a developmentally disabled boy in What's Eating Gilbert Grape (1993). DiCaprio achieved international stardom with the star-crossed romances Romeo + Juliet (1996) and Titanic (1997). After the latter became the highest-grossing film in the world at the time, he reduced his workload for a few years. In an attempt to shed his image of a romantic hero, DiCaprio sought roles in other genres, including the 2002 crime dramas Catch Me If You Can and Gangs of New York; the latter marked the first of his many successful collaborations with director Martin Scorsese.\nDiCaprio continued to gain acclaim for his performances in the biopic The Aviator (2004), the political thriller Blood Diamond (2006), the crime thriller The Departed (2006), and the romantic drama Revolutionary Road (2008). He later made environmental documentaries and starred in several high-profile directors' successful projects, including the thrillers Inception and Shutter Island (both 2010); the western Django Unchained (2012); the romantic drama The Great Gatsby (2013); the biopic The Wolf of Wall Street (2013); the survival drama The Revenant (2015), for which he won the Academy Award for Best Actor; the comedy-drama Once Upon a Time in Hollywood (2019); the crime drama Killers of the Flower Moon (2023) and the action film One Battle After Another (2025).\nDiCaprio is the founder of Appian Way Productions—a production company that has made some of his films and the documentary series Greensburg (2008–2010)—and Leonardo DiCaprio Foundation, a nonprofit organization devoted to promoting environmental awareness. A United Nations Messenger of Peace, he regularly supports charitable causes. In 2005, he was named a Commander of the Order of Arts and Letters for his contributions to the arts, and in 2016, he appeared in Time magazine's 100 most influential people in the world. DiCaprio was voted one of the 50 greatest actors of all time in a 2022 readers' poll by Empire magazine.", + "chunks": [ + "Leonardo Wilhelm DiCaprio ( ; Italian: [diˈkaːprjo]; born November 11, 1974) is an American actor and film producer. Known for his work in biographical and period films, he is the recipient of numerous accolades, including an Academy Award, an Actor Award, a British Academy Film Award, an Emmy Award, a Silver Bear and three Golden Globes. His films as a leading actor have grossed $7 billion worldwide, and he has been placed eight times in annual rankings of the world's highest-paid actors. Born in Los Angeles, DiCaprio began his career in the late 1980s by appearing in television commercials. He had a recurring role in the sitcom Parenthood (1990–1991), and had his first major film part as author Tobias Wolff in This Boy's Life (1993).", + "He received critical acclaim and his first Academy Award nomination for playing a developmentally disabled boy in What's Eating Gilbert Grape (1993). DiCaprio achieved international stardom with the star-crossed romances Romeo + Juliet (1996) and Titanic (1997). After the latter became the highest-grossing film in the world at the time, he reduced his workload for a few years. In an attempt to shed his image of a romantic hero, DiCaprio sought roles in other genres, including the 2002 crime dramas Catch Me If You Can and Gangs of New York; the latter marked the first of his many successful collaborations with director Martin Scorsese. DiCaprio continued to gain acclaim for his performances in the biopic The Aviator (2004), the political thriller Blood Diamond (2006), the crime thriller The Departed (2006), and the romantic drama Revolutionary Road (2008).", + "He later made environmental documentaries and starred in several high-profile directors' successful projects, including the thrillers Inception and Shutter Island (both 2010); the western Django Unchained (2012); the romantic drama The Great Gatsby (2013); the biopic The Wolf of Wall Street (2013); the survival drama The Revenant (2015), for which he won the Academy Award for Best Actor; the comedy-drama Once Upon a Time in Hollywood (2019); the crime drama Killers of the Flower Moon (2023) and the action film One Battle After Another (2025). DiCaprio is the founder of Appian Way Productions—a production company that has made some of his films and the documentary series Greensburg (2008–2010)—and Leonardo DiCaprio Foundation, a nonprofit organization devoted to promoting environmental awareness. A United Nations Messenger of Peace, he regularly supports charitable causes. In 2005, he was named a Commander of the Order of Arts and Letters for his contributions to the arts, and in 2016, he appeared in Time magazine's 100 most influential people in the world. DiCaprio was voted one of the 50 greatest actors of all time in a 2022 readers' poll by Empire magazine." + ], + "ground_truth": { + "vertices": [ + { + "label": "Person", + "type": "vertex", + "id": "1:Leonardo DiCaprio", + "properties": { + "name": "Leonardo DiCaprio" + } + }, + { + "label": "Movie", + "type": "vertex", + "id": "2:What's Eating Gilbert Grape", + "properties": { + "title": "What's Eating Gilbert Grape", + "year": 1993 + } + }, + { + "label": "Movie", + "type": "vertex", + "id": "2:The Aviator", + "properties": { + "title": "The Aviator", + "year": 2004 + } + }, + { + "label": "Movie", + "type": "vertex", + "id": "2:Blood Diamond", + "properties": { + "title": "Blood Diamond", + "year": 2006 + } + }, + { + "label": "Movie", + "type": "vertex", + "id": "2:The Departed", + "properties": { + "title": "The Departed", + "year": 2006 + } + }, + { + "label": "Movie", + "type": "vertex", + "id": "2:Revolutionary Road", + "properties": { + "title": "Revolutionary Road", + "year": 2008 + } + }, + { + "label": "Movie", + "type": "vertex", + "id": "2:Django Unchained", + "properties": { + "title": "Django Unchained", + "year": 2012 + } + }, + { + "label": "Movie", + "type": "vertex", + "id": "2:The Wolf of Wall Street", + "properties": { + "title": "The Wolf of Wall Street", + "year": 2013 + } + }, + { + "label": "Movie", + "type": "vertex", + "id": "2:The Revenant", + "properties": { + "title": "The Revenant", + "year": 2015 + } + }, + { + "label": "Movie", + "type": "vertex", + "id": "2:Once Upon a Time in Hollywood", + "properties": { + "title": "Once Upon a Time in Hollywood", + "year": 2019 + } + }, + { + "label": "Movie", + "type": "vertex", + "id": "2:One Battle After Another", + "properties": { + "title": "One Battle After Another", + "year": 2025 + } + } + ], + "edges": [ + { + "label": "ACTED_IN", + "type": "edge", + "outV": "1:Leonardo DiCaprio", + "inV": "2:What's Eating Gilbert Grape", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {} + }, + { + "label": "ACTED_IN", + "type": "edge", + "outV": "1:Leonardo DiCaprio", + "inV": "2:The Aviator", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {} + }, + { + "label": "ACTED_IN", + "type": "edge", + "outV": "1:Leonardo DiCaprio", + "inV": "2:Blood Diamond", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {} + }, + { + "label": "ACTED_IN", + "type": "edge", + "outV": "1:Leonardo DiCaprio", + "inV": "2:The Departed", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {} + }, + { + "label": "ACTED_IN", + "type": "edge", + "outV": "1:Leonardo DiCaprio", + "inV": "2:Revolutionary Road", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {} + }, + { + "label": "ACTED_IN", + "type": "edge", + "outV": "1:Leonardo DiCaprio", + "inV": "2:Django Unchained", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {} + }, + { + "label": "ACTED_IN", + "type": "edge", + "outV": "1:Leonardo DiCaprio", + "inV": "2:The Wolf of Wall Street", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {} + }, + { + "label": "ACTED_IN", + "type": "edge", + "outV": "1:Leonardo DiCaprio", + "inV": "2:The Revenant", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {} + }, + { + "label": "ACTED_IN", + "type": "edge", + "outV": "1:Leonardo DiCaprio", + "inV": "2:Once Upon a Time in Hollywood", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {} + }, + { + "label": "ACTED_IN", + "type": "edge", + "outV": "1:Leonardo DiCaprio", + "inV": "2:One Battle After Another", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {} + } + ] + }, + "verified_films": [ + { + "title": "What's Eating Gilbert Grape", + "mentioned_title": "What's Eating Gilbert Grape", + "year": 1993, + "wikidata_qid": "Q660894" + }, + { + "title": "The Aviator", + "mentioned_title": "The Aviator", + "year": 2004, + "wikidata_qid": "Q201924" + }, + { + "title": "Blood Diamond", + "mentioned_title": "Blood Diamond", + "year": 2006, + "wikidata_qid": "Q74958" + }, + { + "title": "The Departed", + "mentioned_title": "The Departed", + "year": 2006, + "wikidata_qid": "Q172975" + }, + { + "title": "Revolutionary Road", + "mentioned_title": "Revolutionary Road", + "year": 2008, + "wikidata_qid": "Q276299" + }, + { + "title": "Django Unchained", + "mentioned_title": "Django Unchained", + "year": 2012, + "wikidata_qid": "Q571032" + }, + { + "title": "The Wolf of Wall Street", + "mentioned_title": "The Wolf of Wall Street", + "year": 2013, + "wikidata_qid": "Q1392744" + }, + { + "title": "The Revenant", + "mentioned_title": "The Revenant", + "year": 2015, + "wikidata_qid": "Q18002795" + }, + { + "title": "Once Upon a Time in Hollywood", + "mentioned_title": "Once Upon a Time in Hollywood", + "year": 2019, + "wikidata_qid": "Q47300912" + }, + { + "title": "One Battle After Another", + "mentioned_title": "One Battle After Another", + "year": 2025, + "wikidata_qid": "Q117085614" + } + ], + "rejected_mentions": [ + { + "title": "Tobias Wolff in This Boy's Life", + "year": "1993", + "reason": "no_wikidata_match" + }, + { + "title": "Juliet", + "year": "1996", + "reason": "not_a_film", + "qid": "Q20000410" + }, + { + "title": "Titanic", + "year": "1997", + "reason": "not_a_film", + "qid": "Q25173" + }, + { + "title": "The Great Gatsby", + "year": "2013", + "reason": "not_a_film", + "qid": "Q214371" + }, + { + "title": "Flower Moon", + "year": "2023", + "reason": "not_a_film", + "qid": "Q36865409" + } + ] + }, + { + "name": "Denzel Washington", + "actor_qid": "Q42101", + "wikipedia_title": "Denzel Washington", + "wikipedia_revision": 1361981186, + "wikipedia_url": "https://en.wikipedia.org/w/index.php?oldid=1361981186", + "text": "Denzel Hayes Washington Jr. (born December 28, 1954) is an American actor. Known for his dramatic roles on stage and screen, he has received numerous accolades including two Academy Awards, an Actor Award, two Golden Globes, two Silver Bears and a Tony Award as well as nominations for a Grammy Award and two Primetime Emmy Awards. In 2020, The New York Times named Washington the greatest actor of the 21st century. He has also been honored with the Cecil B. DeMille Award in 2016, AFI Life Achievement Award in 2019, the Honorary Palme d'Or in 2025, and the Presidential Medal of Freedom in 2025. Films in which he has appeared have grossed over $5.1 billion worldwide.\nAfter training at the American Conservatory Theater, Washington began his career in theater, acting in performances off-Broadway. He first came to prominence in the NBC medical drama series St. Elsewhere (1982–1988), and in the war film A Soldier's Story (1984). Washington won Academy Awards for Best Supporting Actor for playing an American Civil War soldier in the war drama Glory (1989) and for Best Actor for playing a corrupt police officer in the crime thriller Training Day (2001). He was Oscar-nominated for his roles in Cry Freedom (1987), Malcolm X (1992), The Hurricane (1999), Flight (2012), Fences (2016), Roman J. Israel, Esq. (2017), and The Tragedy of Macbeth (2021).\nWashington has starred in other notable films, including The Pelican Brief, Philadelphia (both 1993); Crimson Tide, Devil in a Blue Dress (both 1995); He Got Game (1998); Remember the Titans (2000); Man on Fire (2004); Déjà Vu, Inside Man (both 2006); American Gangster (2007); Unstoppable, The Book of Eli (both 2010); The Equalizer trilogy (2014–2023), Gladiator II (2024), and Highest 2 Lowest (2025). Washington has also directed the films Antwone Fisher (2002), The Great Debaters (2007), Fences (2016), and A Journal for Jordan (2021).\nOn stage, he has acted in The Public Theater productions of Coriolanus (1979) and The Tragedy of Richard III (1990). He made his Broadway debut in the Ron Milner play Checkmates (1988). He won the Tony Award for Best Actor in a Play for his role as a disillusioned working class father in the Broadway revival of August Wilson's play Fences (2010). He has also acted in the Broadway revivals of William Shakespeare's Julius Caesar (2005) and Othello (2025), Lorraine Hansberry's play A Raisin in the Sun (2014), and Eugene O'Neill's play The Iceman Cometh (2018).\n\n", + "chunks": [ + "Denzel Hayes Washington Jr. (born December 28, 1954) is an American actor. Known for his dramatic roles on stage and screen, he has received numerous accolades including two Academy Awards, an Actor Award, two Golden Globes, two Silver Bears and a Tony Award as well as nominations for a Grammy Award and two Primetime Emmy Awards. In 2020, The New York Times named Washington the greatest actor of the 21st century. He has also been honored with the Cecil B. DeMille Award in 2016, AFI Life Achievement Award in 2019, the Honorary Palme d'Or in 2025, and the Presidential Medal of Freedom in 2025.", + "Films in which he has appeared have grossed over $5.1 billion worldwide. After training at the American Conservatory Theater, Washington began his career in theater, acting in performances off-Broadway. He first came to prominence in the NBC medical drama series St. Elsewhere (1982–1988), and in the war film A Soldier's Story (1984). Washington won Academy Awards for Best Supporting Actor for playing an American Civil War soldier in the war drama Glory (1989) and for Best Actor for playing a corrupt police officer in the crime thriller Training Day (2001). He was Oscar-nominated for his roles in Cry Freedom (1987), Malcolm X (1992), The Hurricane (1999), Flight (2012), Fences (2016), Roman J.", + "Israel, Esq. (2017), and The Tragedy of Macbeth (2021). Washington has starred in other notable films, including The Pelican Brief, Philadelphia (both 1993); Crimson Tide, Devil in a Blue Dress (both 1995); He Got Game (1998); Remember the Titans (2000); Man on Fire (2004); Déjà Vu, Inside Man (both 2006); American Gangster (2007); Unstoppable, The Book of Eli (both 2010); The Equalizer trilogy (2014–2023), Gladiator II (2024), and Highest 2 Lowest (2025). Washington has also directed the films Antwone Fisher (2002), The Great Debaters (2007), Fences (2016), and A Journal for Jordan (2021). On stage, he has acted in The Public Theater productions of Coriolanus (1979) and The Tragedy of Richard III (1990). He made his Broadway debut in the Ron Milner play Checkmates (1988). He won the Tony Award for Best Actor in a Play for his role as a disillusioned working class father in the Broadway revival of August Wilson's play Fences (2010). He has also acted in the Broadway revivals of William Shakespeare's Julius Caesar (2005) and Othello (2025), Lorraine Hansberry's play A Raisin in the Sun (2014), and Eugene O'Neill's play The Iceman Cometh (2018)." + ], + "ground_truth": { + "vertices": [ + { + "label": "Person", + "type": "vertex", + "id": "1:Denzel Washington", + "properties": { + "name": "Denzel Washington" + } + }, + { + "label": "Movie", + "type": "vertex", + "id": "2:A Soldier's Story", + "properties": { + "title": "A Soldier's Story", + "year": 1984 + } + }, + { + "label": "Movie", + "type": "vertex", + "id": "2:Cry Freedom", + "properties": { + "title": "Cry Freedom", + "year": 1987 + } + }, + { + "label": "Movie", + "type": "vertex", + "id": "2:The Hurricane", + "properties": { + "title": "The Hurricane", + "year": 1999 + } + }, + { + "label": "Movie", + "type": "vertex", + "id": "2:Fences", + "properties": { + "title": "Fences", + "year": 2016 + } + }, + { + "label": "Movie", + "type": "vertex", + "id": "2:Roman J. Israel, Esq.", + "properties": { + "title": "Roman J. Israel, Esq.", + "year": 2017 + } + }, + { + "label": "Movie", + "type": "vertex", + "id": "2:Remember the Titans", + "properties": { + "title": "Remember the Titans", + "year": 2000 + } + }, + { + "label": "Movie", + "type": "vertex", + "id": "2:Gladiator II", + "properties": { + "title": "Gladiator II", + "year": 2024 + } + } + ], + "edges": [ + { + "label": "ACTED_IN", + "type": "edge", + "outV": "1:Denzel Washington", + "inV": "2:A Soldier's Story", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {} + }, + { + "label": "ACTED_IN", + "type": "edge", + "outV": "1:Denzel Washington", + "inV": "2:Cry Freedom", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {} + }, + { + "label": "ACTED_IN", + "type": "edge", + "outV": "1:Denzel Washington", + "inV": "2:The Hurricane", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {} + }, + { + "label": "ACTED_IN", + "type": "edge", + "outV": "1:Denzel Washington", + "inV": "2:Fences", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {} + }, + { + "label": "ACTED_IN", + "type": "edge", + "outV": "1:Denzel Washington", + "inV": "2:Roman J. Israel, Esq.", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {} + }, + { + "label": "ACTED_IN", + "type": "edge", + "outV": "1:Denzel Washington", + "inV": "2:Remember the Titans", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {} + }, + { + "label": "ACTED_IN", + "type": "edge", + "outV": "1:Denzel Washington", + "inV": "2:Gladiator II", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {} + } + ] + }, + "verified_films": [ + { + "title": "A Soldier's Story", + "mentioned_title": "A Soldier's Story", + "year": 1984, + "wikidata_qid": "Q1624196" + }, + { + "title": "Cry Freedom", + "mentioned_title": "Cry Freedom", + "year": 1987, + "wikidata_qid": "Q962363" + }, + { + "title": "The Hurricane", + "mentioned_title": "The Hurricane", + "year": 1999, + "wikidata_qid": "Q832732" + }, + { + "title": "Fences", + "mentioned_title": "Fences", + "year": 2016, + "wikidata_qid": "Q25136220" + }, + { + "title": "Roman J. Israel, Esq.", + "mentioned_title": "Roman J. Israel, Esq.", + "year": 2017, + "wikidata_qid": "Q29096095" + }, + { + "title": "Remember the Titans", + "mentioned_title": "Remember the Titans", + "year": 2000, + "wikidata_qid": "Q117696" + }, + { + "title": "Gladiator II", + "mentioned_title": "Gladiator II", + "year": 2024, + "wikidata_qid": "Q115789958" + } + ], + "rejected_mentions": [ + { + "title": "Glory", + "year": "1989", + "reason": "not_a_film", + "qid": "Q892206" + }, + { + "title": "Training Day", + "year": "2001", + "reason": "not_a_film", + "qid": "Q137971633" + }, + { + "title": "Malcolm X", + "year": "1992", + "reason": "not_a_film", + "qid": "Q43303" + }, + { + "title": "Flight", + "year": "2012", + "reason": "not_a_film", + "qid": "Q13583449" + }, + { + "title": "The Tragedy of Macbeth", + "year": "2021", + "reason": "not_a_film", + "qid": "Q130283" + }, + { + "title": "He Got Game", + "year": "1998", + "reason": "not_a_film", + "qid": "Q1639281" + }, + { + "title": "Man on Fire", + "year": "2004", + "reason": "not_a_film", + "qid": "Q134599078" + }, + { + "title": "American Gangster", + "year": "2007", + "reason": "not_a_film", + "qid": "Q2455341" + } + ] + }, + { + "name": "Julia Roberts", + "actor_qid": "Q40523", + "wikipedia_title": "Julia Roberts", + "wikipedia_revision": 1361873675, + "wikipedia_url": "https://en.wikipedia.org/w/index.php?oldid=1361873675", + "text": "Julia Fiona Roberts (born October 28, 1967) is an American actress. Known for her leading roles across several genres, she has received various accolades, including an Academy Award, a British Academy Film Award, and three Golden Globes. She became known for playing charming and relatable characters in romantic comedies and blockbusters, before expanding into dramas, thrillers, and independent films.\nAfter early breakthroughs in Mystic Pizza (1988) and Steel Magnolias (1989), Roberts cemented herself as a Hollywood leading lady with the romantic comedies Pretty Woman (1990), My Best Friend's Wedding (1997), Notting Hill (1999), and Runaway Bride (1999). She won the Academy Award for Best Actress for her portrayal of the titular role in the biographical drama Erin Brockovich (2000). Roberts then starred in films such as Ocean's Eleven (2001), Ocean's Twelve (2004), Charlie Wilson's War (2007), Valentine's Day (2010), Eat Pray Love (2010), August: Osage County (2013), Wonder (2017), Ticket to Paradise (2022), Leave the World Behind (2023), and After the Hunt (2025). On television, she starred as a physician during the AIDS crisis in the HBO film The Normal Heart (2014), which earned her a Primetime Emmy Award nomination, a social worker in the first season of the streaming series Homecoming (2018), and portrayed Martha Mitchell in the Starz political limited series Gaslit (2022).\nRoberts runs the production company Red Om Films, through which she has served as an executive producer for various projects she has starred in, as well as for the first four films of the American Girl franchise (2004–2008). She has served as the global ambassador for Lancôme since 2009. She was the world's highest-paid actress throughout the majority of the 1990s and the first half of the 2000s. People magazine has named her the most beautiful woman in the world a record five times.\n\n", + "chunks": [ + "Julia Fiona Roberts (born October 28, 1967) is an American actress. Known for her leading roles across several genres, she has received various accolades, including an Academy Award, a British Academy Film Award, and three Golden Globes. She became known for playing charming and relatable characters in romantic comedies and blockbusters, before expanding into dramas, thrillers, and independent films.", + "After early breakthroughs in Mystic Pizza (1988) and Steel Magnolias (1989), Roberts cemented herself as a Hollywood leading lady with the romantic comedies Pretty Woman (1990), My Best Friend's Wedding (1997), Notting Hill (1999), and Runaway Bride (1999). She won the Academy Award for Best Actress for her portrayal of the titular role in the biographical drama Erin Brockovich (2000). Roberts then starred in films such as Ocean's Eleven (2001), Ocean's Twelve (2004), Charlie Wilson's War (2007), Valentine's Day (2010), Eat Pray Love (2010), August: Osage County (2013), Wonder (2017), Ticket to Paradise (2022), Leave the World Behind (2023), and After the Hunt (2025).", + "On television, she starred as a physician during the AIDS crisis in the HBO film The Normal Heart (2014), which earned her a Primetime Emmy Award nomination, a social worker in the first season of the streaming series Homecoming (2018), and portrayed Martha Mitchell in the Starz political limited series Gaslit (2022). Roberts runs the production company Red Om Films, through which she has served as an executive producer for various projects she has starred in, as well as for the first four films of the American Girl franchise (2004–2008). She has served as the global ambassador for Lancôme since 2009. She was the world's highest-paid actress throughout the majority of the 1990s and the first half of the 2000s. People magazine has named her the most beautiful woman in the world a record five times." + ], + "ground_truth": { + "vertices": [ + { + "label": "Person", + "type": "vertex", + "id": "1:Julia Roberts", + "properties": { + "name": "Julia Roberts" + } + }, + { + "label": "Movie", + "type": "vertex", + "id": "2:Mystic Pizza", + "properties": { + "title": "Mystic Pizza", + "year": 1988 + } + }, + { + "label": "Movie", + "type": "vertex", + "id": "2:Steel Magnolias", + "properties": { + "title": "Steel Magnolias", + "year": 1989 + } + }, + { + "label": "Movie", + "type": "vertex", + "id": "2:Pretty Woman", + "properties": { + "title": "Pretty Woman", + "year": 1990 + } + }, + { + "label": "Movie", + "type": "vertex", + "id": "2:My Best Friend's Wedding", + "properties": { + "title": "My Best Friend's Wedding", + "year": 1997 + } + }, + { + "label": "Movie", + "type": "vertex", + "id": "2:Ocean's Eleven", + "properties": { + "title": "Ocean's Eleven", + "year": 2001 + } + }, + { + "label": "Movie", + "type": "vertex", + "id": "2:Ocean's Twelve", + "properties": { + "title": "Ocean's Twelve", + "year": 2004 + } + }, + { + "label": "Movie", + "type": "vertex", + "id": "2:Charlie Wilson's War", + "properties": { + "title": "Charlie Wilson's War", + "year": 2007 + } + }, + { + "label": "Movie", + "type": "vertex", + "id": "2:Eat Pray Love", + "properties": { + "title": "Eat Pray Love", + "year": 2010 + } + }, + { + "label": "Movie", + "type": "vertex", + "id": "2:August: Osage County", + "properties": { + "title": "August: Osage County", + "year": 2013 + } + }, + { + "label": "Movie", + "type": "vertex", + "id": "2:Ticket to Paradise", + "properties": { + "title": "Ticket to Paradise", + "year": 2022 + } + } + ], + "edges": [ + { + "label": "ACTED_IN", + "type": "edge", + "outV": "1:Julia Roberts", + "inV": "2:Mystic Pizza", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {} + }, + { + "label": "ACTED_IN", + "type": "edge", + "outV": "1:Julia Roberts", + "inV": "2:Steel Magnolias", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {} + }, + { + "label": "ACTED_IN", + "type": "edge", + "outV": "1:Julia Roberts", + "inV": "2:Pretty Woman", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {} + }, + { + "label": "ACTED_IN", + "type": "edge", + "outV": "1:Julia Roberts", + "inV": "2:My Best Friend's Wedding", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {} + }, + { + "label": "ACTED_IN", + "type": "edge", + "outV": "1:Julia Roberts", + "inV": "2:Ocean's Eleven", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {} + }, + { + "label": "ACTED_IN", + "type": "edge", + "outV": "1:Julia Roberts", + "inV": "2:Ocean's Twelve", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {} + }, + { + "label": "ACTED_IN", + "type": "edge", + "outV": "1:Julia Roberts", + "inV": "2:Charlie Wilson's War", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {} + }, + { + "label": "ACTED_IN", + "type": "edge", + "outV": "1:Julia Roberts", + "inV": "2:Eat Pray Love", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {} + }, + { + "label": "ACTED_IN", + "type": "edge", + "outV": "1:Julia Roberts", + "inV": "2:August: Osage County", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {} + }, + { + "label": "ACTED_IN", + "type": "edge", + "outV": "1:Julia Roberts", + "inV": "2:Ticket to Paradise", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {} + } + ] + }, + "verified_films": [ + { + "title": "Mystic Pizza", + "mentioned_title": "Mystic Pizza", + "year": 1988, + "wikidata_qid": "Q1331243" + }, + { + "title": "Steel Magnolias", + "mentioned_title": "Steel Magnolias", + "year": 1989, + "wikidata_qid": "Q762140" + }, + { + "title": "Pretty Woman", + "mentioned_title": "Pretty Woman", + "year": 1990, + "wikidata_qid": "Q207954" + }, + { + "title": "My Best Friend's Wedding", + "mentioned_title": "My Best Friend's Wedding", + "year": 1997, + "wikidata_qid": "Q918380" + }, + { + "title": "Ocean's Eleven", + "mentioned_title": "Ocean's Eleven", + "year": 2001, + "wikidata_qid": "Q205447" + }, + { + "title": "Ocean's Twelve", + "mentioned_title": "Ocean's Twelve", + "year": 2004, + "wikidata_qid": "Q504053" + }, + { + "title": "Charlie Wilson's War", + "mentioned_title": "Charlie Wilson's War", + "year": 2007, + "wikidata_qid": "Q273568" + }, + { + "title": "Eat Pray Love", + "mentioned_title": "Eat Pray Love", + "year": 2010, + "wikidata_qid": "Q304030" + }, + { + "title": "August: Osage County", + "mentioned_title": "August: Osage County", + "year": 2013, + "wikidata_qid": "Q4055443" + }, + { + "title": "Ticket to Paradise", + "mentioned_title": "Ticket to Paradise", + "year": 2022, + "wikidata_qid": "Q109511080" + } + ], + "rejected_mentions": [ + { + "title": "Notting Hill", + "year": "1999", + "reason": "not_a_film", + "qid": "Q1411854" + }, + { + "title": "Runaway Bride", + "year": "1999", + "reason": "not_a_film", + "qid": "Q135047589" + }, + { + "title": "Erin Brockovich", + "year": "2000", + "reason": "not_a_film", + "qid": "Q232019" + }, + { + "title": "Valentine's Day", + "year": "2010", + "reason": "not_a_film", + "qid": "Q37587" + }, + { + "title": "Wonder", + "year": "2017", + "reason": "not_a_film", + "qid": "Q103820386" + } + ] + }, + { + "name": "Anthony Hopkins", + "actor_qid": "Q65932", + "wikipedia_title": "Anthony Hopkins", + "wikipedia_revision": 1362122753, + "wikipedia_url": "https://en.wikipedia.org/w/index.php?oldid=1362122753", + "text": "Sir Philip Anthony Hopkins (born 31 December 1937) is a Welsh actor. Considered one of Britain's most recognisable and prolific actors, he is known for his performances on the screen and stage. Hopkins has received numerous accolades, including two Academy Awards, four BAFTA Awards, two Primetime Emmy Awards, and a Laurence Olivier Award. He has also received the Cecil B. DeMille Award in 2005 and the BAFTA Fellowship for lifetime achievement in 2008. He was knighted by Queen Elizabeth II for his services to drama in 1993.\nAfter graduating from the Royal Welsh College of Music & Drama in 1957, Hopkins trained at RADA (the Royal Academy of Dramatic Art) in London. He was then spotted by Laurence Olivier, who invited him to join the Royal National Theatre in 1965. Productions at the National included King Lear (his favourite Shakespeare play), Coriolanus, Macbeth, and Antony and Cleopatra. In 1985, he received acclaim and a Laurence Olivier Award for his performance in the David Hare play Pravda. His last stage play was a West End production of M. Butterfly in 1989.\nHopkins's early film roles include The Lion in Winter (1968), A Bridge Too Far (1977), Magic (1978), and The Elephant Man (1980). He won two Academy Awards for Best Actor for playing Hannibal Lecter in the horror thriller The Silence of the Lambs (1991) and an octogenarian with dementia in the psychological drama The Father (2020). He was also Oscar-nominated for The Remains of the Day (1993), Nixon (1995), Amistad (1997), and The Two Popes (2019). Other notable films include 84 Charing Cross Road (1987), Howards End (1992), Bram Stoker's Dracula (1992), Shadowlands (1993), Legends of the Fall (1994), The Mask of Zorro (1998), and the Marvel Cinematic Universe films Thor (2011), Thor: The Dark World (2013), and Thor: Ragnarok (2017).\nFor his work on television, Hopkins received a British Academy Television Award for Best Actor for his performance in War and Peace (1972). He won two Primetime Emmy Awards for Outstanding Lead Actor in a Limited or Anthology Series or Movie for The Lindbergh Kidnapping Case (1976) and The Bunker (1981). Other notable projects include the BBC film The Dresser (2015), PBS's King Lear (2018), and the HBO series Westworld (2016–2018).\n\n", + "chunks": [ + "Sir Philip Anthony Hopkins (born 31 December 1937) is a Welsh actor. Considered one of Britain's most recognisable and prolific actors, he is known for his performances on the screen and stage. Hopkins has received numerous accolades, including two Academy Awards, four BAFTA Awards, two Primetime Emmy Awards, and a Laurence Olivier Award. He has also received the Cecil B. DeMille Award in 2005 and the BAFTA Fellowship for lifetime achievement in 2008. He was knighted by Queen Elizabeth II for his services to drama in 1993.", + "After graduating from the Royal Welsh College of Music & Drama in 1957, Hopkins trained at RADA (the Royal Academy of Dramatic Art) in London. He was then spotted by Laurence Olivier, who invited him to join the Royal National Theatre in 1965. Productions at the National included King Lear (his favourite Shakespeare play), Coriolanus, Macbeth, and Antony and Cleopatra. In 1985, he received acclaim and a Laurence Olivier Award for his performance in the David Hare play Pravda. His last stage play was a West End production of M. Butterfly in 1989.", + "Hopkins's early film roles include The Lion in Winter (1968), A Bridge Too Far (1977), Magic (1978), and The Elephant Man (1980). He won two Academy Awards for Best Actor for playing Hannibal Lecter in the horror thriller The Silence of the Lambs (1991) and an octogenarian with dementia in the psychological drama The Father (2020). He was also Oscar-nominated for The Remains of the Day (1993), Nixon (1995), Amistad (1997), and The Two Popes (2019). Other notable films include 84 Charing Cross Road (1987), Howards End (1992), Bram Stoker's Dracula (1992), Shadowlands (1993), Legends of the Fall (1994), The Mask of Zorro (1998), and the Marvel Cinematic Universe films Thor (2011), Thor: The Dark World (2013), and Thor: Ragnarok (2017). For his work on television, Hopkins received a British Academy Television Award for Best Actor for his performance in War and Peace (1972). He won two Primetime Emmy Awards for Outstanding Lead Actor in a Limited or Anthology Series or Movie for The Lindbergh Kidnapping Case (1976) and The Bunker (1981). Other notable projects include the BBC film The Dresser (2015), PBS's King Lear (2018), and the HBO series Westworld (2016–2018)." + ], + "ground_truth": { + "vertices": [ + { + "label": "Person", + "type": "vertex", + "id": "1:Anthony Hopkins", + "properties": { + "name": "Anthony Hopkins" + } + }, + { + "label": "Movie", + "type": "vertex", + "id": "2:A Bridge Too Far", + "properties": { + "title": "A Bridge Too Far", + "year": 1977 + } + }, + { + "label": "Movie", + "type": "vertex", + "id": "2:The Elephant Man", + "properties": { + "title": "The Elephant Man", + "year": 1980 + } + }, + { + "label": "Movie", + "type": "vertex", + "id": "2:The Two Popes", + "properties": { + "title": "The Two Popes", + "year": 2019 + } + }, + { + "label": "Movie", + "type": "vertex", + "id": "2:Howards End", + "properties": { + "title": "Howards End", + "year": 1992 + } + } + ], + "edges": [ + { + "label": "ACTED_IN", + "type": "edge", + "outV": "1:Anthony Hopkins", + "inV": "2:A Bridge Too Far", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {} + }, + { + "label": "ACTED_IN", + "type": "edge", + "outV": "1:Anthony Hopkins", + "inV": "2:The Elephant Man", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {} + }, + { + "label": "ACTED_IN", + "type": "edge", + "outV": "1:Anthony Hopkins", + "inV": "2:The Two Popes", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {} + }, + { + "label": "ACTED_IN", + "type": "edge", + "outV": "1:Anthony Hopkins", + "inV": "2:Howards End", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {} + } + ] + }, + "verified_films": [ + { + "title": "A Bridge Too Far", + "mentioned_title": "A Bridge Too Far", + "year": 1977, + "wikidata_qid": "Q467053" + }, + { + "title": "The Elephant Man", + "mentioned_title": "The Elephant Man", + "year": 1980, + "wikidata_qid": "Q272860" + }, + { + "title": "The Two Popes", + "mentioned_title": "The Two Popes", + "year": 2019, + "wikidata_qid": "Q58894738" + }, + { + "title": "Howards End", + "mentioned_title": "Howards End", + "year": 1992, + "wikidata_qid": "Q913389" + } + ], + "rejected_mentions": [ + { + "title": "The Lion in Winter", + "year": "1968", + "reason": "not_a_film", + "qid": "Q133880051" + }, + { + "title": "Magic", + "year": "1978", + "reason": "not_a_film", + "qid": "Q81741" + }, + { + "title": "Lambs", + "year": "1991", + "reason": "not_a_film", + "qid": "Q99517508" + }, + { + "title": "The Father", + "year": "2020", + "reason": "not_a_film", + "qid": "Q186350" + }, + { + "title": "Day", + "year": "1993", + "reason": "not_a_film", + "qid": "Q5242796" + }, + { + "title": "Nixon", + "year": "1995", + "reason": "not_a_film", + "qid": "Q16880460" + }, + { + "title": "Amistad", + "year": "1997", + "reason": "not_a_film", + "qid": "Q8197461" + }, + { + "title": "Charing Cross Road", + "year": "1987", + "reason": "not_a_film", + "qid": "Q1063230" + }, + { + "title": "Bram Stoker's Dracula", + "year": "1992", + "reason": "not_a_film", + "qid": "Q1406611" + }, + { + "title": "Shadowlands", + "year": "1993", + "reason": "not_a_film", + "qid": "Q7460744" + }, + { + "title": "Fall", + "year": "1994", + "reason": "not_a_film", + "qid": "Q4491399" + } + ] + }, + { + "name": "Nicole Kidman", + "actor_qid": "Q37459", + "wikipedia_title": "Nicole Kidman", + "wikipedia_revision": 1362589294, + "wikipedia_url": "https://en.wikipedia.org/w/index.php?oldid=1362589294", + "text": "Nicole Mary Kidman (born 20 June 1967) is an Australian-American actress and producer. Known for her work in blockbusters and independent films across many genres, she has consistently ranked among the world's highest-paid actresses since the late 1990s. Her accolades include an Academy Award, an Actor Award, a British Academy Film Award, six Golden Globe Awards, two Primetime Emmy Awards, and a Volpi Cup.\nKidman began her career in Australia with the 1983 films Bush Christmas and BMX Bandits. In 1989, her breakthrough came with lead roles in Dead Calm and the miniseries Bangkok Hilton. She rose to international prominence with a supporting role in Days of Thunder (1990) followed by leading roles in Far and Away (1992), To Die For (1995), Batman Forever (1995), Practical Magic (1998), and Eyes Wide Shut (1999).\nKidman earned Academy Award nominations for her performances in Moulin Rouge! (2001), Rabbit Hole (2010), Lion (2016), and Being the Ricardos (2021). For her portrayal of Virginia Woolf in The Hours (2002), she received the Academy Award for Best Actress and a shared Silver Bear for Best Actress. She also starred in acclaimed films such as Cold Mountain (2003), The Killing of a Sacred Deer (2017), The Beguiled (2017), Bombshell (2019), The Northman (2022), and Babygirl (2024). She achieved commercial success with the films The Others (2001), The Golden Compass (2007), Australia (2008), Paddington (2014), and Aquaman (2018).\nOn television, she produced and starred in the HBO series Big Little Lies (2017–2019), for which she won the Primetime Emmy Awards for Outstanding Limited Series and Outstanding Lead Actress in a Limited Series or Movie. She also starred in the television movie Hemingway & Gellhorn (2012), as well as television series Top of the Lake: China Girl (2017), The Undoing (2020), Nine Perfect Strangers (2021–present), Special Ops: Lioness (2023–present), Expats (2024), The Perfect Couple (2024), and Scarpetta (2026-present). \nKidman has served as a goodwill ambassador for UNICEF since 1994 and UNWomen since 2006. She was appointed Companion of the Order of Australia in 2006. Divorced from actor Tom Cruise and country musician Keith Urban, she has two children from each marriage. In 2010, she founded the production company Blossom Films. In 2004 and 2018, Time included her on its list of the 100 most influential people in the world, and in 2020, The New York Times named her one of the greatest actors of the 21st century. She was also honored with a star on the Hollywood Walk of Fame in 2003, and in 2024, became the first Australian actor to receive the AFI Life Achievement Award.", + "chunks": [ + "Nicole Mary Kidman (born 20 June 1967) is an Australian-American actress and producer. Known for her work in blockbusters and independent films across many genres, she has consistently ranked among the world's highest-paid actresses since the late 1990s. Her accolades include an Academy Award, an Actor Award, a British Academy Film Award, six Golden Globe Awards, two Primetime Emmy Awards, and a Volpi Cup. Kidman began her career in Australia with the 1983 films Bush Christmas and BMX Bandits. In 1989, her breakthrough came with lead roles in Dead Calm and the miniseries Bangkok Hilton. She rose to international prominence with a supporting role in Days of Thunder (1990) followed by leading roles in Far and Away (1992), To Die For (1995), Batman Forever (1995), Practical Magic (1998), and Eyes Wide Shut (1999).", + "Kidman earned Academy Award nominations for her performances in Moulin Rouge! (2001), Rabbit Hole (2010), Lion (2016), and Being the Ricardos (2021). For her portrayal of Virginia Woolf in The Hours (2002), she received the Academy Award for Best Actress and a shared Silver Bear for Best Actress. She also starred in acclaimed films such as Cold Mountain (2003), The Killing of a Sacred Deer (2017), The Beguiled (2017), Bombshell (2019), The Northman (2022), and Babygirl (2024). She achieved commercial success with the films The Others (2001), The Golden Compass (2007), Australia (2008), Paddington (2014), and Aquaman (2018). On television, she produced and starred in the HBO series Big Little Lies (2017–2019), for which she won the Primetime Emmy Awards for Outstanding Limited Series and Outstanding Lead Actress in a Limited Series or Movie.", + "She also starred in the television movie Hemingway & Gellhorn (2012), as well as television series Top of the Lake: China Girl (2017), The Undoing (2020), Nine Perfect Strangers (2021–present), Special Ops: Lioness (2023–present), Expats (2024), The Perfect Couple (2024), and Scarpetta (2026-present). Kidman has served as a goodwill ambassador for UNICEF since 1994 and UNWomen since 2006. She was appointed Companion of the Order of Australia in 2006. Divorced from actor Tom Cruise and country musician Keith Urban, she has two children from each marriage. In 2010, she founded the production company Blossom Films. In 2004 and 2018, Time included her on its list of the 100 most influential people in the world, and in 2020, The New York Times named her one of the greatest actors of the 21st century. She was also honored with a star on the Hollywood Walk of Fame in 2003, and in 2024, became the first Australian actor to receive the AFI Life Achievement Award." + ], + "ground_truth": { + "vertices": [ + { + "label": "Person", + "type": "vertex", + "id": "1:Nicole Kidman", + "properties": { + "name": "Nicole Kidman" + } + }, + { + "label": "Movie", + "type": "vertex", + "id": "2:Far and Away", + "properties": { + "title": "Far and Away", + "year": 1992 + } + }, + { + "label": "Movie", + "type": "vertex", + "id": "2:To Die For", + "properties": { + "title": "To Die For", + "year": 1995 + } + }, + { + "label": "Movie", + "type": "vertex", + "id": "2:Practical Magic", + "properties": { + "title": "Practical Magic", + "year": 1998 + } + }, + { + "label": "Movie", + "type": "vertex", + "id": "2:Eyes Wide Shut", + "properties": { + "title": "Eyes Wide Shut", + "year": 1999 + } + }, + { + "label": "Movie", + "type": "vertex", + "id": "2:Moulin Rouge!", + "properties": { + "title": "Moulin Rouge!", + "year": 2001 + } + }, + { + "label": "Movie", + "type": "vertex", + "id": "2:Being the Ricardos", + "properties": { + "title": "Being the Ricardos", + "year": 2021 + } + }, + { + "label": "Movie", + "type": "vertex", + "id": "2:The Beguiled", + "properties": { + "title": "The Beguiled", + "year": 2017 + } + } + ], + "edges": [ + { + "label": "ACTED_IN", + "type": "edge", + "outV": "1:Nicole Kidman", + "inV": "2:Far and Away", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {} + }, + { + "label": "ACTED_IN", + "type": "edge", + "outV": "1:Nicole Kidman", + "inV": "2:To Die For", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {} + }, + { + "label": "ACTED_IN", + "type": "edge", + "outV": "1:Nicole Kidman", + "inV": "2:Practical Magic", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {} + }, + { + "label": "ACTED_IN", + "type": "edge", + "outV": "1:Nicole Kidman", + "inV": "2:Eyes Wide Shut", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {} + }, + { + "label": "ACTED_IN", + "type": "edge", + "outV": "1:Nicole Kidman", + "inV": "2:Moulin Rouge!", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {} + }, + { + "label": "ACTED_IN", + "type": "edge", + "outV": "1:Nicole Kidman", + "inV": "2:Being the Ricardos", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {} + }, + { + "label": "ACTED_IN", + "type": "edge", + "outV": "1:Nicole Kidman", + "inV": "2:The Beguiled", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {} + } + ] + }, + "verified_films": [ + { + "title": "Far and Away", + "mentioned_title": "Far and Away", + "year": 1992, + "wikidata_qid": "Q839341" + }, + { + "title": "To Die For", + "mentioned_title": "To Die For", + "year": 1995, + "wikidata_qid": "Q590701" + }, + { + "title": "Practical Magic", + "mentioned_title": "Practical Magic", + "year": 1998, + "wikidata_qid": "Q148429" + }, + { + "title": "Eyes Wide Shut", + "mentioned_title": "Eyes Wide Shut", + "year": 1999, + "wikidata_qid": "Q209481" + }, + { + "title": "Moulin Rouge!", + "mentioned_title": "Moulin Rouge!", + "year": 2001, + "wikidata_qid": "Q193573" + }, + { + "title": "Being the Ricardos", + "mentioned_title": "Being the Ricardos", + "year": 2021, + "wikidata_qid": "Q105907125" + }, + { + "title": "The Beguiled", + "mentioned_title": "The Beguiled", + "year": 2017, + "wikidata_qid": "Q27684970" + } + ], + "rejected_mentions": [ + { + "title": "Days of Thunder", + "year": "1990", + "reason": "not_a_film", + "qid": "Q5243398" + }, + { + "title": "Batman Forever", + "year": "1995", + "reason": "not_a_film", + "qid": "Q2527522" + }, + { + "title": "Rabbit Hole", + "year": "2010", + "reason": "not_a_film", + "qid": "Q110555987" + }, + { + "title": "Lion", + "year": "2016", + "reason": "not_a_film", + "qid": "Q6555385" + }, + { + "title": "Virginia Woolf in The Hours", + "year": "2002", + "reason": "no_wikidata_match" + }, + { + "title": "Cold Mountain", + "year": "2003", + "reason": "not_a_film", + "qid": "Q5142303" + }, + { + "title": "Sacred Deer", + "year": "2017", + "reason": "no_wikidata_match" + }, + { + "title": "Bombshell", + "year": "2019", + "reason": "not_a_film", + "qid": "Q19501393" + } + ] + }, + { + "name": "Morgan Freeman", + "actor_qid": "Q48337", + "wikipedia_title": "Morgan Freeman", + "wikipedia_revision": 1356958182, + "wikipedia_url": "https://en.wikipedia.org/w/index.php?oldid=1356958182", + "text": "Morgan Freeman (born June 1, 1937) is an American actor, producer, and narrator. In a career spanning six decades, he has received numerous accolades, including an Academy Award and a Golden Globe Award, as well as a nomination for a Grammy Award and a Tony Award. He was honored with the Kennedy Center Honor in 2008, an AFI Life Achievement Award in 2011, the Cecil B. DeMille Award in 2012, and Screen Actors Guild Life Achievement Award in 2018. In a 2022 readers' poll by Empire, he was voted one of the 50 greatest actors of all time.\nBorn in Memphis, Tennessee, Freeman was raised in Mississippi, where he began acting in school plays. He studied theater arts in Los Angeles and appeared in stage productions in his early career. He rose to fame in the 1970s for his role in the children's television series The Electric Company. Freeman then appeared in the Shakespearean plays Coriolanus and Julius Caesar, the former of which earned him an Obie Award. In 1978, he was nominated for the Tony Award for Best Featured Actor in a Play for his role as Zeke in the Richard Wesley play The Mighty Gents.\nFreeman received the Academy Award for Best Supporting Actor playing a former boxer in Clint Eastwood's sports drama Million Dollar Baby (2004). He was Oscar-nominated for Street Smart (1987), Driving Miss Daisy (1989), The Shawshank Redemption (1994), and Invictus (2009). He also performed in Glory (1989), Lean on Me (1989), Robin Hood: Prince of Thieves (1991), Unforgiven (1992), Se7en (1995), Amistad (1997), Deep Impact (1998), Gone Baby Gone (2007), and The Bucket List (2007). He also portrayed Lucius Fox in Christopher Nolan's Dark Knight trilogy (2005–2012) and acted in the action films Wanted (2008), Red (2010), Oblivion (2013), the Now You See Me films (2013–2025), and Lucy (2014).\nKnown for his distinctive voice, he has narrated numerous documentaries including The Long Way Home (1997), March of the Penguins (2005), Through the Wormhole (2010–2017), The Story of God with Morgan Freeman (2016–2019), Our Universe (2022) and Life on Our Planet (2023). He made his directorial debut with the drama Bopha! (1993). He founded the film production company Revelations Entertainment with business partner Lori McCreary in 1996, under which they produced projects such as the CBS political drama Madam Secretary (2014–2019).\n\n", + "chunks": [ + "Morgan Freeman (born June 1, 1937) is an American actor, producer, and narrator. In a career spanning six decades, he has received numerous accolades, including an Academy Award and a Golden Globe Award, as well as a nomination for a Grammy Award and a Tony Award. He was honored with the Kennedy Center Honor in 2008, an AFI Life Achievement Award in 2011, the Cecil B. DeMille Award in 2012, and Screen Actors Guild Life Achievement Award in 2018. In a 2022 readers' poll by Empire, he was voted one of the 50 greatest actors of all time. Born in Memphis, Tennessee, Freeman was raised in Mississippi, where he began acting in school plays.", + "He studied theater arts in Los Angeles and appeared in stage productions in his early career. He rose to fame in the 1970s for his role in the children's television series The Electric Company. Freeman then appeared in the Shakespearean plays Coriolanus and Julius Caesar, the former of which earned him an Obie Award. In 1978, he was nominated for the Tony Award for Best Featured Actor in a Play for his role as Zeke in the Richard Wesley play The Mighty Gents. Freeman received the Academy Award for Best Supporting Actor playing a former boxer in Clint Eastwood's sports drama Million Dollar Baby (2004). He was Oscar-nominated for Street Smart (1987), Driving Miss Daisy (1989), The Shawshank Redemption (1994), and Invictus (2009).", + "He also performed in Glory (1989), Lean on Me (1989), Robin Hood: Prince of Thieves (1991), Unforgiven (1992), Se7en (1995), Amistad (1997), Deep Impact (1998), Gone Baby Gone (2007), and The Bucket List (2007). He also portrayed Lucius Fox in Christopher Nolan's Dark Knight trilogy (2005–2012) and acted in the action films Wanted (2008), Red (2010), Oblivion (2013), the Now You See Me films (2013–2025), and Lucy (2014). Known for his distinctive voice, he has narrated numerous documentaries including The Long Way Home (1997), March of the Penguins (2005), Through the Wormhole (2010–2017), The Story of God with Morgan Freeman (2016–2019), Our Universe (2022) and Life on Our Planet (2023). He made his directorial debut with the drama Bopha! (1993). He founded the film production company Revelations Entertainment with business partner Lori McCreary in 1996, under which they produced projects such as the CBS political drama Madam Secretary (2014–2019)." + ], + "ground_truth": { + "vertices": [ + { + "label": "Person", + "type": "vertex", + "id": "1:Morgan Freeman", + "properties": { + "name": "Morgan Freeman" + } + }, + { + "label": "Movie", + "type": "vertex", + "id": "2:Million Dollar Baby", + "properties": { + "title": "Million Dollar Baby", + "year": 2004 + } + }, + { + "label": "Movie", + "type": "vertex", + "id": "2:Driving Miss Daisy", + "properties": { + "title": "Driving Miss Daisy", + "year": 1989 + } + }, + { + "label": "Movie", + "type": "vertex", + "id": "2:The Shawshank Redemption", + "properties": { + "title": "The Shawshank Redemption", + "year": 1994 + } + }, + { + "label": "Movie", + "type": "vertex", + "id": "2:Lean on Me", + "properties": { + "title": "Lean on Me", + "year": 1989 + } + }, + { + "label": "Movie", + "type": "vertex", + "id": "2:Gone Baby Gone", + "properties": { + "title": "Gone Baby Gone", + "year": 2007 + } + }, + { + "label": "Movie", + "type": "vertex", + "id": "2:The Bucket List", + "properties": { + "title": "The Bucket List", + "year": 2007 + } + } + ], + "edges": [ + { + "label": "ACTED_IN", + "type": "edge", + "outV": "1:Morgan Freeman", + "inV": "2:Million Dollar Baby", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {} + }, + { + "label": "ACTED_IN", + "type": "edge", + "outV": "1:Morgan Freeman", + "inV": "2:Driving Miss Daisy", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {} + }, + { + "label": "ACTED_IN", + "type": "edge", + "outV": "1:Morgan Freeman", + "inV": "2:The Shawshank Redemption", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {} + }, + { + "label": "ACTED_IN", + "type": "edge", + "outV": "1:Morgan Freeman", + "inV": "2:Lean on Me", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {} + }, + { + "label": "ACTED_IN", + "type": "edge", + "outV": "1:Morgan Freeman", + "inV": "2:Gone Baby Gone", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {} + }, + { + "label": "ACTED_IN", + "type": "edge", + "outV": "1:Morgan Freeman", + "inV": "2:The Bucket List", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {} + } + ] + }, + "verified_films": [ + { + "title": "Million Dollar Baby", + "mentioned_title": "Million Dollar Baby", + "year": 2004, + "wikidata_qid": "Q184255" + }, + { + "title": "Driving Miss Daisy", + "mentioned_title": "Driving Miss Daisy", + "year": 1989, + "wikidata_qid": "Q211373" + }, + { + "title": "The Shawshank Redemption", + "mentioned_title": "The Shawshank Redemption", + "year": 1994, + "wikidata_qid": "Q172241" + }, + { + "title": "Lean on Me", + "mentioned_title": "Lean on Me", + "year": 1989, + "wikidata_qid": "Q1199323" + }, + { + "title": "Gone Baby Gone", + "mentioned_title": "Gone Baby Gone", + "year": 2007, + "wikidata_qid": "Q247182" + }, + { + "title": "The Bucket List", + "mentioned_title": "The Bucket List", + "year": 2007, + "wikidata_qid": "Q463768" + } + ], + "rejected_mentions": [ + { + "title": "Oscar-nominated for Street Smart", + "year": "1987", + "reason": "no_wikidata_match" + }, + { + "title": "Invictus", + "year": "2009", + "reason": "not_a_film", + "qid": "Q1097535" + }, + { + "title": "Glory", + "year": "1989", + "reason": "not_a_film", + "qid": "Q892206" + }, + { + "title": "Robin Hood: Prince of Thieves", + "year": "1991", + "reason": "not_a_film", + "qid": "Q7352488" + }, + { + "title": "Unforgiven", + "year": "1992", + "reason": "not_a_film", + "qid": "Q7884406" + }, + { + "title": "Se7en", + "year": "1995", + "reason": "not_a_film", + "qid": "Q12051546" + }, + { + "title": "Amistad", + "year": "1997", + "reason": "not_a_film", + "qid": "Q8197461" + }, + { + "title": "Deep Impact", + "year": "1998", + "reason": "not_a_film", + "qid": "Q730381" + }, + { + "title": "Wanted", + "year": "2008", + "reason": "not_a_film", + "qid": "Q7968158" + } + ] + } + ] +} \ No newline at end of file diff --git a/hugegraph-llm/src/tests/operators/llm_op/test_property_graph_benchmark.py b/hugegraph-llm/src/tests/operators/llm_op/test_property_graph_benchmark.py new file mode 100644 index 000000000..45899ea2c --- /dev/null +++ b/hugegraph-llm/src/tests/operators/llm_op/test_property_graph_benchmark.py @@ -0,0 +1,953 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +"""End-to-end benchmark: baseline vs. enhanced extraction strategy. + +Each scenario drives the same ``PropertyGraphExtract`` pipeline twice — once +with ``extract_strategy="baseline"``, once with ``extract_strategy="enhanced"`` +— feeding a deterministic ``FakeLLM`` per-chunk response set. The output of +each run is scored by :class:`GraphExtractionEvaluator` against ground truth. + +Design invariants asserted here: + +* Enhanced never regresses the F1 of any scenario relative to baseline + (``enhanced.overall_f1 >= baseline.overall_f1``). +* Enhanced wins strictly on the scenarios it was designed to fix + (cross-chunk edges, alias mismatch, type coercion). +* Average F1 improvement meets the design-stage threshold of + ``+5% relative`` — encoded as ``DESIGN_F1_RELATIVE_GAIN_MIN``. The threshold + is deliberately conservative; the current implementation clears it by + ~2x, but a regression that drops below +5% must fail CI, not just be + spotted in the report. + +Scenarios s15 and s16 cover *domain limitations* where neither strategy +wins: a fictional character with a schema-valid name (s15) and a +first-chunk pronoun with no antecedent for the assembler to bind to (s16). +Their presence forces the mock benchmark to report an honest +``enhanced avg F1 < 1.00``, matching the ceiling observed on real LLM output. + +The numeric table produced by :func:`test_benchmark_produces_comparison_table` +is the source of truth for ``docs/quality/schema-based-graph-extract-report.md``. +""" + +from __future__ import annotations + +import json +import time +from dataclasses import dataclass +from typing import Any, Dict, List, Mapping, Optional, Sequence, Tuple + +import pytest + +from hugegraph_llm.operators.llm_op.property_graph_extract import PropertyGraphExtract +from hugegraph_llm.operators.llm_op.property_graph_extract_enhanced import ( + EvaluationReport, + GraphExtractionEvaluator, + GraphSchemaIndex, +) +from tests.fixtures.fake_llm import FakeLLM + +pytestmark = pytest.mark.contract + +# Design-stage F1 improvement threshold (Issue #74 rubric check #6). +# The mock benchmark is a rubric-coverage tool, not the effect headline — +# it must clear a floor that would break loudly if a regression flattened +# the F1 gap. Set at +5% relative so the threshold survives small future +# additions to the scenario set without accidental green-washing. +DESIGN_F1_RELATIVE_GAIN_MIN = 0.05 + +# --------------------------------------------------------------------- schema + +SCHEMA: Dict[str, Any] = { + "propertykeys": [ + {"name": "name", "data_type": "TEXT", "cardinality": "SINGLE"}, + {"name": "age", "data_type": "INT", "cardinality": "SINGLE"}, + {"name": "title", "data_type": "TEXT", "cardinality": "SINGLE"}, + {"name": "year", "data_type": "INT", "cardinality": "SINGLE"}, + {"name": "role", "data_type": "TEXT", "cardinality": "SINGLE"}, + ], + "vertexlabels": [ + { + "id": 1, + "name": "Person", + "id_strategy": "PRIMARY_KEY", + "primary_keys": ["name"], + "properties": ["name", "age"], + "nullable_keys": ["age"], + }, + { + "id": 2, + "name": "Movie", + "id_strategy": "PRIMARY_KEY", + "primary_keys": ["title"], + "properties": ["title", "year"], + "nullable_keys": ["year"], + }, + ], + "edgelabels": [ + { + "name": "ACTED_IN", + "source_label": "Person", + "target_label": "Movie", + "properties": ["role"], + }, + ], +} + + +# Multi-primary-key schema used exclusively by s14. Keeping it isolated so the +# other nine scenarios remain readable against a single-PK mental model. +SCHEMA_MULTI_PK: Dict[str, Any] = { + "propertykeys": [ + {"name": "name", "data_type": "TEXT", "cardinality": "SINGLE"}, + {"name": "company", "data_type": "TEXT", "cardinality": "SINGLE"}, + {"name": "title", "data_type": "TEXT", "cardinality": "SINGLE"}, + ], + "vertexlabels": [ + { + "id": 3, + "name": "Employee", + "id_strategy": "PRIMARY_KEY", + "primary_keys": ["name", "company"], + "properties": ["name", "company", "title"], + "nullable_keys": ["title"], + }, + ], + "edgelabels": [], +} + + +# --------------------------------------------------------------- scenario type + + +@dataclass(frozen=True) +class Scenario: + """A benchmark scenario: LLM responses, chunks, and expected ground truth. + + ``schema`` overrides the module-level default when a scenario needs a + different vocabulary (currently only s14 for multi-primary-key coverage). + """ + + name: str + description: str + chunks: Sequence[str] + responses: Sequence[str] + expected: Mapping[str, Sequence[Mapping[str, Any]]] + schema: Optional[Mapping[str, Any]] = None + + +def _vertex(label: str, vid: str, **props): + return {"label": label, "type": "vertex", "id": vid, "properties": props} + + +def _edge(label: str, out_v: str, in_v: str, **props): + return { + "label": label, + "type": "edge", + "outV": out_v, + "inV": in_v, + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": props, + } + + +def _r(vertices, edges) -> str: + return json.dumps({"vertices": vertices, "edges": edges}, ensure_ascii=False) + + +# ------------------------------------------------------- benchmark scenarios + + +def _build_scenarios() -> List[Scenario]: + scenarios: List[Scenario] = [] + + # 1. Simple well-formed — both should be perfect. + scenarios.append( + Scenario( + name="s01_simple_well_formed", + description="Well-formed single-chunk extraction; both strategies must be perfect.", + chunks=["Tom Hanks starred in Forrest Gump."], + responses=[ + _r( + [ + {"label": "Person", "type": "vertex", "id": "v1", "properties": {"name": "Tom Hanks"}}, + {"label": "Movie", "type": "vertex", "id": "v2", "properties": {"title": "Forrest Gump"}}, + ], + [ + { + "label": "ACTED_IN", + "type": "edge", + "outV": "v1", + "inV": "v2", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {"role": "Forrest"}, + } + ], + ) + ], + expected={ + "vertices": [ + _vertex("Person", "1:Tom Hanks", name="Tom Hanks"), + _vertex("Movie", "2:Forrest Gump", title="Forrest Gump"), + ], + "edges": [_edge("ACTED_IN", "1:Tom Hanks", "2:Forrest Gump", role="Forrest")], + }, + ) + ) + + # 2. Extra property (not in schema) — both filter it out. + scenarios.append( + Scenario( + name="s02_extra_property_filtered", + description="LLM emits a property outside the schema; both strategies must drop it.", + chunks=["Tom Hanks is an actor."], + responses=[ + _r( + [ + { + "label": "Person", + "type": "vertex", + "id": "v1", + "properties": {"name": "Tom Hanks", "email": "tom@example.com"}, + } + ], + [], + ) + ], + expected={"vertices": [_vertex("Person", "1:Tom Hanks", name="Tom Hanks")], "edges": []}, + ) + ) + + # 3. Wrong property type — enhanced coerces "62" (str) → 62 (int); baseline keeps str. + scenarios.append( + Scenario( + name="s03_type_coercion_int", + description="LLM emits INT age as string; enhanced coerces, baseline keeps str.", + chunks=["Tom Hanks is 62."], + responses=[ + _r( + [ + { + "label": "Person", + "type": "vertex", + "id": "v1", + "properties": {"name": "Tom Hanks", "age": "62"}, + } + ], + [], + ) + ], + expected={"vertices": [_vertex("Person", "1:Tom Hanks", name="Tom Hanks", age=62)], "edges": []}, + ) + ) + + # 4. Missing primary key — both strategies drop the vertex. + scenarios.append( + Scenario( + name="s04_missing_primary_key", + description="LLM emits a Person with no name; both strategies must drop it.", + chunks=["Someone starred in Forrest Gump."], + responses=[ + _r( + [ + { + "label": "Person", + "type": "vertex", + "id": "v1", + "properties": {"age": 60}, + } + ], + [], + ) + ], + expected={"vertices": [], "edges": []}, + ) + ) + + # 5. Duplicate vertex across chunks — enhanced dedupes; baseline reports raw duplicates. + scenarios.append( + Scenario( + name="s05_duplicate_vertex_across_chunks", + description="Same Person mentioned in two chunks; enhanced dedupes.", + chunks=[ + "Tom Hanks starred in Forrest Gump.", + "Tom Hanks won an Oscar.", + ], + responses=[ + _r( + [ + {"label": "Person", "type": "vertex", "id": "v1", "properties": {"name": "Tom Hanks"}}, + ], + [], + ), + _r( + [ + {"label": "Person", "type": "vertex", "id": "v1", "properties": {"name": "Tom Hanks"}}, + ], + [], + ), + ], + expected={"vertices": [_vertex("Person", "1:Tom Hanks", name="Tom Hanks")], "edges": []}, + ) + ) + + # 6. Cross-chunk edge — enhanced repairs via alias table; baseline drops. + scenarios.append( + Scenario( + name="s06_cross_chunk_edge", + description="Edge references a vertex defined in a different chunk.", + chunks=[ + "Tom Hanks is an actor.", + "He starred in Forrest Gump as Forrest.", + ], + responses=[ + _r( + [ + {"label": "Person", "type": "vertex", "id": "v1", "properties": {"name": "Tom Hanks"}}, + ], + [], + ), + _r( + [ + {"label": "Movie", "type": "vertex", "id": "v2", "properties": {"title": "Forrest Gump"}}, + ], + [ + { + "label": "ACTED_IN", + "type": "edge", + "outV": "v1", + "inV": "v2", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {"role": "Forrest"}, + } + ], + ), + ], + expected={ + "vertices": [ + _vertex("Person", "1:Tom Hanks", name="Tom Hanks"), + _vertex("Movie", "2:Forrest Gump", title="Forrest Gump"), + ], + "edges": [_edge("ACTED_IN", "1:Tom Hanks", "2:Forrest Gump", role="Forrest")], + }, + ) + ) + + # 7. Alias mismatch — chunk 1 emits raw id, chunk 2 emits canonical id for same entity. + scenarios.append( + Scenario( + name="s07_alias_mismatch", + description="Chunk 1 uses raw id 'v1'; chunk 2 refers to the same entity by canonical id.", + chunks=[ + "Tom Hanks is an actor.", + "Tom Hanks starred in Forrest Gump.", + ], + responses=[ + _r( + [ + {"label": "Person", "type": "vertex", "id": "v1", "properties": {"name": "Tom Hanks"}}, + ], + [], + ), + _r( + [ + { + "label": "Movie", + "type": "vertex", + "id": "v2", + "properties": {"title": "Forrest Gump"}, + }, + ], + [ + { + "label": "ACTED_IN", + "type": "edge", + "outV": "1:Tom Hanks", + "inV": "v2", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {"role": "Forrest"}, + } + ], + ), + ], + expected={ + "vertices": [ + _vertex("Person", "1:Tom Hanks", name="Tom Hanks"), + _vertex("Movie", "2:Forrest Gump", title="Forrest Gump"), + ], + "edges": [_edge("ACTED_IN", "1:Tom Hanks", "2:Forrest Gump", role="Forrest")], + }, + ) + ) + + # 8. Invalid label — both strategies drop unknown labels. + scenarios.append( + Scenario( + name="s08_invalid_label", + description="LLM emits a label outside the schema; both strategies must drop it.", + chunks=["Directed by someone."], + responses=[ + _r( + [ + {"label": "Director", "type": "vertex", "id": "v1", "properties": {"name": "Someone"}}, + {"label": "Person", "type": "vertex", "id": "v2", "properties": {"name": "Tom Hanks"}}, + ], + [], + ) + ], + expected={"vertices": [_vertex("Person", "1:Tom Hanks", name="Tom Hanks")], "edges": []}, + ) + ) + + # 9. Malformed JSON — both strategies handle gracefully; neither emits vertices/edges. + scenarios.append( + Scenario( + name="s09_malformed_json", + description="LLM output not parseable as JSON; both strategies must return empty.", + chunks=["Empty chunk."], + responses=["Not JSON at all; the LLM went off script."], + expected={"vertices": [], "edges": []}, + ) + ) + + # 10. Combined win: coerce + cross-chunk edge. + scenarios.append( + Scenario( + name="s10_coerce_and_cross_chunk", + description="Cross-chunk edge with a string-typed INT property; enhanced wins on both.", + chunks=[ + "Tom Hanks is 62.", + "He starred in Forrest Gump released in 1994.", + ], + responses=[ + _r( + [ + { + "label": "Person", + "type": "vertex", + "id": "v1", + "properties": {"name": "Tom Hanks", "age": "62"}, + } + ], + [], + ), + _r( + [ + { + "label": "Movie", + "type": "vertex", + "id": "v2", + "properties": {"title": "Forrest Gump", "year": "1994"}, + }, + ], + [ + { + "label": "ACTED_IN", + "type": "edge", + "outV": "v1", + "inV": "v2", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {"role": "Forrest"}, + } + ], + ), + ], + expected={ + "vertices": [ + _vertex("Person", "1:Tom Hanks", name="Tom Hanks", age=62), + _vertex("Movie", "2:Forrest Gump", title="Forrest Gump", year=1994), + ], + "edges": [_edge("ACTED_IN", "1:Tom Hanks", "2:Forrest Gump", role="Forrest")], + }, + ) + ) + + # 11. Missing endpoint referent — edge points to a vertex that is never + # defined in any chunk. Both strategies must drop the edge. + scenarios.append( + Scenario( + name="s11_missing_endpoint_referent", + description="Edge references v99 which no chunk ever defines; both must drop it.", + chunks=["Tom Hanks starred in something unspecified."], + responses=[ + _r( + [ + {"label": "Person", "type": "vertex", "id": "v1", "properties": {"name": "Tom Hanks"}}, + ], + [ + { + "label": "ACTED_IN", + "type": "edge", + "outV": "v1", + "inV": "v99", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {"role": "Star"}, + } + ], + ) + ], + expected={"vertices": [_vertex("Person", "1:Tom Hanks", name="Tom Hanks")], "edges": []}, + ) + ) + + # 12. Wrong edge direction — schema requires Person → Movie; LLM inverts. + # Both strategies must drop the edge; the two vertices survive. + scenarios.append( + Scenario( + name="s12_wrong_edge_direction", + description="ACTED_IN emitted Movie → Person; endpoint labels violate schema direction.", + chunks=["Forrest Gump features Tom Hanks."], + responses=[ + _r( + [ + {"label": "Person", "type": "vertex", "id": "v1", "properties": {"name": "Tom Hanks"}}, + {"label": "Movie", "type": "vertex", "id": "v2", "properties": {"title": "Forrest Gump"}}, + ], + [ + { + "label": "ACTED_IN", + "type": "edge", + "outV": "v2", + "inV": "v1", + "outVLabel": "Movie", + "inVLabel": "Person", + "properties": {"role": "Forrest"}, + } + ], + ) + ], + expected={ + "vertices": [ + _vertex("Person", "1:Tom Hanks", name="Tom Hanks"), + _vertex("Movie", "2:Forrest Gump", title="Forrest Gump"), + ], + "edges": [], + }, + ) + ) + + # 13. Duplicate edge across chunks — same edge emitted in two chunks. + # Both get F1=1.0 (set-based) but enhanced dedupes at raw level too; + # baseline appends both, producing extra commit load downstream. + _dup_edge_response = _r( + [ + {"label": "Person", "type": "vertex", "id": "v1", "properties": {"name": "Tom Hanks"}}, + {"label": "Movie", "type": "vertex", "id": "v2", "properties": {"title": "Forrest Gump"}}, + ], + [ + { + "label": "ACTED_IN", + "type": "edge", + "outV": "v1", + "inV": "v2", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {"role": "Forrest"}, + } + ], + ) + scenarios.append( + Scenario( + name="s13_duplicate_edge_across_chunks", + description="Same ACTED_IN edge emitted in two chunks; enhanced dedupes at raw level.", + chunks=[ + "Tom Hanks starred in Forrest Gump.", + "Tom Hanks acted in Forrest Gump as Forrest.", + ], + responses=[_dup_edge_response, _dup_edge_response], + expected={ + "vertices": [ + _vertex("Person", "1:Tom Hanks", name="Tom Hanks"), + _vertex("Movie", "2:Forrest Gump", title="Forrest Gump"), + ], + "edges": [_edge("ACTED_IN", "1:Tom Hanks", "2:Forrest Gump", role="Forrest")], + }, + ) + ) + + # 14. Multi-primary-key vertex — Employee keyed by (name, company). + # Both strategies compute canonical id "3:Alice!Acme"; validates that + # the "!"-join rule works for composite PKs on both sides. + scenarios.append( + Scenario( + name="s14_multi_primary_key", + description="Employee with composite primary key (name, company).", + chunks=["Alice is a Principal Engineer at Acme Corp."], + responses=[ + _r( + [ + { + "label": "Employee", + "type": "vertex", + "id": "v1", + "properties": {"name": "Alice", "company": "Acme", "title": "Principal Engineer"}, + } + ], + [], + ) + ], + expected={ + "vertices": [ + { + "label": "Employee", + "type": "vertex", + "id": "3:Alice!Acme", + "properties": {"name": "Alice", "company": "Acme", "title": "Principal Engineer"}, + } + ], + "edges": [], + }, + schema=SCHEMA_MULTI_PK, + ) + ) + + # 15. Character promoted to Person — a semantic error the schema layer + # cannot catch. Enhanced has NO advantage here: "Chuck Noland" has a + # valid PK, passes every schema check, and neither strategy has + # access to the world knowledge that Chuck is a fictional character. + # This is the exact failure mode observed on the live DeepSeek run. + scenarios.append( + Scenario( + name="s15_character_promoted_to_person", + description=( + "LLM emits a fictional character as a Person vertex. Schema-valid " + "but semantically wrong; neither strategy can drop it." + ), + chunks=["In Cast Away, Tom Hanks played Chuck Noland, a FedEx executive."], + responses=[ + _r( + [ + {"label": "Person", "type": "vertex", "id": "v1", "properties": {"name": "Tom Hanks"}}, + {"label": "Movie", "type": "vertex", "id": "v2", "properties": {"title": "Cast Away"}}, + {"label": "Person", "type": "vertex", "id": "v3", "properties": {"name": "Chuck Noland"}}, + ], + [ + { + "label": "ACTED_IN", + "type": "edge", + "outV": "v1", + "inV": "v2", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {"role": "Chuck Noland"}, + } + ], + ) + ], + expected={ + "vertices": [ + _vertex("Person", "1:Tom Hanks", name="Tom Hanks"), + _vertex("Movie", "2:Cast Away", title="Cast Away"), + ], + "edges": [ + _edge("ACTED_IN", "1:Tom Hanks", "2:Cast Away", role="Chuck Noland"), + ], + }, + ) + ) + + # 16. Pronoun promoted to Person, no prior chunk to resolve the alias + # against. Enhanced's document assembler can only repair across + # chunks; a single-chunk pronoun ghost gets kept by both. Its + # spurious outgoing edge also gets kept by both — F1 tanks equally. + scenarios.append( + Scenario( + name="s16_pronoun_ghost_no_prior_context", + description=( + "Single-chunk pronoun-as-Person with no earlier antecedent. " + "Enhanced's cross-chunk alias table has nothing to link to; " + "both strategies emit the ghost Person and its edge." + ), + chunks=["He voiced Woody in the Toy Story franchise."], + responses=[ + _r( + [ + {"label": "Person", "type": "vertex", "id": "v1", "properties": {"name": "He"}}, + {"label": "Movie", "type": "vertex", "id": "v2", "properties": {"title": "Toy Story"}}, + ], + [ + { + "label": "ACTED_IN", + "type": "edge", + "outV": "v1", + "inV": "v2", + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {"role": "Woody"}, + } + ], + ) + ], + expected={ + "vertices": [ + _vertex("Movie", "2:Toy Story", title="Toy Story"), + ], + "edges": [], + }, + ) + ) + + return scenarios + + +# ------------------------------------------------------------ runtime harness + + +def _run(scenario: Scenario, *, strategy: str) -> Tuple[Dict[str, Any], float, int]: + """Runs one strategy against a scenario; returns (graph, latency_ms, call_count). + + Latency includes JSON parsing, normalization, and assembly (for enhanced) or + the baseline's per-chunk normalization pass. With FakeLLM the LLM call is a + dict pop, so the measurement is dominated by post-LLM processing — a useful + signal for the pipeline overhead added by the enhanced strategy. + """ + schema = scenario.schema if scenario.schema is not None else SCHEMA + llm = FakeLLM(scenario.responses) + extractor = PropertyGraphExtract(llm=llm, example_prompt="", extract_strategy=strategy) + context = {"schema": schema, "chunks": list(scenario.chunks)} + start = time.perf_counter() + result = extractor.run(context) + elapsed_ms = (time.perf_counter() - start) * 1000.0 + return ( + {"vertices": result.get("vertices", []), "edges": result.get("edges", [])}, + elapsed_ms, + int(result.get("call_count", 0)), + ) + + +def _evaluate(scenario: Scenario, predicted: Mapping[str, Any]) -> EvaluationReport: + schema = scenario.schema if scenario.schema is not None else SCHEMA + evaluator = GraphExtractionEvaluator(GraphSchemaIndex(schema)) + return evaluator.evaluate(predicted, scenario.expected) + + +@pytest.fixture(scope="module") +def scenarios() -> List[Scenario]: + return _build_scenarios() + + +# --------------------------------------------------------------- benchmark + + +def test_benchmark_produces_comparison_table(scenarios, capsys): + """Runs every scenario against both strategies and prints a Markdown table. + + Columns reported: + + * ``b F1`` / ``e F1`` — overall structural F1 for baseline vs. enhanced. + * ``b match%`` / ``e match%`` — property_exact_match_rate on TP items. + * ``calls`` — LLM invocation count (equal by design: one per chunk). + * ``b ms`` / ``e ms`` — post-LLM processing latency (FakeLLM makes the LLM + call itself effectively free, so these are enhanced-vs-baseline pipeline + overhead numbers, not end-to-end). + * ``b vraw`` / ``e vraw`` — raw predicted vertex counts (enhanced dedupes + across chunks; baseline does not). + + Use ``pytest -s`` to see the table locally. + """ + header = "| scenario | b F1 | e F1 | b match% | e match% | calls | b ms | e ms | b vraw | e vraw |" + separator = "|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|" + rows: List[str] = [header, separator] + baseline_f1_sum = 0.0 + enhanced_f1_sum = 0.0 + baseline_ms_sum = 0.0 + enhanced_ms_sum = 0.0 + total_calls = 0 + for scenario in scenarios: + baseline_pred, baseline_ms, baseline_calls = _run(scenario, strategy="baseline") + enhanced_pred, enhanced_ms, enhanced_calls = _run(scenario, strategy="enhanced") + baseline_report = _evaluate(scenario, baseline_pred) + enhanced_report = _evaluate(scenario, enhanced_pred) + + # Both strategies make one LLM call per chunk — verify the invariant. + assert baseline_calls == enhanced_calls == len(scenario.chunks), ( + f"{scenario.name}: expected call_count == {len(scenario.chunks)} for both strategies, " + f"got baseline={baseline_calls}, enhanced={enhanced_calls}" + ) + + rows.append( + "| {name} | {bf1:.2f} | {ef1:.2f} | {bm:.2f} | {em:.2f} | {c} | {bms:.2f} | {ems:.2f} | {bvraw} | {evraw} |".format( + name=scenario.name, + bf1=baseline_report.overall_f1, + ef1=enhanced_report.overall_f1, + bm=baseline_report.property_metrics.property_exact_match_rate, + em=enhanced_report.property_metrics.property_exact_match_rate, + c=baseline_calls, + bms=baseline_ms, + ems=enhanced_ms, + bvraw=baseline_report.vertex_metrics.predicted_count_raw, + evraw=enhanced_report.vertex_metrics.predicted_count_raw, + ) + ) + baseline_f1_sum += baseline_report.overall_f1 + enhanced_f1_sum += enhanced_report.overall_f1 + baseline_ms_sum += baseline_ms + enhanced_ms_sum += enhanced_ms + total_calls += baseline_calls + + # Design invariant: enhanced never regresses F1. + assert enhanced_report.overall_f1 + 1e-9 >= baseline_report.overall_f1, ( + f"{scenario.name}: enhanced F1 {enhanced_report.overall_f1} regressed vs " + f"baseline {baseline_report.overall_f1}" + ) + + n = len(scenarios) + avg_baseline = baseline_f1_sum / n + avg_enhanced = enhanced_f1_sum / n + f1_relative_gain_pct = (avg_enhanced - avg_baseline) / avg_baseline * 100.0 if avg_baseline else 0.0 + rows.append( + "| **average / total** | **{ab:.2f}** | **{ae:.2f}** | | | **{c}** | **{bms:.2f}** | **{ems:.2f}** | | |".format( + ab=avg_baseline, + ae=avg_enhanced, + c=total_calls, + bms=baseline_ms_sum, + ems=enhanced_ms_sum, + ) + ) + rows.append("| **F1 relative gain** | | **+{pct:.1f}%** | | | | | | | |".format(pct=f1_relative_gain_pct)) + # Print to stdout so `pytest -s` captures the exact table to paste into the report. + print("\n" + "\n".join(rows)) + assert avg_enhanced >= avg_baseline + # Enforce the design-stage F1-improvement threshold documented in the report. + # This is a floor; regressions below +5% relative must fail CI. + if avg_baseline > 0: + actual_relative_gain = (avg_enhanced - avg_baseline) / avg_baseline + assert actual_relative_gain >= DESIGN_F1_RELATIVE_GAIN_MIN, ( + f"design-stage F1 improvement threshold not met: got " + f"{actual_relative_gain * 100:.2f}% relative, " + f"need >= {DESIGN_F1_RELATIVE_GAIN_MIN * 100:.1f}%" + ) + + +def test_enhanced_wins_on_cross_chunk_edge(scenarios): + scenario = next(s for s in scenarios if s.name == "s06_cross_chunk_edge") + baseline_pred, _, _ = _run(scenario, strategy="baseline") + enhanced_pred, _, _ = _run(scenario, strategy="enhanced") + baseline_report = _evaluate(scenario, baseline_pred) + enhanced_report = _evaluate(scenario, enhanced_pred) + assert baseline_report.edge_metrics.f1 == 0.0 + assert enhanced_report.edge_metrics.f1 == 1.0 + + +def test_enhanced_wins_on_alias_mismatch(scenarios): + scenario = next(s for s in scenarios if s.name == "s07_alias_mismatch") + baseline_pred, _, _ = _run(scenario, strategy="baseline") + enhanced_pred, _, _ = _run(scenario, strategy="enhanced") + baseline_report = _evaluate(scenario, baseline_pred) + enhanced_report = _evaluate(scenario, enhanced_pred) + assert baseline_report.edge_metrics.f1 == 0.0 + assert enhanced_report.edge_metrics.f1 == 1.0 + + +def test_enhanced_wins_on_type_coercion(scenarios): + scenario = next(s for s in scenarios if s.name == "s03_type_coercion_int") + baseline_pred, _, _ = _run(scenario, strategy="baseline") + enhanced_pred, _, _ = _run(scenario, strategy="enhanced") + baseline_report = _evaluate(scenario, baseline_pred) + enhanced_report = _evaluate(scenario, enhanced_pred) + assert enhanced_report.property_metrics.property_exact_match_rate == 1.0 + assert baseline_report.property_metrics.property_exact_match_rate < 1.0 + + +def test_enhanced_matches_baseline_on_well_formed_input(scenarios): + scenario = next(s for s in scenarios if s.name == "s01_simple_well_formed") + baseline_pred, _, _ = _run(scenario, strategy="baseline") + enhanced_pred, _, _ = _run(scenario, strategy="enhanced") + baseline_report = _evaluate(scenario, baseline_pred) + enhanced_report = _evaluate(scenario, enhanced_pred) + assert baseline_report.overall_f1 == 1.0 + assert enhanced_report.overall_f1 == 1.0 + + +def test_missing_endpoint_referent_dropped_by_both(scenarios): + scenario = next(s for s in scenarios if s.name == "s11_missing_endpoint_referent") + baseline_pred, _, _ = _run(scenario, strategy="baseline") + enhanced_pred, _, _ = _run(scenario, strategy="enhanced") + assert _evaluate(scenario, baseline_pred).overall_f1 == 1.0 + assert _evaluate(scenario, enhanced_pred).overall_f1 == 1.0 + assert len(baseline_pred["edges"]) == 0 + assert len(enhanced_pred["edges"]) == 0 + + +def test_wrong_edge_direction_dropped_by_both(scenarios): + scenario = next(s for s in scenarios if s.name == "s12_wrong_edge_direction") + baseline_pred, _, _ = _run(scenario, strategy="baseline") + enhanced_pred, _, _ = _run(scenario, strategy="enhanced") + assert len(baseline_pred["edges"]) == 0 + assert len(enhanced_pred["edges"]) == 0 + + +def test_duplicate_edge_deduped_only_by_enhanced(scenarios): + scenario = next(s for s in scenarios if s.name == "s13_duplicate_edge_across_chunks") + baseline_pred, _, _ = _run(scenario, strategy="baseline") + enhanced_pred, _, _ = _run(scenario, strategy="enhanced") + # Both have equal set-based F1, but baseline appends the duplicate raw copy. + assert len(baseline_pred["edges"]) == 2, "baseline is expected to keep raw duplicates" + assert len(enhanced_pred["edges"]) == 1, "enhanced is expected to dedupe raw duplicates" + + +def test_multi_primary_key_handled_by_both(scenarios): + scenario = next(s for s in scenarios if s.name == "s14_multi_primary_key") + baseline_pred, _, _ = _run(scenario, strategy="baseline") + enhanced_pred, _, _ = _run(scenario, strategy="enhanced") + # Both should produce canonical id "3:Alice!Acme". + assert baseline_pred["vertices"][0]["id"] == "3:Alice!Acme" + assert enhanced_pred["vertices"][0]["id"] == "3:Alice!Acme" + + +def test_character_promoted_to_person_hurts_both_strategies(scenarios): + """s15: enhanced has no advantage over baseline on semantic errors. + + "Chuck Noland" is a fictional character with a schema-valid ``name``. + Neither strategy has the world knowledge to distinguish it from a real + Person. Both keep the ghost vertex, both take an equal precision hit — + documented so the report can honestly disclose enhanced's ceiling. + """ + scenario = next(s for s in scenarios if s.name == "s15_character_promoted_to_person") + baseline_pred, _, _ = _run(scenario, strategy="baseline") + enhanced_pred, _, _ = _run(scenario, strategy="enhanced") + baseline_report = _evaluate(scenario, baseline_pred) + enhanced_report = _evaluate(scenario, enhanced_pred) + # Both fail: neither reaches F1=1.0. + assert baseline_report.overall_f1 < 1.0 + assert enhanced_report.overall_f1 < 1.0 + # Both fail EQUALLY: enhanced has no schema-level lever to catch this. + assert abs(enhanced_report.overall_f1 - baseline_report.overall_f1) < 1e-9 + + +def test_pronoun_ghost_hurts_both_when_no_prior_context(scenarios): + """s16: enhanced's cross-chunk alias table cannot help a first-chunk pronoun. + + Enhanced beats baseline on s06/s07 because it can look back across + chunks. When the pronoun appears in the *first* chunk with no + antecedent, the alias table is empty and enhanced cannot repair the + edge either. Both strategies emit the ghost Person and the spurious edge. + """ + scenario = next(s for s in scenarios if s.name == "s16_pronoun_ghost_no_prior_context") + baseline_pred, _, _ = _run(scenario, strategy="baseline") + enhanced_pred, _, _ = _run(scenario, strategy="enhanced") + baseline_report = _evaluate(scenario, baseline_pred) + enhanced_report = _evaluate(scenario, enhanced_pred) + assert baseline_report.overall_f1 < 1.0 + assert enhanced_report.overall_f1 < 1.0 + assert abs(enhanced_report.overall_f1 - baseline_report.overall_f1) < 1e-9 diff --git a/hugegraph-llm/src/tests/operators/llm_op/test_property_graph_evaluator.py b/hugegraph-llm/src/tests/operators/llm_op/test_property_graph_evaluator.py new file mode 100644 index 000000000..f9afdbb8b --- /dev/null +++ b/hugegraph-llm/src/tests/operators/llm_op/test_property_graph_evaluator.py @@ -0,0 +1,308 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +"""Unit tests for :mod:`property_graph_extract_enhanced.evaluator`. + +These tests treat the evaluator as strategy-agnostic: they feed pre-built +predicted and expected graphs and verify F1 / property-fidelity math. The +benchmark that actually drives baseline vs. enhanced pipelines lives in +``test_property_graph_benchmark.py``. +""" + +from __future__ import annotations + +import pytest + +from hugegraph_llm.operators.llm_op.property_graph_extract_enhanced import ( + GraphExtractionEvaluator, + GraphSchemaIndex, +) + +pytestmark = pytest.mark.contract + + +def _schema() -> dict: + return { + "propertykeys": [ + {"name": "name", "data_type": "TEXT", "cardinality": "SINGLE"}, + {"name": "age", "data_type": "INT", "cardinality": "SINGLE"}, + {"name": "title", "data_type": "TEXT", "cardinality": "SINGLE"}, + {"name": "year", "data_type": "INT", "cardinality": "SINGLE"}, + {"name": "role", "data_type": "TEXT", "cardinality": "SINGLE"}, + ], + "vertexlabels": [ + { + "id": 1, + "name": "Person", + "id_strategy": "PRIMARY_KEY", + "primary_keys": ["name"], + "properties": ["name", "age"], + "nullable_keys": ["age"], + }, + { + "id": 2, + "name": "Movie", + "id_strategy": "PRIMARY_KEY", + "primary_keys": ["title"], + "properties": ["title", "year"], + "nullable_keys": ["year"], + }, + ], + "edgelabels": [ + { + "name": "ACTED_IN", + "source_label": "Person", + "target_label": "Movie", + "properties": ["role"], + }, + ], + } + + +@pytest.fixture() +def evaluator() -> GraphExtractionEvaluator: + return GraphExtractionEvaluator(GraphSchemaIndex(_schema())) + + +def _vertex(label: str, vid: str, **props): + return {"label": label, "type": "vertex", "id": vid, "properties": props} + + +def _edge(label: str, out_v: str, in_v: str, out_label: str, in_label: str, **props): + return { + "label": label, + "type": "edge", + "outV": out_v, + "inV": in_v, + "outVLabel": out_label, + "inVLabel": in_label, + "properties": props, + } + + +# -------------------------------------------------------------- structural F1 + + +def test_perfect_prediction_yields_f1_one(evaluator): + predicted = { + "vertices": [_vertex("Person", "1:Tom", name="Tom"), _vertex("Movie", "2:X", title="X")], + "edges": [_edge("ACTED_IN", "1:Tom", "2:X", "Person", "Movie")], + } + expected = predicted + report = evaluator.evaluate(predicted, expected) + assert report.vertex_metrics.f1 == 1.0 + assert report.edge_metrics.f1 == 1.0 + assert report.overall_f1 == 1.0 + assert report.overall_precision == 1.0 + assert report.overall_recall == 1.0 + + +def test_both_empty_treated_as_perfect_match(evaluator): + report = evaluator.evaluate({"vertices": [], "edges": []}, {"vertices": [], "edges": []}) + assert report.vertex_metrics.f1 == 1.0 + assert report.edge_metrics.f1 == 1.0 + assert report.overall_f1 == 1.0 + assert report.property_metrics.property_valid_ratio == 1.0 + assert report.property_metrics.property_exact_match_rate == 1.0 + + +def test_missing_predictions_have_zero_recall(evaluator): + predicted = {"vertices": [], "edges": []} + expected = { + "vertices": [_vertex("Person", "1:Tom", name="Tom")], + "edges": [], + } + report = evaluator.evaluate(predicted, expected) + assert report.vertex_metrics.precision == 1.0 + assert report.vertex_metrics.recall == 0.0 + assert report.vertex_metrics.f1 == 0.0 + assert report.overall_f1 == 0.0 + + +def test_hallucinated_predictions_have_zero_precision(evaluator): + predicted = { + "vertices": [_vertex("Person", "1:Ghost", name="Ghost")], + "edges": [], + } + expected = {"vertices": [], "edges": []} + report = evaluator.evaluate(predicted, expected) + assert report.vertex_metrics.precision == 0.0 + assert report.vertex_metrics.recall == 1.0 + assert report.vertex_metrics.f1 == 0.0 + assert report.overall_f1 == 0.0 + + +def test_half_recall_reflected_in_f1(evaluator): + predicted = {"vertices": [_vertex("Person", "1:Tom", name="Tom")], "edges": []} + expected = { + "vertices": [ + _vertex("Person", "1:Tom", name="Tom"), + _vertex("Person", "1:Meg", name="Meg"), + ], + "edges": [], + } + report = evaluator.evaluate(predicted, expected) + assert report.vertex_metrics.precision == 1.0 + assert report.vertex_metrics.recall == 0.5 + assert report.vertex_metrics.f1 == pytest.approx(2 / 3) + + +def test_duplicate_predictions_deduped_for_f1_but_raw_count_preserved(evaluator): + dup = _vertex("Person", "1:Tom", name="Tom") + predicted = {"vertices": [dup, dict(dup)], "edges": []} + expected = {"vertices": [_vertex("Person", "1:Tom", name="Tom")], "edges": []} + report = evaluator.evaluate(predicted, expected) + assert report.vertex_metrics.predicted_count_raw == 2 + assert report.vertex_metrics.predicted_count_unique == 1 + assert report.vertex_metrics.f1 == 1.0 + + +def test_edges_matched_by_label_and_endpoints(evaluator): + predicted = { + "vertices": [], + "edges": [_edge("ACTED_IN", "1:Tom", "2:X", "Person", "Movie", role="Star")], + } + expected = { + "vertices": [], + "edges": [_edge("ACTED_IN", "1:Tom", "2:Y", "Person", "Movie", role="Star")], + } + report = evaluator.evaluate(predicted, expected) + assert report.edge_metrics.f1 == 0.0 + + +# ------------------------------------------------------------ property metrics + + +def test_property_valid_ratio_excludes_extra_property(evaluator): + predicted = { + "vertices": [_vertex("Person", "1:Tom", name="Tom", email="tom@example.com")], + "edges": [], + } + expected = {"vertices": [_vertex("Person", "1:Tom", name="Tom")], "edges": []} + report = evaluator.evaluate(predicted, expected) + # 2 predicted properties (name, email); 1 valid (name). + assert report.property_metrics.predicted_property_count == 2 + assert report.property_metrics.valid_property_count == 1 + assert report.property_metrics.property_valid_ratio == 0.5 + + +def test_property_exact_match_penalizes_type_mismatch(evaluator): + # Same vertex matches structurally; property comparison is type-strict. + predicted = { + "vertices": [_vertex("Person", "1:Tom", name="Tom", age="62")], + "edges": [], + } + expected = { + "vertices": [_vertex("Person", "1:Tom", name="Tom", age=62)], + "edges": [], + } + report = evaluator.evaluate(predicted, expected) + # 1 TP vertex, 2 predicted props (name, age), 2 expected props; + # only name (Tom == Tom) exact matches. + assert report.property_metrics.expected_tp_property_count == 2 + assert report.property_metrics.predicted_tp_property_count == 2 + assert report.property_metrics.exact_match_property_count == 1 + assert report.property_metrics.property_exact_match_rate == 0.5 + + +def test_property_exact_match_all_when_types_match(evaluator): + predicted = { + "vertices": [_vertex("Person", "1:Tom", name="Tom", age=62)], + "edges": [], + } + expected = predicted + report = evaluator.evaluate(predicted, expected) + assert report.property_metrics.property_exact_match_rate == 1.0 + assert report.property_metrics.property_valid_ratio == 1.0 + + +# -------------------------------------------------------------- key extraction + + +def test_vertex_without_id_falls_back_to_canonical(evaluator): + predicted = { + "vertices": [ + # No id field; canonical id should be reconstructed from PK. + {"label": "Person", "type": "vertex", "properties": {"name": "Tom"}} + ], + "edges": [], + } + expected = {"vertices": [_vertex("Person", "1:Tom", name="Tom")], "edges": []} + report = evaluator.evaluate(predicted, expected) + assert report.vertex_metrics.true_positive_count == 1 + assert report.vertex_metrics.f1 == 1.0 + + +def test_vertex_without_id_or_pk_is_skipped(evaluator): + predicted = { + "vertices": [ + # No id, no name → canonical id cannot be computed → excluded. + {"label": "Person", "type": "vertex", "properties": {"age": 62}} + ], + "edges": [], + } + expected = {"vertices": [], "edges": []} + report = evaluator.evaluate(predicted, expected) + assert report.vertex_metrics.predicted_count_raw == 0 + + +def test_edge_without_endpoints_is_skipped(evaluator): + predicted = { + "vertices": [], + "edges": [ + # Missing outV → not a valid comparable item. + {"label": "ACTED_IN", "type": "edge", "inV": "2:X", "properties": {}} + ], + } + expected = {"vertices": [], "edges": []} + report = evaluator.evaluate(predicted, expected) + assert report.edge_metrics.predicted_count_raw == 0 + + +def test_evaluation_report_to_dict_is_serializable(evaluator): + report = evaluator.evaluate( + {"vertices": [_vertex("Person", "1:Tom", name="Tom")], "edges": []}, + {"vertices": [_vertex("Person", "1:Tom", name="Tom")], "edges": []}, + ) + payload = report.to_dict() + # to_dict is nested-serializable; check a handful of leaves rather than + # locking every key name (avoids brittle golden output). + assert payload["overall_f1"] == 1.0 + assert payload["vertex_metrics"]["true_positive_count"] == 1 + assert "property_valid_ratio" in payload["property_metrics"] + + +# ---------------------------------------------------------- overall precision + + +def test_overall_precision_and_recall_combine_vertices_and_edges(evaluator): + predicted = { + "vertices": [_vertex("Person", "1:Tom", name="Tom"), _vertex("Movie", "2:X", title="X")], + "edges": [ + _edge("ACTED_IN", "1:Tom", "2:X", "Person", "Movie"), + _edge("ACTED_IN", "1:Tom", "2:Ghost", "Person", "Movie"), + ], + } + expected = { + "vertices": [_vertex("Person", "1:Tom", name="Tom"), _vertex("Movie", "2:X", title="X")], + "edges": [_edge("ACTED_IN", "1:Tom", "2:X", "Person", "Movie")], + } + report = evaluator.evaluate(predicted, expected) + # 2 TP vertices + 1 TP edge = 3 TP out of (2+2)=4 predicted, (2+1)=3 expected + assert report.overall_precision == pytest.approx(0.75) + assert report.overall_recall == 1.0 + assert report.overall_f1 == pytest.approx(6 / 7) diff --git a/hugegraph-llm/src/tests/operators/llm_op/test_property_graph_extract.py b/hugegraph-llm/src/tests/operators/llm_op/test_property_graph_extract.py index efb1e28fe..4256687b6 100644 --- a/hugegraph-llm/src/tests/operators/llm_op/test_property_graph_extract.py +++ b/hugegraph-llm/src/tests/operators/llm_op/test_property_graph_extract.py @@ -134,6 +134,235 @@ def test_init(self): self.assertEqual(extractor.llm, self.mock_llm) self.assertEqual(extractor.example_prompt, custom_prompt) self.assertEqual(extractor.NECESSARY_ITEM_KEYS, {"label", "type", "properties"}) + # Default plumbing values for the enhanced strategy stay opt-in. + self.assertEqual(extractor.extract_strategy, "baseline") + self.assertFalse(extractor.include_debug) + + def test_init_captures_extract_strategy_and_include_debug(self): + """Enhanced strategy flags flow through the constructor when provided.""" + extractor = PropertyGraphExtract( + llm=self.mock_llm, + example_prompt="prompt", + extract_strategy="enhanced", + include_debug=True, + ) + self.assertEqual(extractor.extract_strategy, "enhanced") + self.assertTrue(extractor.include_debug) + + def test_init_normalizes_unknown_strategy_to_baseline(self): + """Unknown strategy values must not silently activate an unsupported path.""" + extractor = PropertyGraphExtract( + llm=self.mock_llm, + example_prompt="prompt", + extract_strategy="aggressive", + ) + self.assertEqual(extractor.extract_strategy, "baseline") + + def test_run_records_extract_strategy_in_context(self): + """run() surfaces the resolved strategy for downstream API meta consumers.""" + extractor = PropertyGraphExtract( + llm=self.mock_llm, + example_prompt="prompt", + extract_strategy="enhanced", + ) + self.mock_llm.generate.return_value = json.dumps({"vertices": [], "edges": []}) + context = {"schema": self.schema, "chunks": ["chunk-1"]} + result = extractor.run(context) + self.assertEqual(result["extract_strategy"], "enhanced") + + # ------------------------------------------------------ enhanced strategy + def test_enhanced_strategy_runs_full_pipeline_and_returns_canonical_ids(self): + """Enhanced end-to-end: parse → normalize → assemble → quality gate.""" + # A local schema with propertykeys so year is explicitly typed INT, + # exercising the coerce path. The default self.schema has no + # propertykeys and defaults every property to TEXT/SINGLE. + typed_schema = { + **self.schema, + "propertykeys": [ + {"name": "name", "data_type": "TEXT", "cardinality": "SINGLE"}, + {"name": "title", "data_type": "TEXT", "cardinality": "SINGLE"}, + {"name": "year", "data_type": "INT", "cardinality": "SINGLE"}, + {"name": "age", "data_type": "INT", "cardinality": "SINGLE"}, + {"name": "role", "data_type": "TEXT", "cardinality": "SINGLE"}, + ], + } + extractor = PropertyGraphExtract( + llm=self.mock_llm, + example_prompt="prompt", + extract_strategy="enhanced", + ) + # LLM emits a valid grouped payload referencing vertices by LLM raw id. + self.mock_llm.generate.return_value = json.dumps( + { + "vertices": [ + { + "type": "vertex", + "id": "v1", + "label": "person", + "properties": {"name": "Tom Hanks"}, + }, + { + "type": "vertex", + "id": "v2", + "label": "movie", + "properties": {"title": "Forrest Gump", "year": "1994"}, + }, + ], + "edges": [ + { + "type": "edge", + "label": "acted_in", + "outV": "v1", + "inV": "v2", + "outVLabel": "person", + "inVLabel": "movie", + "properties": {"role": "Forrest"}, + } + ], + } + ) + result = extractor.run({"schema": typed_schema, "chunks": ["chunk-0"]}) + + # Vertices carry canonical ids computed from the schema. + self.assertEqual(len(result["vertices"]), 2) + vertex_ids = sorted(v["id"] for v in result["vertices"]) + self.assertIn("1:Tom Hanks", vertex_ids) + self.assertIn("2:Forrest Gump", vertex_ids) + + # Edge endpoints resolved to canonical form. + self.assertEqual(len(result["edges"]), 1) + self.assertEqual(result["edges"][0]["outV"], "1:Tom Hanks") + self.assertEqual(result["edges"][0]["inV"], "2:Forrest Gump") + + # Enhanced meta fields populated in the returned context. + self.assertEqual(result["extract_strategy"], "enhanced") + self.assertEqual(result["chunk_count"], 1) + self.assertIn("structured_warnings", result) + self.assertIn("quality_metrics", result) + # Year coerce from "1994" → 1994 should have emitted PROPERTY_COERCED. + codes = {w["code"] for w in result["structured_warnings"]} + self.assertIn("PROPERTY_COERCED", codes) + # The coerced value is reflected in the output. + movie = next(v for v in result["vertices"] if v["label"] == "movie") + self.assertEqual(movie["properties"]["year"], 1994) + + def test_enhanced_strategy_prompt_appends_constraint_block(self): + """The constraint block must reach the LLM prompt in enhanced mode.""" + extractor = PropertyGraphExtract( + llm=self.mock_llm, + example_prompt="EXAMPLE_HEADER", + extract_strategy="enhanced", + ) + self.mock_llm.generate.return_value = json.dumps({"vertices": [], "edges": []}) + extractor.run({"schema": self.schema, "chunks": ["chunk-0"]}) + + called_prompt = self.mock_llm.generate.call_args.kwargs["prompt"] + # The caller's example header stays, and the constraint block was appended. + self.assertIn("EXAMPLE_HEADER", called_prompt) + self.assertIn("Enhanced-strategy constraints", called_prompt) + # Schema-declared labels are enumerated in the constraint block. + self.assertIn("person", called_prompt) + self.assertIn("acted_in", called_prompt) + + def test_enhanced_strategy_drops_schema_invalid_items_with_warnings(self): + extractor = PropertyGraphExtract(llm=self.mock_llm, example_prompt="p", extract_strategy="enhanced") + # Vertex label 'robot' is not in the schema → dropped. + self.mock_llm.generate.return_value = json.dumps( + { + "vertices": [{"type": "vertex", "label": "robot", "properties": {"name": "R2D2"}}], + "edges": [], + } + ) + result = extractor.run({"schema": self.schema, "chunks": ["c"]}) + + self.assertEqual(result["vertices"], []) + codes = {w["code"] for w in result["structured_warnings"]} + self.assertIn("VERTEX_LABEL_NOT_IN_SCHEMA", codes) + + def test_enhanced_strategy_dedupes_across_chunks(self): + """Two chunks emit the same vertex+edge → assembler merges/dedupes.""" + extractor = PropertyGraphExtract(llm=self.mock_llm, example_prompt="p", extract_strategy="enhanced") + chunk_payload = json.dumps( + { + "vertices": [ + {"type": "vertex", "label": "person", "properties": {"name": "Tom"}}, + { + "type": "vertex", + "label": "movie", + "properties": {"title": "FG", "year": 1994}, + }, + ], + "edges": [ + { + "type": "edge", + "label": "acted_in", + "source": {"label": "person", "properties": {"name": "Tom"}}, + "target": { + "label": "movie", + "properties": {"title": "FG", "year": 1994}, + }, + "properties": {"role": "Forrest"}, + } + ], + } + ) + self.mock_llm.generate.return_value = chunk_payload + + result = extractor.run({"schema": self.schema, "chunks": ["c0", "c1"]}) + + # After merge/dedup only one vertex per (label,id) and one edge survives. + self.assertEqual(len(result["vertices"]), 2) + self.assertEqual(len(result["edges"]), 1) + self.assertEqual(result["call_count"], 2) + self.assertEqual(result["chunk_count"], 2) + codes = [w["code"] for w in result["structured_warnings"]] + self.assertIn("DUPLICATE_VERTEX_MERGED", codes) + self.assertIn("DUPLICATE_EDGE_MERGED", codes) + + def test_enhanced_strategy_include_debug_records_per_chunk_info(self): + extractor = PropertyGraphExtract( + llm=self.mock_llm, + example_prompt="p", + extract_strategy="enhanced", + include_debug=True, + ) + self.mock_llm.generate.return_value = json.dumps( + { + "vertices": [{"type": "vertex", "label": "person", "properties": {"name": "Tom"}}], + "edges": [], + } + ) + result = extractor.run({"schema": self.schema, "chunks": ["c0", "c1"]}) + + debug = result.get("debug_info") + self.assertIsNotNone(debug) + self.assertEqual(len(debug["chunks"]), 2) + self.assertIn("raw_output", debug["chunks"][0]) + self.assertIn("candidate_vertex_count", debug["chunks"][0]) + self.assertIn("warning_code_distribution", debug) + + def test_enhanced_strategy_populates_quality_metrics_shape(self): + extractor = PropertyGraphExtract(llm=self.mock_llm, example_prompt="p", extract_strategy="enhanced") + self.mock_llm.generate.return_value = json.dumps( + { + "vertices": [{"type": "vertex", "label": "person", "properties": {"name": "Tom"}}], + "edges": [], + } + ) + result = extractor.run({"schema": self.schema, "chunks": ["c"]}) + metrics = result["quality_metrics"] + for key in ( + "schema_valid_vertex_ratio", + "schema_valid_edge_ratio", + "endpoint_resolution_rate", + "duplicate_vertex_reduction", + "duplicate_edge_reduction", + "property_valid_ratio", + "dropped_item_count", + "coerced_property_count", + "endpoint_repair_count", + ): + self.assertIn(key, metrics) def test_generate_extract_property_graph_prompt(self): """Test the generate_extract_property_graph_prompt function.""" diff --git a/hugegraph-llm/src/tests/operators/llm_op/test_property_graph_postprocess.py b/hugegraph-llm/src/tests/operators/llm_op/test_property_graph_postprocess.py new file mode 100644 index 000000000..a4db25b0b --- /dev/null +++ b/hugegraph-llm/src/tests/operators/llm_op/test_property_graph_postprocess.py @@ -0,0 +1,1013 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +"""Unit tests for CandidateGraphParser and CandidateGraph (Issue #74).""" + +from __future__ import annotations + +import json +from typing import List + +import pytest + +from hugegraph_llm.operators.llm_op.property_graph_extract_enhanced import ( + PENDING_IN_KEY, + PENDING_OUT_KEY, + CandidateGraph, + CandidateGraphParser, + DocumentGraphAssembler, + GraphSchemaIndex, + NormalizedChunkGraph, + SchemaAwareNormalizer, + StructuredWarning, + WarningCode, +) + +pytestmark = pytest.mark.contract + + +def _codes(warnings: List[StructuredWarning]) -> List[WarningCode]: + return [w.code for w in warnings] + + +# --------------------------------------------------------------- CandidateGraph +class TestCandidateGraphDataclass: + def test_default_construction_is_empty(self): + g = CandidateGraph() + assert g.vertices == [] + assert g.edges == [] + assert g.is_empty is True + + def test_populated_graph_is_not_empty(self): + g = CandidateGraph(vertices=[{"label": "person"}], edges=[]) + assert not g.is_empty + + def test_defaults_do_not_alias_across_instances(self): + """Frozen dataclass with default_factory must not share the list across instances.""" + a = CandidateGraph() + b = CandidateGraph() + a.vertices.append({"label": "x"}) + assert b.vertices == [] + + +# ---------------------------------------------------- Format 1: plain grouped JSON +class TestGroupedJson: + def test_grouped_object_with_both_sections(self): + parser = CandidateGraphParser() + graph, warnings = parser.parse( + json.dumps( + { + "vertices": [{"type": "vertex", "label": "person", "properties": {"name": "Tom"}}], + "edges": [ + { + "type": "edge", + "label": "acted_in", + "outV": "v1", + "inV": "v2", + "properties": {}, + } + ], + } + ) + ) + assert warnings == [] + assert len(graph.vertices) == 1 + assert graph.vertices[0]["label"] == "person" + assert graph.vertices[0]["type"] == "vertex" + assert len(graph.edges) == 1 + + def test_missing_edges_section_emits_warning_and_leaves_empty(self): + parser = CandidateGraphParser() + graph, warnings = parser.parse( + json.dumps({"vertices": [{"type": "vertex", "label": "person", "properties": {}}]}) + ) + assert _codes(warnings) == [WarningCode.GRAPH_SECTION_MISSING] + assert warnings[0].reason.startswith("'edges'") + assert graph.edges == [] + + def test_missing_vertices_section_emits_warning_and_leaves_empty(self): + parser = CandidateGraphParser() + graph, warnings = parser.parse(json.dumps({"edges": []})) + assert _codes(warnings) == [WarningCode.GRAPH_SECTION_MISSING] + assert warnings[0].reason.startswith("'vertices'") + assert graph.vertices == [] + + def test_both_sections_missing_emits_two_warnings(self): + parser = CandidateGraphParser() + graph, warnings = parser.parse(json.dumps({"unrelated": "value"})) + codes = _codes(warnings) + assert codes.count(WarningCode.GRAPH_SECTION_MISSING) == 2 + assert graph.is_empty + + def test_section_value_not_list_falls_back_to_empty(self): + parser = CandidateGraphParser() + graph, warnings = parser.parse(json.dumps({"vertices": "oops", "edges": []})) + # Non-list section is coerced to empty; no warning about it (the collector + # sees an empty list and has nothing to complain about). + assert graph.vertices == [] + assert graph.edges == [] + # But GRAPH_SECTION_MISSING is not emitted because the key was present. + assert WarningCode.GRAPH_SECTION_MISSING not in _codes(warnings) + + +# ------------------------------------------------------- Format 2: fenced JSON +class TestFencedJson: + def test_json_fence_with_language_tag(self): + parser = CandidateGraphParser() + raw = '```json\n{"vertices": [{"type": "vertex", "label": "person", "properties": {}}], "edges": []}\n```' + graph, warnings = parser.parse(raw) + assert warnings == [] + assert len(graph.vertices) == 1 + + def test_plain_fence_without_language_tag(self): + parser = CandidateGraphParser() + raw = '```\n{"vertices": [], "edges": []}\n```' + graph, warnings = parser.parse(raw) + assert warnings == [] + assert graph.is_empty + + def test_uppercase_fence_language_tag(self): + parser = CandidateGraphParser() + raw = '```JSON\n{"vertices": [], "edges": []}\n```' + graph, warnings = parser.parse(raw) + assert warnings == [] + + +# ------------------------------------------------- Format 3: JSON with prose +class TestJsonWithProse: + def test_prose_before_and_after_json(self): + parser = CandidateGraphParser() + raw = ( + "Based on the input text, here is the extracted graph:\n" + '{"vertices": [{"type": "vertex", "label": "person", "properties": {"name": "Tom"}}], "edges": []}\n' + "Note: some entities may be uncertain." + ) + graph, warnings = parser.parse(raw) + assert warnings == [] + assert graph.vertices[0]["label"] == "person" + + def test_only_prose_no_json_emits_json_not_found(self): + parser = CandidateGraphParser() + graph, warnings = parser.parse("This chunk does not contain any structured output.") + assert _codes(warnings) == [WarningCode.JSON_NOT_FOUND] + assert graph.is_empty + + def test_empty_string_emits_json_not_found(self): + parser = CandidateGraphParser() + graph, warnings = parser.parse("") + assert _codes(warnings) == [WarningCode.JSON_NOT_FOUND] + assert graph.is_empty + + def test_whitespace_only_after_fence_stripping_emits_json_not_found(self): + parser = CandidateGraphParser() + graph, warnings = parser.parse("```\n```\n") + assert _codes(warnings) == [WarningCode.JSON_NOT_FOUND] + + +# ------------------------------------------------------ Format 4: flat array +class TestFlatArray: + def test_flat_array_partitions_by_type(self): + parser = CandidateGraphParser() + raw = json.dumps( + [ + {"type": "vertex", "label": "person", "properties": {"name": "Tom"}}, + {"type": "edge", "label": "acted_in", "outV": "v1", "inV": "v2", "properties": {}}, + {"type": "vertex", "label": "movie", "properties": {"title": "FG"}}, + ] + ) + graph, warnings = parser.parse(raw) + assert warnings == [] + assert len(graph.vertices) == 2 + assert len(graph.edges) == 1 + + def test_flat_array_item_without_type_is_dropped(self): + parser = CandidateGraphParser() + raw = json.dumps( + [ + {"label": "person", "properties": {}}, # no type + {"type": "vertex", "label": "movie", "properties": {}}, + ] + ) + graph, warnings = parser.parse(raw) + assert _codes(warnings) == [WarningCode.ITEM_TYPE_MISMATCH] + assert warnings[0].context == {"label": "person"} + assert len(graph.vertices) == 1 + assert graph.vertices[0]["label"] == "movie" + + def test_flat_array_with_non_dict_items(self): + parser = CandidateGraphParser() + raw = json.dumps( + [ + "not a dict", + 42, + {"type": "vertex", "label": "person", "properties": {}}, + ] + ) + graph, warnings = parser.parse(raw) + codes = _codes(warnings) + assert codes.count(WarningCode.ITEM_NOT_OBJECT) == 2 + assert len(graph.vertices) == 1 + + +# ------------------------------------------------------ Broken JSON handling +class TestBrokenJson: + def test_truncated_json_emits_json_decode_failed(self): + parser = CandidateGraphParser() + raw = '{"vertices": [{"label": "person", "properties":' + graph, warnings = parser.parse(raw) + assert _codes(warnings) == [WarningCode.JSON_DECODE_FAILED] + assert graph.is_empty + + def test_scalar_payload_emits_section_missing_and_empties(self): + """`42` parses cleanly but is not a graph — treat as section-missing.""" + parser = CandidateGraphParser() + graph, warnings = parser.parse("42") + assert _codes(warnings) == [WarningCode.GRAPH_SECTION_MISSING] + assert graph.is_empty + + +# ------------------------------- Explicit type conflict inside grouped format +class TestGroupedItemTypeConflict: + def test_edge_typed_item_inside_vertices_array_is_dropped(self): + parser = CandidateGraphParser() + raw = json.dumps( + { + "vertices": [ + {"type": "edge", "label": "acted_in", "properties": {}}, + {"type": "vertex", "label": "person", "properties": {}}, + ], + "edges": [], + } + ) + graph, warnings = parser.parse(raw) + assert _codes(warnings) == [WarningCode.ITEM_TYPE_MISMATCH] + assert warnings[0].label == "acted_in" + assert len(graph.vertices) == 1 + assert graph.vertices[0]["label"] == "person" + + def test_vertex_typed_item_inside_edges_array_is_dropped(self): + parser = CandidateGraphParser() + raw = json.dumps( + { + "vertices": [], + "edges": [ + {"type": "vertex", "label": "person", "properties": {}}, + {"type": "edge", "label": "acted_in", "outV": "v1", "inV": "v2", "properties": {}}, + ], + } + ) + graph, warnings = parser.parse(raw) + assert _codes(warnings) == [WarningCode.ITEM_TYPE_MISMATCH] + assert warnings[0].label == "person" + assert len(graph.edges) == 1 + + def test_non_dict_item_in_grouped_section_emits_item_not_object(self): + parser = CandidateGraphParser() + raw = json.dumps( + { + "vertices": ["not-a-dict"], + "edges": [{"type": "edge", "label": "acted_in", "outV": "v1", "inV": "v2", "properties": {}}], + } + ) + graph, warnings = parser.parse(raw) + assert _codes(warnings) == [WarningCode.ITEM_NOT_OBJECT] + assert len(graph.vertices) == 0 + assert len(graph.edges) == 1 + + +# --------------------------------------------------------------- chunk_id +class TestChunkIdPropagation: + def test_warning_carries_chunk_id(self): + parser = CandidateGraphParser() + _, warnings = parser.parse("no json here", chunk_id=7) + assert warnings[0].chunk_id == 7 + + def test_omitted_chunk_id_stays_none(self): + parser = CandidateGraphParser() + _, warnings = parser.parse("no json here") + assert warnings[0].chunk_id is None + + +# --------------------------------------------------- item type normalization +class TestItemTypeNormalization: + def test_grouped_vertex_without_type_is_typed_by_the_parser(self): + """The collector fills in item['type'] so downstream never needs the check.""" + parser = CandidateGraphParser() + raw = json.dumps( + { + "vertices": [{"label": "person", "properties": {"name": "Tom"}}], + "edges": [], + } + ) + graph, warnings = parser.parse(raw) + assert warnings == [] + assert graph.vertices[0]["type"] == "vertex" + + def test_original_item_is_not_mutated(self): + parser = CandidateGraphParser() + original = {"label": "person", "properties": {"name": "Tom"}} + raw = json.dumps({"vertices": [original], "edges": []}) + graph, _ = parser.parse(raw) + # graph.vertices[0] is a copy — modifying it should not affect any + # dict that the caller may still be referencing. + assert "type" not in original # 'original' was the pre-JSON literal + assert graph.vertices[0]["type"] == "vertex" + + +# ============================================================================== +# SchemaAwareNormalizer +# ============================================================================== + + +def _schema() -> dict: + """Minimal schema with vertex-label ids (mirrors HugeGraph server output).""" + return { + "propertykeys": [ + {"name": "name", "data_type": "TEXT", "cardinality": "SINGLE"}, + {"name": "title", "data_type": "TEXT", "cardinality": "SINGLE"}, + {"name": "year", "data_type": "INT", "cardinality": "SINGLE"}, + {"name": "role", "data_type": "TEXT", "cardinality": "SINGLE"}, + ], + "vertexlabels": [ + { + "id": 1, + "name": "person", + "properties": ["name"], + "primary_keys": ["name"], + "nullable_keys": [], + "id_strategy": "PRIMARY_KEY", + }, + { + "id": 2, + "name": "movie", + "properties": ["title", "year"], + "primary_keys": ["title"], + "nullable_keys": ["year"], + "id_strategy": "PRIMARY_KEY", + }, + ], + "edgelabels": [ + { + "name": "acted_in", + "source_label": "person", + "target_label": "movie", + "properties": ["role"], + } + ], + } + + +def _normalizer() -> SchemaAwareNormalizer: + return SchemaAwareNormalizer(GraphSchemaIndex(_schema())) + + +# --------------------------------------------------------- vertex: label +class TestNormalizerVertexLabel: + def test_valid_vertex_produces_canonical_id(self): + norm = _normalizer() + candidate = CandidateGraph(vertices=[{"type": "vertex", "label": "person", "properties": {"name": "Tom"}}]) + graph, warnings = norm.normalize(candidate) + assert warnings == [] + assert graph.vertices == [{"type": "vertex", "label": "person", "properties": {"name": "Tom"}, "id": "1:Tom"}] + + def test_unknown_label_drops_vertex(self): + norm = _normalizer() + candidate = CandidateGraph(vertices=[{"type": "vertex", "label": "robot", "properties": {"name": "R2D2"}}]) + graph, warnings = norm.normalize(candidate) + assert _codes(warnings) == [WarningCode.VERTEX_LABEL_NOT_IN_SCHEMA] + assert graph.vertices == [] + + def test_missing_label_drops_vertex(self): + norm = _normalizer() + candidate = CandidateGraph(vertices=[{"type": "vertex", "properties": {"name": "Tom"}}]) + graph, warnings = norm.normalize(candidate) + assert _codes(warnings) == [WarningCode.VERTEX_LABEL_NOT_IN_SCHEMA] + assert graph.vertices == [] + + +# --------------------------------------------- vertex: property filter/coerce +class TestNormalizerVertexProperties: + def test_property_not_in_schema_is_dropped(self): + norm = _normalizer() + candidate = CandidateGraph( + vertices=[ + { + "type": "vertex", + "label": "person", + "properties": {"name": "Tom", "planet": "Earth"}, + } + ] + ) + graph, warnings = norm.normalize(candidate) + assert _codes(warnings) == [WarningCode.PROPERTY_NOT_IN_SCHEMA] + assert graph.vertices[0]["properties"] == {"name": "Tom"} + + def test_string_year_coerced_to_int_with_soft_warning(self): + norm = _normalizer() + candidate = CandidateGraph( + vertices=[{"type": "vertex", "label": "movie", "properties": {"title": "FG", "year": "1994"}}] + ) + graph, warnings = norm.normalize(candidate) + assert _codes(warnings) == [WarningCode.PROPERTY_COERCED] + assert graph.vertices[0]["properties"] == {"title": "FG", "year": 1994} + + def test_non_pk_property_coercion_failure_drops_only_that_property(self): + norm = _normalizer() + candidate = CandidateGraph( + vertices=[{"type": "vertex", "label": "movie", "properties": {"title": "FG", "year": "N/A"}}] + ) + graph, warnings = norm.normalize(candidate) + codes = _codes(warnings) + assert WarningCode.PROPERTY_COERCION_FAILED in codes + # The vertex survives without the broken property. + assert graph.vertices[0]["properties"] == {"title": "FG"} + + def test_pk_property_coercion_failure_drops_the_whole_vertex(self): + norm = _normalizer() + # `title` is a PK (TEXT) — a bool value fails the TEXT coercion? No, + # TEXT accepts anything via str(). Use INT PK — build a schema where + # the PK is INT. + schema = _schema() + # Change movie's PK to year (INT). + schema["vertexlabels"][1]["primary_keys"] = ["year"] + idx = GraphSchemaIndex(schema) + norm = SchemaAwareNormalizer(idx) + candidate = CandidateGraph( + vertices=[{"type": "vertex", "label": "movie", "properties": {"title": "FG", "year": "N/A"}}] + ) + graph, warnings = norm.normalize(candidate) + assert WarningCode.VERTEX_PRIMARY_KEY_INVALID in _codes(warnings) + assert graph.vertices == [] + + +# ------------------------------------------------ vertex: primary key checks +class TestNormalizerVertexPrimaryKey: + def test_missing_pk_drops_vertex(self): + norm = _normalizer() + candidate = CandidateGraph(vertices=[{"type": "vertex", "label": "person", "properties": {}}]) + graph, warnings = norm.normalize(candidate) + assert _codes(warnings) == [WarningCode.VERTEX_PRIMARY_KEY_MISSING] + assert graph.vertices == [] + + def test_empty_string_pk_drops_vertex(self): + norm = _normalizer() + candidate = CandidateGraph(vertices=[{"type": "vertex", "label": "person", "properties": {"name": ""}}]) + graph, warnings = norm.normalize(candidate) + assert _codes(warnings) == [WarningCode.VERTEX_PRIMARY_KEY_MISSING] + assert graph.vertices == [] + + +# ---------------------------------------------------- vertex: id and alias +class TestNormalizerVertexIdAndAlias: + def test_llm_original_id_becomes_alias(self): + norm = _normalizer() + candidate = CandidateGraph( + vertices=[{"type": "vertex", "id": "v1", "label": "person", "properties": {"name": "Tom"}}] + ) + graph, warnings = norm.normalize(candidate) + assert _codes(warnings) == [WarningCode.VERTEX_ALIAS_RECORDED] + # Alias table contains both the raw-id-to-canonical mapping and the + # identity mapping for the canonical id itself. + assert graph.aliases[("person", "v1")] == "1:Tom" + assert graph.aliases[("person", "1:Tom")] == "1:Tom" + + def test_llm_original_id_matching_canonical_produces_no_alias_warning(self): + norm = _normalizer() + candidate = CandidateGraph( + vertices=[ + { + "type": "vertex", + "id": "1:Tom", + "label": "person", + "properties": {"name": "Tom"}, + } + ] + ) + graph, warnings = norm.normalize(candidate) + assert _codes(warnings) == [] + assert graph.aliases[("person", "1:Tom")] == "1:Tom" + + +# ----------------------------------------------------------- edge: label +class TestNormalizerEdgeLabel: + def test_unknown_edge_label_drops_edge(self): + norm = _normalizer() + candidate = CandidateGraph(edges=[{"type": "edge", "label": "directed", "properties": {}}]) + graph, warnings = norm.normalize(candidate) + assert _codes(warnings) == [WarningCode.EDGE_LABEL_NOT_IN_SCHEMA] + assert graph.edges == [] + + +# ----------------------------------------------------- edge: property filter +class TestNormalizerEdgeProperties: + def test_edge_property_not_in_schema_is_dropped(self): + norm = _normalizer() + candidate = CandidateGraph( + vertices=[ + {"type": "vertex", "label": "person", "properties": {"name": "Tom"}}, + {"type": "vertex", "label": "movie", "properties": {"title": "FG"}}, + ], + edges=[ + { + "type": "edge", + "label": "acted_in", + "source": {"label": "person", "properties": {"name": "Tom"}}, + "target": {"label": "movie", "properties": {"title": "FG"}}, + "properties": {"role": "Forrest", "budget": 999}, + } + ], + ) + graph, warnings = norm.normalize(candidate) + assert WarningCode.PROPERTY_NOT_IN_SCHEMA in _codes(warnings) + assert graph.edges[0]["properties"] == {"role": "Forrest"} + + +# --------------------------------------------------- edge: endpoint direction +class TestNormalizerEdgeEndpointCompatibility: + def test_reversed_endpoints_drop_edge(self): + norm = _normalizer() + candidate = CandidateGraph( + edges=[ + { + "type": "edge", + "label": "acted_in", + "outVLabel": "movie", + "inVLabel": "person", + "outV": "2:FG", + "inV": "1:Tom", + "properties": {}, + } + ] + ) + graph, warnings = norm.normalize(candidate) + assert _codes(warnings) == [WarningCode.EDGE_ENDPOINT_MISMATCH] + assert graph.edges == [] + + def test_missing_endpoint_labels_are_filled_from_schema(self): + """LLM sometimes omits redundant outVLabel/inVLabel; use schema.""" + norm = _normalizer() + candidate = CandidateGraph( + vertices=[ + {"type": "vertex", "label": "person", "properties": {"name": "Tom"}}, + {"type": "vertex", "label": "movie", "properties": {"title": "FG"}}, + ], + edges=[ + { + "type": "edge", + "label": "acted_in", + "outV": "1:Tom", + "inV": "2:FG", + "properties": {}, + } + ], + ) + graph, warnings = norm.normalize(candidate) + assert warnings == [] + assert graph.edges[0]["outVLabel"] == "person" + assert graph.edges[0]["inVLabel"] == "movie" + + +# --------------------------------------------------- edge: endpoint resolution +class TestNormalizerEdgeEndpointResolution: + def test_legacy_source_target_resolves_to_canonical(self): + norm = _normalizer() + candidate = CandidateGraph( + vertices=[ + {"type": "vertex", "label": "person", "properties": {"name": "Tom"}}, + {"type": "vertex", "label": "movie", "properties": {"title": "FG"}}, + ], + edges=[ + { + "type": "edge", + "label": "acted_in", + "source": {"label": "person", "properties": {"name": "Tom"}}, + "target": {"label": "movie", "properties": {"title": "FG"}}, + "properties": {"role": "Forrest"}, + } + ], + ) + graph, warnings = norm.normalize(candidate) + assert warnings == [] + assert graph.edges[0]["outV"] == "1:Tom" + assert graph.edges[0]["inV"] == "2:FG" + assert PENDING_OUT_KEY not in graph.edges[0] + assert PENDING_IN_KEY not in graph.edges[0] + + def test_llm_raw_id_resolves_via_chunk_aliases(self): + norm = _normalizer() + candidate = CandidateGraph( + vertices=[ + {"type": "vertex", "id": "v1", "label": "person", "properties": {"name": "Tom"}}, + {"type": "vertex", "id": "v2", "label": "movie", "properties": {"title": "FG"}}, + ], + edges=[ + { + "type": "edge", + "label": "acted_in", + "outV": "v1", + "inV": "v2", + "outVLabel": "person", + "inVLabel": "movie", + "properties": {}, + } + ], + ) + graph, warnings = norm.normalize(candidate) + assert not any(w.code is WarningCode.ENDPOINT_PENDING_REPAIR for w in warnings) + assert graph.edges[0]["outV"] == "1:Tom" + assert graph.edges[0]["inV"] == "2:FG" + + def test_unresolvable_out_endpoint_marks_pending(self): + """LLM references an id that has no vertex in this chunk.""" + norm = _normalizer() + candidate = CandidateGraph( + vertices=[ + {"type": "vertex", "id": "v2", "label": "movie", "properties": {"title": "FG"}}, + ], + edges=[ + { + "type": "edge", + "label": "acted_in", + "outV": "v99", + "inV": "v2", + "outVLabel": "person", + "inVLabel": "movie", + "properties": {}, + } + ], + ) + graph, warnings = norm.normalize(candidate) + assert WarningCode.ENDPOINT_PENDING_REPAIR in _codes(warnings) + edge = graph.edges[0] + assert edge.get("outV") is None + assert edge[PENDING_OUT_KEY] == {"original_id": "v99"} + assert edge["inV"] == "2:FG" + + +# --------------------------------------------------------- integration +class TestNormalizerIntegration: + def test_end_to_end_document_flow_smoke(self): + """Parse + normalize round-trip on a mixed-format grouped payload.""" + parser = CandidateGraphParser() + norm = _normalizer() + raw = json.dumps( + { + "vertices": [ + { + "type": "vertex", + "id": "v1", + "label": "person", + "properties": {"name": "Tom Hanks"}, + }, + { + "type": "vertex", + "id": "v2", + "label": "movie", + "properties": {"title": "Forrest Gump", "year": "1994"}, + }, + ], + "edges": [ + { + "type": "edge", + "label": "acted_in", + "outV": "v1", + "inV": "v2", + "outVLabel": "person", + "inVLabel": "movie", + "properties": {"role": "Forrest"}, + } + ], + } + ) + candidate, parser_warnings = parser.parse(raw) + assert parser_warnings == [] + graph, norm_warnings = norm.normalize(candidate, chunk_id=0) + codes = _codes(norm_warnings) + # Two alias records (one per vertex) + one soft coerce for year. + assert codes.count(WarningCode.VERTEX_ALIAS_RECORDED) == 2 + assert codes.count(WarningCode.PROPERTY_COERCED) == 1 + assert graph.vertices[0]["id"] == "1:Tom Hanks" + assert graph.vertices[1]["id"] == "2:Forrest Gump" + assert graph.vertices[1]["properties"]["year"] == 1994 + assert graph.edges[0]["outV"] == "1:Tom Hanks" + assert graph.edges[0]["inV"] == "2:Forrest Gump" + + def test_chunk_id_flows_into_all_normalizer_warnings(self): + norm = _normalizer() + candidate = CandidateGraph(vertices=[{"type": "vertex", "label": "robot", "properties": {}}]) + _, warnings = norm.normalize(candidate, chunk_id=5) + assert warnings[0].chunk_id == 5 + + +# ============================================================================== +# DocumentGraphAssembler +# ============================================================================== + + +def _assembler() -> DocumentGraphAssembler: + return DocumentGraphAssembler(GraphSchemaIndex(_schema())) + + +def _v(label: str, vid: str, **props) -> dict: + return {"type": "vertex", "label": label, "id": vid, "properties": dict(props)} + + +def _e_resolved(label: str, out_v: str, in_v: str, out_label: str, in_label: str, **props) -> dict: + return { + "type": "edge", + "label": label, + "outV": out_v, + "inV": in_v, + "outVLabel": out_label, + "inVLabel": in_label, + "properties": dict(props), + } + + +# ---------------------------------------------------------- vertex merge +class TestAssemblerVertexMerge: + def test_same_key_across_chunks_merges_and_emits_warning(self): + asm = _assembler() + chunk_a = NormalizedChunkGraph(vertices=[_v("person", "1:Tom", name="Tom")]) + chunk_b = NormalizedChunkGraph(vertices=[_v("person", "1:Tom", name="Tom")]) + graph, warnings = asm.assemble([chunk_a, chunk_b]) + assert len(graph.vertices) == 1 + assert graph.pre_merge_vertex_count == 2 + codes = _codes(warnings) + assert codes.count(WarningCode.DUPLICATE_VERTEX_MERGED) == 1 + + def test_property_conflict_first_wins(self): + asm = _assembler() + chunk_a = NormalizedChunkGraph(vertices=[_v("movie", "2:FG", title="FG", year=1994)]) + chunk_b = NormalizedChunkGraph(vertices=[_v("movie", "2:FG", title="FG", year=1993)]) + graph, warnings = asm.assemble([chunk_a, chunk_b]) + assert graph.vertices[0]["properties"]["year"] == 1994 + codes = _codes(warnings) + assert WarningCode.PROPERTY_CONFLICT in codes + conflict = next(w for w in warnings if w.code is WarningCode.PROPERTY_CONFLICT) + assert conflict.context == {"property": "year", "kept": 1994, "discarded": 1993} + + def test_missing_property_completed_from_later_chunk(self): + asm = _assembler() + chunk_a = NormalizedChunkGraph(vertices=[_v("movie", "2:FG", title="FG")]) + chunk_b = NormalizedChunkGraph(vertices=[_v("movie", "2:FG", title="FG", year=1994)]) + graph, _ = asm.assemble([chunk_a, chunk_b]) + assert graph.vertices[0]["properties"] == {"title": "FG", "year": 1994} + + def test_different_keys_stay_separate(self): + asm = _assembler() + chunk_a = NormalizedChunkGraph(vertices=[_v("person", "1:Tom", name="Tom")]) + chunk_b = NormalizedChunkGraph(vertices=[_v("person", "1:Alice", name="Alice")]) + graph, warnings = asm.assemble([chunk_a, chunk_b]) + assert len(graph.vertices) == 2 + assert WarningCode.DUPLICATE_VERTEX_MERGED not in _codes(warnings) + + def test_output_order_preserves_first_appearance(self): + asm = _assembler() + chunk_a = NormalizedChunkGraph( + vertices=[ + _v("person", "1:Tom", name="Tom"), + _v("movie", "2:FG", title="FG"), + ] + ) + chunk_b = NormalizedChunkGraph( + vertices=[ + _v("movie", "2:FG", title="FG"), + _v("person", "1:Tom", name="Tom"), + ] + ) + graph, _ = asm.assemble([chunk_a, chunk_b]) + assert [v["id"] for v in graph.vertices] == ["1:Tom", "2:FG"] + + def test_vertex_without_id_is_kept_verbatim(self): + asm = _assembler() + vertex_no_id = {"type": "vertex", "label": "person", "properties": {"name": "Anon"}} + chunk = NormalizedChunkGraph(vertices=[vertex_no_id, vertex_no_id]) + graph, warnings = asm.assemble([chunk]) + # Both occurrences kept — no merge (no way to identify). + assert len(graph.vertices) == 2 + assert WarningCode.DUPLICATE_VERTEX_MERGED not in _codes(warnings) + + +# --------------------------------------------------------- endpoint repair +class TestAssemblerEndpointRepair: + def test_pending_out_endpoint_repaired_via_cross_chunk_alias(self): + """Chunk A has the vertex + alias; chunk B has the edge referencing the raw id.""" + asm = _assembler() + chunk_a = NormalizedChunkGraph( + vertices=[_v("person", "1:Tom", name="Tom"), _v("movie", "2:FG", title="FG")], + aliases={("person", "v1"): "1:Tom", ("movie", "2:FG"): "2:FG", ("person", "1:Tom"): "1:Tom"}, + ) + edge = { + "type": "edge", + "label": "acted_in", + "inV": "2:FG", + "inVLabel": "movie", + "outVLabel": "person", + "properties": {}, + PENDING_OUT_KEY: {"original_id": "v1"}, + } + chunk_b = NormalizedChunkGraph(edges=[edge]) + graph, warnings = asm.assemble([chunk_a, chunk_b]) + assert graph.endpoint_repair_count == 1 + assert graph.edges[0]["outV"] == "1:Tom" + # No leftover pending marker. + assert PENDING_OUT_KEY not in graph.edges[0] + # Also no ENDPOINT_UNRESOLVED / AMBIGUOUS. + codes = _codes(warnings) + assert WarningCode.ENDPOINT_UNRESOLVED not in codes + assert WarningCode.ENDPOINT_AMBIGUOUS not in codes + + def test_pending_endpoint_with_no_alias_is_dropped_as_unresolved(self): + asm = _assembler() + edge = { + "type": "edge", + "label": "acted_in", + "inV": "2:FG", + "inVLabel": "movie", + "outVLabel": "person", + "properties": {}, + PENDING_OUT_KEY: {"original_id": "ghost"}, + } + chunk_b = NormalizedChunkGraph(edges=[edge]) + graph, warnings = asm.assemble([chunk_b]) + assert graph.edges == [] + assert WarningCode.ENDPOINT_UNRESOLVED in _codes(warnings) + + def test_ambiguous_alias_yields_endpoint_ambiguous(self): + """The same LLM raw id maps to different canonical ids across chunks.""" + asm = _assembler() + chunk_a = NormalizedChunkGraph(aliases={("person", "v1"): "1:Tom"}) + chunk_b = NormalizedChunkGraph(aliases={("person", "v1"): "1:Alice"}) + edge = { + "type": "edge", + "label": "acted_in", + "inV": "2:FG", + "inVLabel": "movie", + "outVLabel": "person", + "properties": {}, + PENDING_OUT_KEY: {"original_id": "v1"}, + } + chunk_c = NormalizedChunkGraph(edges=[edge]) + graph, warnings = asm.assemble([chunk_a, chunk_b, chunk_c]) + assert graph.edges == [] + assert WarningCode.ENDPOINT_AMBIGUOUS in _codes(warnings) + assert WarningCode.ENDPOINT_UNRESOLVED not in _codes(warnings) + + def test_edge_already_resolved_at_chunk_level_passes_through(self): + asm = _assembler() + chunk = NormalizedChunkGraph( + vertices=[_v("person", "1:Tom", name="Tom"), _v("movie", "2:FG", title="FG")], + edges=[_e_resolved("acted_in", "1:Tom", "2:FG", "person", "movie", role="Forrest")], + ) + graph, warnings = asm.assemble([chunk]) + assert graph.endpoint_repair_count == 0 + assert len(graph.edges) == 1 + assert graph.edges[0]["outV"] == "1:Tom" + + +# --------------------------------------------------------------- edge dedup +class TestAssemblerEdgeDedupe: + def test_identical_edges_deduped(self): + asm = _assembler() + edge_1 = _e_resolved("acted_in", "1:Tom", "2:FG", "person", "movie", role="Forrest") + edge_2 = _e_resolved("acted_in", "1:Tom", "2:FG", "person", "movie", role="Forrest") + chunk = NormalizedChunkGraph( + vertices=[_v("person", "1:Tom", name="Tom"), _v("movie", "2:FG", title="FG")], + edges=[edge_1, edge_2], + ) + graph, warnings = asm.assemble([chunk]) + assert len(graph.edges) == 1 + assert graph.pre_merge_edge_count == 2 + assert WarningCode.DUPLICATE_EDGE_MERGED in _codes(warnings) + + def test_same_endpoints_different_properties_are_both_kept(self): + """Different property signatures represent distinct facts — don't merge.""" + asm = _assembler() + edge_1 = _e_resolved("acted_in", "1:Tom", "2:FG", "person", "movie", role="Forrest") + edge_2 = _e_resolved("acted_in", "1:Tom", "2:FG", "person", "movie", role="Narrator") + chunk = NormalizedChunkGraph( + vertices=[_v("person", "1:Tom", name="Tom"), _v("movie", "2:FG", title="FG")], + edges=[edge_1, edge_2], + ) + graph, warnings = asm.assemble([chunk]) + assert len(graph.edges) == 2 + assert WarningCode.DUPLICATE_EDGE_MERGED not in _codes(warnings) + + def test_dedupe_handles_nested_list_property_values(self): + """LIST/SET cardinality property values (unhashable as tuples) still dedupe.""" + asm = _assembler() + edge = { + "type": "edge", + "label": "acted_in", + "outV": "1:Tom", + "inV": "2:FG", + "outVLabel": "person", + "inVLabel": "movie", + "properties": {"tags": ["lead", "drama"]}, + } + chunk = NormalizedChunkGraph(edges=[dict(edge), dict(edge)]) + graph, warnings = asm.assemble([chunk]) + assert len(graph.edges) == 1 + assert WarningCode.DUPLICATE_EDGE_MERGED in _codes(warnings) + + +# --------------------------------------------------------------- integration +class TestAssemblerIntegration: + def test_multi_chunk_end_to_end(self): + """Two chunks reference the same entity — merged, deduped, endpoint repaired across chunks.""" + asm = _assembler() + # Chunk 0: Tom + Forrest Gump + acted_in edge (fully local). + chunk_0 = NormalizedChunkGraph( + vertices=[_v("person", "1:Tom", name="Tom"), _v("movie", "2:FG", title="FG")], + edges=[_e_resolved("acted_in", "1:Tom", "2:FG", "person", "movie", role="Forrest")], + ) + # Chunk 1: Duplicate Tom + duplicate edge with same shape (should merge/dedupe). + chunk_1 = NormalizedChunkGraph( + vertices=[_v("person", "1:Tom", name="Tom"), _v("movie", "2:FG", title="FG")], + edges=[_e_resolved("acted_in", "1:Tom", "2:FG", "person", "movie", role="Forrest")], + ) + graph, warnings = asm.assemble([chunk_0, chunk_1]) + assert len(graph.vertices) == 2 + assert len(graph.edges) == 1 + assert graph.pre_merge_vertex_count == 4 + assert graph.pre_merge_edge_count == 2 + codes = _codes(warnings) + assert codes.count(WarningCode.DUPLICATE_VERTEX_MERGED) == 2 + assert codes.count(WarningCode.DUPLICATE_EDGE_MERGED) == 1 + + def test_full_pipeline_parser_normalizer_assembler(self): + """Parse two chunks' raw output, normalize, assemble; verify the whole loop.""" + parser = CandidateGraphParser() + norm = _normalizer() + asm = _assembler() + + chunk_0_raw = json.dumps( + { + "vertices": [ + {"type": "vertex", "id": "v1", "label": "person", "properties": {"name": "Tom Hanks"}}, + { + "type": "vertex", + "id": "v2", + "label": "movie", + "properties": {"title": "Forrest Gump", "year": "1994"}, + }, + ], + "edges": [ + { + "type": "edge", + "label": "acted_in", + "outV": "v1", + "inV": "v2", + "outVLabel": "person", + "inVLabel": "movie", + "properties": {"role": "Forrest"}, + } + ], + } + ) + chunk_1_raw = json.dumps( + { + "vertices": [{"type": "vertex", "id": "u1", "label": "person", "properties": {"name": "Tom Hanks"}}], + "edges": [ + { + "type": "edge", + "label": "acted_in", + "source": {"label": "person", "properties": {"name": "Tom Hanks"}}, + "target": {"label": "movie", "properties": {"title": "Forrest Gump"}}, + "properties": {}, + } + ], + } + ) + + chunk_graphs = [] + for i, raw in enumerate([chunk_0_raw, chunk_1_raw]): + cand, _ = parser.parse(raw, chunk_id=i) + ng, _ = norm.normalize(cand, chunk_id=i) + chunk_graphs.append(ng) + + doc, warnings = asm.assemble(chunk_graphs) + # Two unique vertices (Tom + Forrest Gump); one unique edge shape after dedup + # (chunk_0 has role=Forrest, chunk_1 has empty properties → both kept as + # distinct signatures — that matches the "different property sig, keep both" + # rule). + assert len(doc.vertices) == 2 + assert len(doc.edges) == 2 + # Vertex Tom appeared twice → 1 merge. + codes = _codes(warnings) + assert codes.count(WarningCode.DUPLICATE_VERTEX_MERGED) == 1 diff --git a/hugegraph-llm/src/tests/operators/llm_op/test_property_graph_quality_gate.py b/hugegraph-llm/src/tests/operators/llm_op/test_property_graph_quality_gate.py new file mode 100644 index 000000000..9a97afba5 --- /dev/null +++ b/hugegraph-llm/src/tests/operators/llm_op/test_property_graph_quality_gate.py @@ -0,0 +1,417 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +"""Unit tests for GraphQualityGate and QualityMetrics (Issue #74).""" + +from __future__ import annotations + +import json + +import pytest + +from hugegraph_llm.operators.llm_op.property_graph_extract_enhanced import ( + DocumentGraph, + GraphQualityGate, + QualityMetrics, + StructuredWarning, + WarningCode, +) + +pytestmark = pytest.mark.contract + + +def _w(code: WarningCode, item_type: str = "vertex", **kwargs) -> StructuredWarning: + return StructuredWarning(code=code, item_type=item_type, reason="test", **kwargs) + + +# ------------------------------------------------------------ zero-input safety +class TestZeroInputSafety: + def test_empty_graph_no_warnings_all_ratios_one(self): + """No candidates, no problems → every ratio reports 1.0 (design contract).""" + metrics = GraphQualityGate.compute( + DocumentGraph(), + warnings=[], + candidate_vertex_count=0, + candidate_edge_count=0, + ) + assert metrics.schema_valid_vertex_ratio == 1.0 + assert metrics.schema_valid_edge_ratio == 1.0 + assert metrics.endpoint_resolution_rate == 1.0 + # Reduction metrics stay at 0.0 for the empty case (no reduction achieved). + assert metrics.duplicate_vertex_reduction == 0.0 + assert metrics.duplicate_edge_reduction == 0.0 + assert metrics.property_valid_ratio == 1.0 + + def test_no_nan_ever(self): + """Every ratio field must be a plain float, no matter the input state.""" + metrics = GraphQualityGate.compute( + DocumentGraph(), + warnings=[], + candidate_vertex_count=0, + candidate_edge_count=0, + ) + for name in [ + "schema_valid_vertex_ratio", + "schema_valid_edge_ratio", + "endpoint_resolution_rate", + "duplicate_vertex_reduction", + "duplicate_edge_reduction", + "property_valid_ratio", + ]: + value = getattr(metrics, name) + assert value == value # NaN != NaN, so this catches NaN. + assert 0.0 <= value <= 1.0 + + +# ------------------------------------------------------ schema-validity ratios +class TestSchemaValidRatios: + def test_vertex_ratio_reflects_normalization_drops(self): + """4 candidate vertices, 3 survived normalization → 3/4.""" + graph = DocumentGraph(vertices=[{}, {}, {}], pre_merge_vertex_count=3) + metrics = GraphQualityGate.compute( + graph, + warnings=[_w(WarningCode.VERTEX_LABEL_NOT_IN_SCHEMA)], + candidate_vertex_count=4, + candidate_edge_count=0, + ) + assert metrics.schema_valid_vertex_ratio == 0.75 + + def test_edge_ratio_reflects_normalization_drops(self): + graph = DocumentGraph(edges=[{}, {}], pre_merge_edge_count=2) + metrics = GraphQualityGate.compute( + graph, + warnings=[ + _w(WarningCode.EDGE_LABEL_NOT_IN_SCHEMA, item_type="edge"), + _w(WarningCode.EDGE_ENDPOINT_MISMATCH, item_type="edge"), + ], + candidate_vertex_count=0, + candidate_edge_count=4, + ) + assert metrics.schema_valid_edge_ratio == 0.5 + + def test_zero_candidates_ratio_is_one_regardless_of_kept(self): + """Downstream should never divide by zero; empty candidate → 1.0 by contract.""" + metrics = GraphQualityGate.compute( + DocumentGraph(), + warnings=[], + candidate_vertex_count=0, + candidate_edge_count=0, + ) + assert metrics.schema_valid_vertex_ratio == 1.0 + + +# --------------------------------------------------- endpoint resolution rate +class TestEndpointResolution: + def test_all_pending_resolved(self): + """2 pending edges, 0 unresolved / 0 ambiguous → 1.0.""" + graph = DocumentGraph(edges=[{}, {}], pre_merge_edge_count=2, endpoint_repair_count=2) + warnings = [ + _w(WarningCode.ENDPOINT_PENDING_REPAIR, item_type="edge"), + _w(WarningCode.ENDPOINT_PENDING_REPAIR, item_type="edge"), + ] + metrics = GraphQualityGate.compute( + graph, + warnings=warnings, + candidate_vertex_count=0, + candidate_edge_count=2, + ) + assert metrics.endpoint_resolution_rate == 1.0 + assert metrics.endpoint_repair_count == 2 + + def test_partial_resolution(self): + """3 pending, 1 unresolved → 2/3.""" + warnings = [ + _w(WarningCode.ENDPOINT_PENDING_REPAIR, item_type="edge"), + _w(WarningCode.ENDPOINT_PENDING_REPAIR, item_type="edge"), + _w(WarningCode.ENDPOINT_PENDING_REPAIR, item_type="edge"), + _w(WarningCode.ENDPOINT_UNRESOLVED, item_type="edge"), + ] + metrics = GraphQualityGate.compute( + DocumentGraph(), + warnings=warnings, + candidate_vertex_count=0, + candidate_edge_count=0, + ) + assert metrics.endpoint_resolution_rate == 2 / 3 + + def test_ambiguous_counts_toward_unresolved(self): + warnings = [ + _w(WarningCode.ENDPOINT_PENDING_REPAIR, item_type="edge"), + _w(WarningCode.ENDPOINT_PENDING_REPAIR, item_type="edge"), + _w(WarningCode.ENDPOINT_AMBIGUOUS, item_type="edge"), + ] + metrics = GraphQualityGate.compute( + DocumentGraph(), + warnings=warnings, + candidate_vertex_count=0, + candidate_edge_count=0, + ) + assert metrics.endpoint_resolution_rate == 0.5 + + def test_zero_pending_rate_is_one(self): + metrics = GraphQualityGate.compute( + DocumentGraph(), + warnings=[], + candidate_vertex_count=0, + candidate_edge_count=0, + ) + assert metrics.endpoint_resolution_rate == 1.0 + + +# ------------------------------------------------------ duplicate reduction +class TestDuplicateReduction: + def test_half_of_vertices_merged(self): + graph = DocumentGraph(vertices=[{}, {}], pre_merge_vertex_count=4) + metrics = GraphQualityGate.compute(graph, warnings=[], candidate_vertex_count=4, candidate_edge_count=0) + assert metrics.duplicate_vertex_reduction == 0.5 + + def test_all_edges_merged(self): + graph = DocumentGraph(edges=[{}], pre_merge_edge_count=5) + metrics = GraphQualityGate.compute(graph, warnings=[], candidate_vertex_count=0, candidate_edge_count=5) + assert metrics.duplicate_edge_reduction == 0.8 + + def test_no_pre_merge_yields_zero_reduction(self): + """Empty document should not surface reduction as 1.0 — nothing was reduced.""" + graph = DocumentGraph() + metrics = GraphQualityGate.compute(graph, warnings=[], candidate_vertex_count=0, candidate_edge_count=0) + assert metrics.duplicate_vertex_reduction == 0.0 + assert metrics.duplicate_edge_reduction == 0.0 + + +# ------------------------------------------------------ property valid ratio +class TestPropertyValidRatio: + def test_full_valid(self): + graph = DocumentGraph( + vertices=[{"properties": {"name": "Tom", "aliases": []}}], + edges=[{"properties": {"role": "Forrest"}}], + ) + metrics = GraphQualityGate.compute(graph, warnings=[], candidate_vertex_count=1, candidate_edge_count=1) + assert metrics.property_valid_ratio == 1.0 + + def test_half_dropped(self): + """2 kept, 2 dropped → 2/4 = 0.5.""" + graph = DocumentGraph(vertices=[{"properties": {"name": "Tom", "aliases": []}}]) + warnings = [ + _w(WarningCode.PROPERTY_NOT_IN_SCHEMA), + _w(WarningCode.PROPERTY_COERCION_FAILED), + ] + metrics = GraphQualityGate.compute(graph, warnings=warnings, candidate_vertex_count=1, candidate_edge_count=0) + assert metrics.property_valid_ratio == 0.5 + + def test_property_conflict_is_not_counted_as_invalid(self): + """First-wins conflict preserves the value; it must not depress property_valid_ratio.""" + graph = DocumentGraph(vertices=[{"properties": {"name": "Tom"}}]) + warnings = [_w(WarningCode.PROPERTY_CONFLICT)] + metrics = GraphQualityGate.compute(graph, warnings=warnings, candidate_vertex_count=1, candidate_edge_count=0) + assert metrics.property_valid_ratio == 1.0 + assert metrics.property_conflict_count == 1 + + +# -------------------------------------------------------------- counters +class TestCounters: + def test_dropped_item_count_sums_all_drop_codes(self): + warnings = [ + _w(WarningCode.VERTEX_LABEL_NOT_IN_SCHEMA), + _w(WarningCode.EDGE_LABEL_NOT_IN_SCHEMA, item_type="edge"), + _w(WarningCode.ENDPOINT_UNRESOLVED, item_type="edge"), + _w(WarningCode.ITEM_NOT_OBJECT, item_type="graph"), + _w(WarningCode.PROPERTY_NOT_IN_SCHEMA), + ] + metrics = GraphQualityGate.compute( + DocumentGraph(), + warnings=warnings, + candidate_vertex_count=0, + candidate_edge_count=0, + ) + assert metrics.dropped_item_count == 5 + + def test_dropping_codes_exclude_duplicates_and_coercions(self): + """DUPLICATE_*_MERGED / PROPERTY_COERCED must NOT inflate dropped_item_count.""" + warnings = [ + _w(WarningCode.DUPLICATE_VERTEX_MERGED), + _w(WarningCode.DUPLICATE_EDGE_MERGED, item_type="edge"), + _w(WarningCode.PROPERTY_COERCED), + _w(WarningCode.PROPERTY_CONFLICT), + ] + metrics = GraphQualityGate.compute( + DocumentGraph(), + warnings=warnings, + candidate_vertex_count=0, + candidate_edge_count=0, + ) + assert metrics.dropped_item_count == 0 + + def test_coerced_property_count(self): + warnings = [_w(WarningCode.PROPERTY_COERCED), _w(WarningCode.PROPERTY_COERCED)] + metrics = GraphQualityGate.compute( + DocumentGraph(), + warnings=warnings, + candidate_vertex_count=0, + candidate_edge_count=0, + ) + assert metrics.coerced_property_count == 2 + + def test_endpoint_repair_count_pulled_from_graph(self): + graph = DocumentGraph(endpoint_repair_count=7) + metrics = GraphQualityGate.compute(graph, warnings=[], candidate_vertex_count=0, candidate_edge_count=0) + assert metrics.endpoint_repair_count == 7 + + +# ------------------------------------------------ warning_code_distribution +class TestWarningCodeDistribution: + def test_distribution_matches_warning_stream(self): + warnings = [ + _w(WarningCode.PROPERTY_COERCED), + _w(WarningCode.PROPERTY_COERCED), + _w(WarningCode.DUPLICATE_VERTEX_MERGED), + ] + metrics = GraphQualityGate.compute( + DocumentGraph(), + warnings=warnings, + candidate_vertex_count=0, + candidate_edge_count=0, + ) + assert metrics.warning_code_distribution == { + "PROPERTY_COERCED": 2, + "DUPLICATE_VERTEX_MERGED": 1, + } + + def test_distribution_empty_for_no_warnings(self): + metrics = GraphQualityGate.compute( + DocumentGraph(), + warnings=[], + candidate_vertex_count=0, + candidate_edge_count=0, + ) + assert metrics.warning_code_distribution == {} + + +# ------------------------------------------------------ serialization +class TestQualityMetricsSerialization: + def test_to_dict_rounds_ratios_to_four_decimals(self): + graph = DocumentGraph(vertices=[{}], pre_merge_vertex_count=3) + metrics = GraphQualityGate.compute( + graph, + warnings=[ + _w(WarningCode.VERTEX_LABEL_NOT_IN_SCHEMA), + _w(WarningCode.VERTEX_LABEL_NOT_IN_SCHEMA), + _w(WarningCode.VERTEX_LABEL_NOT_IN_SCHEMA), + _w(WarningCode.VERTEX_LABEL_NOT_IN_SCHEMA), + ], + candidate_vertex_count=7, # → 3/7 = 0.42857... + candidate_edge_count=0, + ) + d = metrics.to_dict() + assert d["schema_valid_vertex_ratio"] == 0.4286 + + def test_to_dict_is_pure_json(self): + """Round-trip through json.dumps to prove the shape is JSON-safe.""" + graph = DocumentGraph(vertices=[{"properties": {"n": 1}}], pre_merge_vertex_count=1) + metrics = GraphQualityGate.compute(graph, warnings=[], candidate_vertex_count=1, candidate_edge_count=0) + encoded = json.dumps(metrics.to_dict()) + decoded = json.loads(encoded) + assert set(decoded) == { + "schema_valid_vertex_ratio", + "schema_valid_edge_ratio", + "endpoint_resolution_rate", + "duplicate_vertex_reduction", + "duplicate_edge_reduction", + "property_valid_ratio", + "dropped_item_count", + "coerced_property_count", + "endpoint_repair_count", + "property_conflict_count", + "warning_code_distribution", + } + + def test_metrics_are_frozen(self): + """QualityMetrics is a frozen dataclass — no post-hoc mutation.""" + import dataclasses + + metrics = GraphQualityGate.compute( + DocumentGraph(), + warnings=[], + candidate_vertex_count=0, + candidate_edge_count=0, + ) + try: + metrics.dropped_item_count = 99 # type: ignore[misc] + failed = False + except dataclasses.FrozenInstanceError: + failed = True + assert failed + + +# ------------------------------------------------------ robustness +class TestGateRobustness: + def test_ignores_non_structured_warning_entries(self): + """A malformed warning at the tail should not sink the metrics.""" + warnings = [ + _w(WarningCode.PROPERTY_COERCED), + "not a warning", # type: ignore[list-item] + {"code": "also not a warning"}, # type: ignore[list-item] + ] + metrics = GraphQualityGate.compute( + DocumentGraph(), + warnings=warnings, # type: ignore[arg-type] + candidate_vertex_count=0, + candidate_edge_count=0, + ) + # The single valid warning is counted; the malformed entries are skipped. + assert metrics.coerced_property_count == 1 + assert metrics.warning_code_distribution == {"PROPERTY_COERCED": 1} + + +# ------------------------------------------------------ integration +class TestGateIntegration: + def test_realistic_multi_metric_scenario(self): + """Everything hits at once: drops, coercions, merges, repairs.""" + graph = DocumentGraph( + vertices=[{"properties": {"name": "Tom"}}, {"properties": {"title": "FG", "year": 1994}}], + edges=[{"properties": {"role": "Forrest"}}], + pre_merge_vertex_count=4, + pre_merge_edge_count=3, + endpoint_repair_count=1, + ) + warnings = [ + _w(WarningCode.VERTEX_LABEL_NOT_IN_SCHEMA), + _w(WarningCode.PROPERTY_NOT_IN_SCHEMA), + _w(WarningCode.PROPERTY_COERCED), + _w(WarningCode.ENDPOINT_PENDING_REPAIR, item_type="edge"), + _w(WarningCode.DUPLICATE_VERTEX_MERGED), + _w(WarningCode.DUPLICATE_VERTEX_MERGED), + _w(WarningCode.DUPLICATE_EDGE_MERGED, item_type="edge"), + _w(WarningCode.DUPLICATE_EDGE_MERGED, item_type="edge"), + _w(WarningCode.PROPERTY_CONFLICT), + ] + metrics = GraphQualityGate.compute( + graph, + warnings=warnings, + candidate_vertex_count=5, + candidate_edge_count=3, + ) + # Sanity checks — every field populated with a plausible value. + assert metrics.schema_valid_vertex_ratio == 0.8 # 4/5 + assert metrics.schema_valid_edge_ratio == 1.0 # 3/3 + assert metrics.endpoint_resolution_rate == 1.0 # 1 pending, 0 unresolved + assert metrics.duplicate_vertex_reduction == 0.5 # (4-2)/4 + assert metrics.duplicate_edge_reduction == 2 / 3 + assert metrics.dropped_item_count == 2 # 1 vertex + 1 property + assert metrics.coerced_property_count == 1 + assert metrics.endpoint_repair_count == 1 + assert metrics.property_conflict_count == 1 + assert isinstance(metrics, QualityMetrics) diff --git a/hugegraph-llm/src/tests/operators/llm_op/test_property_graph_schema.py b/hugegraph-llm/src/tests/operators/llm_op/test_property_graph_schema.py new file mode 100644 index 000000000..d47c0dbc9 --- /dev/null +++ b/hugegraph-llm/src/tests/operators/llm_op/test_property_graph_schema.py @@ -0,0 +1,363 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +"""Unit tests for the schema-aware graph extraction index (Issue #74).""" + +from __future__ import annotations + +import json + +import pytest + +from hugegraph_llm.operators.llm_op.property_graph_extract_enhanced import GraphSchemaIndex + +pytestmark = pytest.mark.contract + + +def _schema_with_ids() -> dict: + """A minimal but complete schema with vertex-label ids populated. + + Mirrors what SchemaManager returns from a live HugeGraph server (the ``id`` + field is required for baseline canonical id generation). + """ + return { + "propertykeys": [ + {"name": "name", "data_type": "TEXT", "cardinality": "SINGLE"}, + {"name": "title", "data_type": "TEXT", "cardinality": "SINGLE"}, + {"name": "year", "data_type": "INT", "cardinality": "SINGLE"}, + {"name": "rating", "data_type": "FLOAT", "cardinality": "SINGLE"}, + {"name": "is_lead", "data_type": "BOOLEAN", "cardinality": "SINGLE"}, + {"name": "released_on", "data_type": "DATE", "cardinality": "SINGLE"}, + {"name": "aliases", "data_type": "TEXT", "cardinality": "LIST"}, + {"name": "genres", "data_type": "TEXT", "cardinality": "SET"}, + {"name": "role", "data_type": "TEXT", "cardinality": "SINGLE"}, + ], + "vertexlabels": [ + { + "id": 1, + "name": "person", + "properties": ["name", "aliases"], + "primary_keys": ["name"], + "nullable_keys": ["aliases"], + "id_strategy": "PRIMARY_KEY", + }, + { + "id": 2, + "name": "movie", + "properties": ["title", "year", "rating", "released_on", "genres"], + "primary_keys": ["title", "year"], + "nullable_keys": ["rating", "released_on", "genres"], + "id_strategy": "PRIMARY_KEY", + }, + ], + "edgelabels": [ + { + "name": "acted_in", + "source_label": "person", + "target_label": "movie", + "properties": ["role", "is_lead"], + } + ], + } + + +def _schema_without_ids() -> dict: + """A schema that mimics an inline user dict (no vertex-label ``id``).""" + schema = _schema_with_ids() + for v in schema["vertexlabels"]: + v.pop("id", None) + return schema + + +# --------------------------------------------------------------------------- constructor +class TestConstructor: + def test_accepts_dict_schema(self): + idx = GraphSchemaIndex(_schema_with_ids()) + assert idx.is_vertex_label("person") + assert idx.is_edge_label("acted_in") + + def test_accepts_json_string_schema(self): + idx = GraphSchemaIndex.from_schema(json.dumps(_schema_with_ids())) + assert idx.is_vertex_label("person") + + def test_rejects_non_mapping_input(self): + with pytest.raises(TypeError): + GraphSchemaIndex(["not", "a", "dict"]) # type: ignore[arg-type] + + def test_rejects_schema_missing_required_sections(self): + with pytest.raises(ValueError): + GraphSchemaIndex({"vertexlabels": []}) + + def test_rejects_named_graph_string(self): + with pytest.raises(ValueError): + GraphSchemaIndex.from_schema("hugegraph") + + def test_rejects_unparseable_json_string(self): + with pytest.raises(ValueError): + GraphSchemaIndex.from_schema("{not valid json") + + +# ------------------------------------------------------------------------------- labels +class TestLabels: + def setup_method(self): + self.idx = GraphSchemaIndex(_schema_with_ids()) + + def test_vertex_label_lookup(self): + assert self.idx.is_vertex_label("person") + assert not self.idx.is_vertex_label("robot") + assert self.idx.vertex_label("person")["primary_keys"] == ["name"] + assert self.idx.vertex_label("nope") is None + + def test_edge_label_lookup(self): + assert self.idx.is_edge_label("acted_in") + assert not self.idx.is_edge_label("directed") + assert self.idx.edge_label("acted_in")["source_label"] == "person" + + def test_label_name_sets(self): + assert self.idx.vertex_label_names() == frozenset({"person", "movie"}) + assert self.idx.edge_label_names() == frozenset({"acted_in"}) + + +# --------------------------------------------------------------------------- properties +class TestProperties: + def setup_method(self): + self.idx = GraphSchemaIndex(_schema_with_ids()) + + def test_allowed_properties_for_vertex(self): + assert self.idx.allowed_properties("vertex", "person") == frozenset({"name", "aliases"}) + assert self.idx.allowed_properties("vertex", "movie") == frozenset( + {"title", "year", "rating", "released_on", "genres"} + ) + + def test_allowed_properties_for_edge(self): + assert self.idx.allowed_properties("edge", "acted_in") == frozenset({"role", "is_lead"}) + + def test_allowed_properties_unknown_label_or_type(self): + assert self.idx.allowed_properties("vertex", "robot") == frozenset() + assert self.idx.allowed_properties("edge", "directed") == frozenset() + assert self.idx.allowed_properties("triple", "person") == frozenset() + + def test_property_type_and_cardinality_lookup(self): + assert self.idx.property_data_type("year") == "INT" + assert self.idx.property_data_type("released_on") == "DATE" + assert self.idx.property_cardinality("aliases") == "LIST" + assert self.idx.property_cardinality("genres") == "SET" + assert self.idx.property_cardinality("name") == "SINGLE" + + def test_unknown_property_defaults_to_text_single(self): + assert self.idx.property_data_type("mystery") == "TEXT" + assert self.idx.property_cardinality("mystery") == "SINGLE" + + +# ------------------------------------------------------------------------ primary key id +class TestCanonicalVertexId: + def setup_method(self): + self.idx = GraphSchemaIndex(_schema_with_ids()) + + def test_single_primary_key(self): + assert self.idx.canonical_vertex_id("person", {"name": "Tom Hanks"}) == "1:Tom Hanks" + + def test_multi_primary_key_uses_bang_separator(self): + assert self.idx.canonical_vertex_id("movie", {"title": "Forrest Gump", "year": 1994}) == "2:Forrest Gump!1994" + + def test_missing_primary_key_returns_none(self): + assert self.idx.canonical_vertex_id("person", {}) is None + assert self.idx.canonical_vertex_id("movie", {"title": "Forrest Gump"}) is None + + def test_empty_string_primary_key_returns_none(self): + assert self.idx.canonical_vertex_id("person", {"name": ""}) is None + + def test_unknown_label_returns_none(self): + assert self.idx.canonical_vertex_id("robot", {"name": "R2D2"}) is None + + def test_customize_id_strategy_returns_none(self): + schema = _schema_with_ids() + schema["vertexlabels"][0]["id_strategy"] = "CUSTOMIZE_STRING" + idx = GraphSchemaIndex(schema) + assert idx.canonical_vertex_id("person", {"name": "Tom Hanks"}) is None + + def test_schema_without_id_falls_back_to_none(self): + """Inline user schemas without vertex label ``id`` cannot use baseline canonical.""" + idx = GraphSchemaIndex(_schema_without_ids()) + assert idx.canonical_vertex_id("person", {"name": "Tom Hanks"}) is None + + def test_stringifies_non_string_primary_key_values(self): + assert self.idx.canonical_vertex_id("movie", {"title": "Forrest Gump", "year": 1994}) == "2:Forrest Gump!1994" + + +# ------------------------------------------------------------------------------- edges +class TestEdgeEndpoints: + def setup_method(self): + self.idx = GraphSchemaIndex(_schema_with_ids()) + + def test_endpoint_spec_returns_tuple(self): + assert self.idx.edge_endpoint_spec("acted_in") == ("person", "movie") + + def test_endpoint_spec_unknown_edge_returns_none(self): + assert self.idx.edge_endpoint_spec("directed") is None + + def test_endpoint_compatible_matches_direction(self): + assert self.idx.is_endpoint_compatible("acted_in", "person", "movie") + assert not self.idx.is_endpoint_compatible("acted_in", "movie", "person") + + def test_endpoint_compatible_unknown_edge(self): + assert not self.idx.is_endpoint_compatible("directed", "person", "movie") + + +# ---------------------------------------------------------------------- coercion: TEXT +class TestCoerceText: + def setup_method(self): + self.idx = GraphSchemaIndex(_schema_with_ids()) + + def test_text_passthrough(self): + assert self.idx.coerce_property_value("name", "Tom Hanks") == ("Tom Hanks", None) + + def test_text_stringifies_number(self): + assert self.idx.coerce_property_value("name", 42) == ("42", None) + + def test_text_none_rejected(self): + value, reason = self.idx.coerce_property_value("name", None) + assert value is None + assert reason is not None + + +# --------------------------------------------------------------------- coercion: INT +class TestCoerceInt: + def setup_method(self): + self.idx = GraphSchemaIndex(_schema_with_ids()) + + def test_int_passthrough(self): + assert self.idx.coerce_property_value("year", 1994) == (1994, None) + + def test_int_from_lossless_string(self): + assert self.idx.coerce_property_value("year", "1994") == (1994, None) + + def test_int_from_whole_float(self): + assert self.idx.coerce_property_value("year", 1994.0) == (1994, None) + + def test_int_rejects_lossy_float(self): + value, reason = self.idx.coerce_property_value("year", 1994.5) + assert value is None + assert "lossless" in reason + + def test_int_rejects_lossy_string(self): + value, reason = self.idx.coerce_property_value("year", "1994.5") + assert value is None + assert reason is not None + + def test_int_rejects_bool(self): + value, reason = self.idx.coerce_property_value("year", True) + assert value is None + assert "bool" in reason + + +# ------------------------------------------------------------------- coercion: FLOAT +class TestCoerceFloat: + def setup_method(self): + self.idx = GraphSchemaIndex(_schema_with_ids()) + + def test_float_from_int_and_float(self): + assert self.idx.coerce_property_value("rating", 5) == (5.0, None) + assert self.idx.coerce_property_value("rating", 8.5) == (8.5, None) + + def test_float_from_string(self): + assert self.idx.coerce_property_value("rating", "8.5") == (8.5, None) + + def test_float_rejects_garbage(self): + value, reason = self.idx.coerce_property_value("rating", "eight") + assert value is None + assert reason is not None + + def test_float_rejects_bool(self): + value, reason = self.idx.coerce_property_value("rating", False) + assert value is None + assert "bool" in reason + + +# -------------------------------------------------------------- coercion: BOOLEAN +class TestCoerceBoolean: + def setup_method(self): + self.idx = GraphSchemaIndex(_schema_with_ids()) + + def test_bool_passthrough(self): + assert self.idx.coerce_property_value("is_lead", True) == (True, None) + assert self.idx.coerce_property_value("is_lead", False) == (False, None) + + def test_bool_from_yes_no_true_false_strings(self): + assert self.idx.coerce_property_value("is_lead", "yes")[0] is True + assert self.idx.coerce_property_value("is_lead", "True")[0] is True + assert self.idx.coerce_property_value("is_lead", "NO")[0] is False + assert self.idx.coerce_property_value("is_lead", "false")[0] is False + + def test_bool_from_zero_or_one(self): + assert self.idx.coerce_property_value("is_lead", 1)[0] is True + assert self.idx.coerce_property_value("is_lead", 0)[0] is False + + def test_bool_rejects_unrelated_strings(self): + value, reason = self.idx.coerce_property_value("is_lead", "maybe") + assert value is None + assert reason is not None + + +# ------------------------------------------------------------------ coercion: DATE +class TestCoerceDate: + def setup_method(self): + self.idx = GraphSchemaIndex(_schema_with_ids()) + + def test_iso_date_passthrough(self): + assert self.idx.coerce_property_value("released_on", "1994-07-06") == ( + "1994-07-06", + None, + ) + + def test_alt_format_rejected(self): + value, reason = self.idx.coerce_property_value("released_on", "1994/07/06") + assert value is None + assert reason is not None + + def test_non_string_date_rejected(self): + value, reason = self.idx.coerce_property_value("released_on", 19940706) + assert value is None + assert reason is not None + + +# ----------------------------------------------------------- coercion: LIST / SET +class TestCoerceCardinality: + def setup_method(self): + self.idx = GraphSchemaIndex(_schema_with_ids()) + + def test_list_of_text_coerces_each_item(self): + value, reason = self.idx.coerce_property_value("aliases", ["Tom", 42, "Hanks"]) + assert value == ["Tom", "42", "Hanks"] + assert reason is None + + def test_list_expects_list_input(self): + value, reason = self.idx.coerce_property_value("aliases", "not-a-list") + assert value is None + assert reason is not None + + def test_list_drops_none_items_with_warning(self): + value, reason = self.idx.coerce_property_value("aliases", ["Tom", None, "Hanks"]) + assert value == ["Tom", "Hanks"] + assert reason is not None + assert "1 items" in reason + + def test_set_deduplicates_after_coercion(self): + value, reason = self.idx.coerce_property_value("genres", ["drama", "Drama", "drama", "romance"]) + # Deduplication is exact-string on coerced values (case-sensitive by design). + assert value == ["drama", "Drama", "romance"] + assert reason is None diff --git a/hugegraph-llm/src/tests/operators/llm_op/test_property_graph_warnings.py b/hugegraph-llm/src/tests/operators/llm_op/test_property_graph_warnings.py new file mode 100644 index 000000000..3432eca97 --- /dev/null +++ b/hugegraph-llm/src/tests/operators/llm_op/test_property_graph_warnings.py @@ -0,0 +1,289 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +"""Unit tests for the structured warning registry (Issue #74).""" + +from __future__ import annotations + +import dataclasses +import json + +import pytest + +from hugegraph_llm.operators.llm_op.property_graph_extract_enhanced import ( + StructuredWarning, + WarningCode, + warning_code_distribution, +) + +pytestmark = pytest.mark.contract + + +# ---------------------------------------------------------------- WarningCode +class TestWarningCode: + def test_code_is_str_subclass_for_json_encoding(self): + """A ``str, Enum`` code JSON-serializes to its bare name without a custom encoder.""" + payload = {"code": WarningCode.ENDPOINT_UNRESOLVED} + assert json.loads(json.dumps(payload, default=str)) == {"code": "ENDPOINT_UNRESOLVED"} + + def test_code_equals_its_string_value(self): + assert WarningCode.JSON_NOT_FOUND == "JSON_NOT_FOUND" + assert WarningCode.VERTEX_LABEL_NOT_IN_SCHEMA == "VERTEX_LABEL_NOT_IN_SCHEMA" + + def test_code_registry_covers_full_design_surface(self): + """Guard against silent removal — every code the design contract names must exist.""" + expected = { + "JSON_NOT_FOUND", + "JSON_DECODE_FAILED", + "GRAPH_SECTION_MISSING", + "ITEM_NOT_OBJECT", + "ITEM_TYPE_MISMATCH", + "VERTEX_LABEL_NOT_IN_SCHEMA", + "VERTEX_PRIMARY_KEY_MISSING", + "VERTEX_PRIMARY_KEY_INVALID", + "VERTEX_ALIAS_RECORDED", + "EDGE_LABEL_NOT_IN_SCHEMA", + "EDGE_ENDPOINT_MISMATCH", + "PROPERTY_NOT_IN_SCHEMA", + "PROPERTY_COERCED", + "PROPERTY_COERCION_FAILED", + "ENDPOINT_PENDING_REPAIR", + "ENDPOINT_UNRESOLVED", + "ENDPOINT_AMBIGUOUS", + "DUPLICATE_VERTEX_MERGED", + "DUPLICATE_EDGE_MERGED", + "PROPERTY_CONFLICT", + } + actual = {c.value for c in WarningCode} + missing = expected - actual + assert not missing, f"missing codes: {sorted(missing)}" + + +# ---------------------------------------------------------- StructuredWarning +class TestStructuredWarningConstruction: + def test_full_construction(self): + w = StructuredWarning( + code=WarningCode.ENDPOINT_UNRESOLVED, + item_type="edge", + reason="target vertex cannot be resolved", + label="acted_in", + chunk_id=2, + strategy="enhanced", + context={"edge_key": "acted_in|1:Tom Hanks|?"}, + ) + assert w.code is WarningCode.ENDPOINT_UNRESOLVED + assert w.item_type == "edge" + assert w.label == "acted_in" + assert w.chunk_id == 2 + assert w.strategy == "enhanced" + assert w.context == {"edge_key": "acted_in|1:Tom Hanks|?"} + + def test_defaults_are_sensible(self): + w = StructuredWarning( + code=WarningCode.GRAPH_SECTION_MISSING, + item_type="graph", + reason="edges missing", + ) + assert w.label is None + assert w.chunk_id is None + assert w.strategy == "enhanced" + assert w.context is None + + def test_accepts_raw_string_code(self): + """Callers may pass the bare string without importing WarningCode.""" + w = StructuredWarning(code="JSON_NOT_FOUND", item_type="graph", reason="no json") + assert w.code is WarningCode.JSON_NOT_FOUND + + def test_rejects_unknown_string_code(self): + with pytest.raises(ValueError): + StructuredWarning(code="WHATEVER", item_type="graph", reason="") + + def test_rejects_invalid_item_type(self): + with pytest.raises(ValueError): + StructuredWarning( + code=WarningCode.JSON_NOT_FOUND, + item_type="triple", + reason="oops", + ) + + def test_rejects_negative_chunk_id(self): + with pytest.raises(ValueError): + StructuredWarning( + code=WarningCode.JSON_NOT_FOUND, + item_type="graph", + reason="", + chunk_id=-1, + ) + + +# ---------------------------------------------------- StructuredWarning shape +class TestStructuredWarningShape: + def test_is_frozen_dataclass(self): + w = StructuredWarning(code=WarningCode.JSON_NOT_FOUND, item_type="graph", reason="") + with pytest.raises(dataclasses.FrozenInstanceError): + w.reason = "mutated" # type: ignore[misc] + + def test_hashable_and_dedupable(self): + a = StructuredWarning(code=WarningCode.JSON_NOT_FOUND, item_type="graph", reason="x") + b = StructuredWarning(code=WarningCode.JSON_NOT_FOUND, item_type="graph", reason="x") + assert hash(a) == hash(b) + # Warnings without context can be placed in a set. + assert len({a, b}) == 1 + + def test_context_is_copied_defensively(self): + source_ctx = {"key": "value"} + w = StructuredWarning( + code=WarningCode.JSON_NOT_FOUND, + item_type="graph", + reason="", + context=source_ctx, + ) + source_ctx["key"] = "mutated" + assert w.context == {"key": "value"} + + +# ----------------------------------------------------- StructuredWarning.to_dict +class TestSerialization: + def test_to_dict_full(self): + w = StructuredWarning( + code=WarningCode.PROPERTY_COERCED, + item_type="vertex", + reason="year converted from string to INT", + label="movie", + chunk_id=0, + context={"property": "year"}, + ) + assert w.to_dict() == { + "code": "PROPERTY_COERCED", + "item_type": "vertex", + "reason": "year converted from string to INT", + "strategy": "enhanced", + "label": "movie", + "chunk_id": 0, + "context": {"property": "year"}, + } + + def test_to_dict_omits_none_optional_fields(self): + w = StructuredWarning( + code=WarningCode.JSON_NOT_FOUND, + item_type="graph", + reason="no json", + ) + d = w.to_dict() + assert d == { + "code": "JSON_NOT_FOUND", + "item_type": "graph", + "reason": "no json", + "strategy": "enhanced", + } + # Explicit — None fields are absent, not present with a None value. + assert "label" not in d + assert "chunk_id" not in d + assert "context" not in d + + def test_to_dict_is_json_serializable_without_custom_encoder(self): + w = StructuredWarning( + code=WarningCode.DUPLICATE_EDGE_MERGED, + item_type="edge", + reason="merged", + label="acted_in", + chunk_id=3, + context={"count": 2, "keys": ["a", "b"]}, + ) + encoded = json.dumps(w.to_dict()) + assert json.loads(encoded)["code"] == "DUPLICATE_EDGE_MERGED" + + +# ------------------------------------------------------- surface-affecting +class TestSurfaceAffecting: + def test_dropped_items_are_surface_affecting(self): + assert ( + StructuredWarning( + code=WarningCode.VERTEX_LABEL_NOT_IN_SCHEMA, + item_type="vertex", + reason="", + ).is_surface_affecting + is True + ) + assert ( + StructuredWarning( + code=WarningCode.ENDPOINT_UNRESOLVED, + item_type="edge", + reason="", + ).is_surface_affecting + is True + ) + + def test_merges_and_aliases_are_surface_affecting_or_not_per_design(self): + """DUPLICATE_*_MERGED is surface-affecting (item was removed / consolidated); + VERTEX_ALIAS_RECORDED is not (purely observational).""" + assert ( + StructuredWarning( + code=WarningCode.DUPLICATE_VERTEX_MERGED, item_type="vertex", reason="" + ).is_surface_affecting + is True + ) + assert ( + StructuredWarning( + code=WarningCode.VERTEX_ALIAS_RECORDED, item_type="vertex", reason="" + ).is_surface_affecting + is False + ) + + def test_pending_repair_and_section_missing_not_surface_affecting(self): + """Intermediate-state markers do not, by themselves, drop or alter output.""" + assert ( + StructuredWarning( + code=WarningCode.ENDPOINT_PENDING_REPAIR, item_type="edge", reason="" + ).is_surface_affecting + is False + ) + assert ( + StructuredWarning(code=WarningCode.GRAPH_SECTION_MISSING, item_type="graph", reason="").is_surface_affecting + is False + ) + + def test_property_conflict_not_surface_affecting(self): + """First-wins merge conflict retains a value; the conflict is reported but no data is lost.""" + assert ( + StructuredWarning(code=WarningCode.PROPERTY_CONFLICT, item_type="vertex", reason="").is_surface_affecting + is False + ) + + +# ----------------------------------------------------- code distribution helper +class TestWarningCodeDistribution: + def test_empty_input(self): + assert warning_code_distribution([]) == {} + + def test_counts_by_code(self): + warnings = [ + StructuredWarning(code=WarningCode.JSON_NOT_FOUND, item_type="graph", reason=""), + StructuredWarning(code=WarningCode.JSON_NOT_FOUND, item_type="graph", reason=""), + StructuredWarning(code=WarningCode.ENDPOINT_UNRESOLVED, item_type="edge", reason=""), + ] + dist = warning_code_distribution(warnings) + assert dist == {"JSON_NOT_FOUND": 2, "ENDPOINT_UNRESOLVED": 1} + + def test_silently_skips_foreign_objects(self): + """A malformed entry at the tail of a long pipeline should not sink the report.""" + warnings = [ + StructuredWarning(code=WarningCode.JSON_NOT_FOUND, item_type="graph", reason=""), + "not a warning", + {"code": "also not a warning"}, + ] + assert warning_code_distribution(warnings) == {"JSON_NOT_FOUND": 1} diff --git a/pyproject.toml b/pyproject.toml index 43ff9cbcc..7df59016e 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -188,6 +188,7 @@ ignore = [ "tests/**/*.py" = ["T20"] "hugegraph-ml/src/hugegraph_ml/examples/**/*.py" = ["T20"] "hugegraph-python-client/src/pyhugegraph/structure/*.py" = ["N802"] +"scripts/**/*.py" = ["T20"] [tool.ruff.lint.isort] known-first-party = ["hugegraph_llm", "hugegraph_python_client", "hugegraph_ml", "vermeer_python_client"] diff --git a/scripts/build_public_actor_corpus.py b/scripts/build_public_actor_corpus.py new file mode 100644 index 000000000..cdf3cf012 --- /dev/null +++ b/scripts/build_public_actor_corpus.py @@ -0,0 +1,484 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +"""Build a public, reproducible actor-filmography corpus for the schema-based +graph extraction benchmark. + +The output is a JSON file whose *text* comes from Wikipedia article lead +sections and whose *ground truth* is verified against Wikidata's +``P161 (cast member)`` claims. Every corpus entry records the Wikipedia +revision id + Wikidata verification timestamp so a reader can independently +audit every extraction target. + +Rationale: the earlier live benchmark used hand-authored corpora + hand- +authored ground truth, which suffered from two-layer selection bias +(the author chose which failures to expose AND wrote the answer key). +Sourcing text from Wikipedia and answers from Wikidata removes both +biases while still fitting the existing Person + Movie + ACTED_IN schema. + +Usage:: + + python scripts/build_public_actor_corpus.py \\ + --output hugegraph-llm/src/tests/data/public_actor_corpus.json + +Network requirements: + +* ``en.wikipedia.org/w/api.php`` — plain-text lead extract + revision id +* ``www.wikidata.org/w/api.php`` — ``wbsearchentities`` (title -> Q-id) + and ``wbgetentities`` (fetch P161 cast members). No SPARQL is used. + +The script is deliberately kept OUT of the test tree because it depends +on external services. It is only invoked when refreshing the corpus. +""" + +from __future__ import annotations + +import argparse +import json +import re +import sys +import time +import urllib.error +import urllib.parse +import urllib.request +from dataclasses import dataclass, field +from datetime import datetime, timezone +from pathlib import Path +from typing import Any + +USER_AGENT = "Mozilla/5.0 (compatible; hugegraph-ai-research/1.0; +https://github.com/apache/hugegraph-ai)" +WIKIPEDIA_API = "https://en.wikipedia.org/w/api.php" +WIKIDATA_API = "https://www.wikidata.org/w/api.php" + +# Actors chosen for lead-section density of specific "Film (YYYY)" mentions. +# Q-ids are auto-resolved at build time, not hard-coded here. +ACTORS: list[str] = [ + "Tom Hanks", + "Meryl Streep", + "Leonardo DiCaprio", + "Denzel Washington", + "Julia Roberts", + "Anthony Hopkins", + "Nicole Kidman", + "Morgan Freeman", +] + +# Matches "Title Case Phrase (YYYY)" — the standard Wikipedia film-citation +# form. Deliberately permissive on characters common in film titles. +FILM_YEAR_RE = re.compile( + r"(?:(?<=^)|(?<=[\s\"'(]))" + r"([A-Z][A-Za-z0-9\-'.:!&,]*(?:\s+(?:the|of|and|a|an|in|on|to|for|with|" + r"vs\.|von|de|le|la|el)?\s*[A-Z0-9][A-Za-z0-9\-'.:!&,]*){0,6})" + r"\s+\((\d{4})\)" +) + + +@dataclass +class ActorRecord: + name: str + qid: str + wikipedia_title: str + wikipedia_revid: int + extract: str + chunks: list[str] + vertices: list[dict[str, Any]] + edges: list[dict[str, Any]] + verified_films: list[dict[str, Any]] = field(default_factory=list) + rejected_mentions: list[dict[str, Any]] = field(default_factory=list) + + +def http_json(url: str, retries: int = 5) -> dict[str, Any]: + """GETs a URL that returns JSON, retrying on transient errors. + + Uses generous backoff because Wikidata is currently 429-limiting during + an active WDQS outage — even the entity API is stressed. Delays: + 30s, 60s, 90s, 120s. + """ + last_err: Exception | None = None + for attempt in range(retries): + try: + req = urllib.request.Request( + url, + headers={"User-Agent": USER_AGENT, "Accept": "application/json"}, + ) + with urllib.request.urlopen(req, timeout=30) as resp: + return json.loads(resp.read().decode("utf-8")) + except urllib.error.HTTPError as e: + last_err = e + if e.code == 429 and attempt < retries - 1: + wait = 30 * (attempt + 1) + print(f" 429 rate-limited, sleeping {wait}s...", file=sys.stderr) + time.sleep(wait) + elif attempt < retries - 1: + time.sleep(2) + else: + break + except Exception as e: + last_err = e + if attempt < retries - 1: + time.sleep(2) + assert last_err is not None + raise last_err + + +# Cache Wikidata search results across actors so shared film titles don't +# waste API calls (many actors are cited alongside the same well-known films). +_QID_CACHE: dict[str, str | None] = {} +_WIKIDATA_CALL_INTERVAL = 5.0 # seconds — Wikidata is currently stressed +_last_wikidata_call = 0.0 + + +def _pace_wikidata() -> None: + global _last_wikidata_call + elapsed = time.perf_counter() - _last_wikidata_call + if elapsed < _WIKIDATA_CALL_INTERVAL: + time.sleep(_WIKIDATA_CALL_INTERVAL - elapsed) + _last_wikidata_call = time.perf_counter() + + +def fetch_wikipedia_lead(title: str) -> tuple[str, int, str]: + """Returns (plain-text lead extract, revision id, canonical title).""" + params = { + "action": "query", + "prop": "extracts|revisions", + "exintro": "1", + "explaintext": "1", + "rvprop": "ids", + "redirects": "1", + "format": "json", + "titles": title, + } + url = f"{WIKIPEDIA_API}?{urllib.parse.urlencode(params)}" + data = http_json(url) + pages = data["query"]["pages"] + page = next(iter(pages.values())) + if "extract" not in page: + raise RuntimeError(f"no extract for {title!r}: {page}") + return page["extract"], int(page["revisions"][0]["revid"]), page["title"] + + +def resolve_qid(title: str) -> str | None: + """Uses wbsearchentities to find a Wikidata Q-id for a given label.""" + if title in _QID_CACHE: + return _QID_CACHE[title] + params = { + "action": "wbsearchentities", + "search": title, + "language": "en", + "type": "item", + "limit": "5", + "format": "json", + } + url = f"{WIKIDATA_API}?{urllib.parse.urlencode(params)}" + _pace_wikidata() + data = http_json(url) + hits = data.get("search", []) + qid = hits[0]["id"] if hits else None + _QID_CACHE[title] = qid + return qid + + +def fetch_wikidata_entities(qids: list[str]) -> dict[str, dict[str, Any]]: + """Fetches claims + labels for up to 50 Q-ids per call (Wikidata cap).""" + if not qids: + return {} + results: dict[str, dict[str, Any]] = {} + for i in range(0, len(qids), 50): + batch = qids[i : i + 50] + params = { + "action": "wbgetentities", + "ids": "|".join(batch), + "props": "claims|labels", + "languages": "en", + "format": "json", + } + url = f"{WIKIDATA_API}?{urllib.parse.urlencode(params)}" + _pace_wikidata() + data = http_json(url) + results.update(data.get("entities", {})) + return results + + +def is_cast_member(entity: dict[str, Any], actor_qid: str) -> bool: + """True if the film entity has actor_qid under P161 (cast member).""" + for claim in entity.get("claims", {}).get("P161", []): + try: + value_id = claim["mainsnak"]["datavalue"]["value"]["id"] + except (KeyError, TypeError): + continue + if value_id == actor_qid: + return True + return False + + +def is_film_instance(entity: dict[str, Any]) -> bool: + """True if the entity's ``instance of`` (P31) lists a film class. + + Excludes TV series, books, video games, etc. Films are typically: + Q11424 (film), Q24856 (film series), Q506240 (television film), + Q202866 (animated film), Q229390 (3D film), Q506240 (TV film). + """ + film_classes = { + "Q11424", # film + "Q24856", # film series + "Q202866", # animated film + "Q506240", # television film + "Q220898", # feature film + "Q351658", # live-action film + "Q93204", # science fiction film + } + for claim in entity.get("claims", {}).get("P31", []): + try: + value_id = claim["mainsnak"]["datavalue"]["value"]["id"] + except (KeyError, TypeError): + continue + if value_id in film_classes: + return True + return False + + +def extract_publication_year(entity: dict[str, Any]) -> int | None: + """Reads P577 (publication date) and returns the earliest year.""" + years: list[int] = [] + for claim in entity.get("claims", {}).get("P577", []): + try: + time_str = claim["mainsnak"]["datavalue"]["value"]["time"] + year = int(time_str.lstrip("+").split("-")[0]) + years.append(year) + except (KeyError, TypeError, ValueError): + continue + return min(years) if years else None + + +def canonical_label(entity: dict[str, Any]) -> str | None: + """Wikidata's English label for the entity.""" + return entity.get("labels", {}).get("en", {}).get("value") + + +def chunk_extract(text: str, target_chunks: int = 3) -> list[str]: + """Splits an extract into ~equal chunks on sentence boundaries.""" + sentences = re.split(r"(?<=[.!?])\s+", text.strip()) + sentences = [s for s in sentences if s] + if len(sentences) <= target_chunks: + return sentences + per_chunk = max(1, len(sentences) // target_chunks) + chunks: list[str] = [] + for i in range(0, len(sentences), per_chunk): + chunk = " ".join(sentences[i : i + per_chunk]) + chunks.append(chunk) + # Fold trailing tiny chunk into previous if we overshot the target. + while len(chunks) > target_chunks: + chunks[-2] = chunks[-2] + " " + chunks[-1] + chunks.pop() + return chunks + + +def dedupe_mentions(pairs: list[tuple[str, str]]) -> list[tuple[str, str]]: + """Removes near-duplicate (title, year) tuples, keeping first occurrence.""" + seen: set[str] = set() + out: list[tuple[str, str]] = [] + for title, year in pairs: + norm = title.strip().lower() + if norm in seen: + continue + seen.add(norm) + out.append((title.strip(), year)) + return out + + +def build_actor_record(actor_name: str) -> ActorRecord | None: + print(f"[{actor_name}] resolving Wikidata Q-id ...", file=sys.stderr) + actor_qid = resolve_qid(actor_name) + if actor_qid is None: + print(f" ! no Wikidata match for {actor_name!r}, skipping", file=sys.stderr) + return None + + print(f"[{actor_name}] Q-id={actor_qid}, fetching Wikipedia lead ...", file=sys.stderr) + extract, revid, canonical_title = fetch_wikipedia_lead(actor_name) + print(f" extract={len(extract)} chars, rev={revid}", file=sys.stderr) + + raw_pairs = FILM_YEAR_RE.findall(extract) + candidate_pairs = dedupe_mentions(raw_pairs) + # Cap per-actor candidates to bound runtime under Wikidata's aggressive + # rate limiting. First N unique mentions is enough — the intro tends to + # list a wide filmography and 15 films × 8 actors gives ~120 GT items. + candidate_pairs = candidate_pairs[:15] + print(f" candidate film mentions (capped at 15): {len(candidate_pairs)}", file=sys.stderr) + + # Resolve each candidate title to a Wikidata Q-id. + title_to_qid: dict[str, str] = {} + for title, _year in candidate_pairs: + qid = resolve_qid(title) + if qid is not None: + title_to_qid[title] = qid + + # Batch-fetch entities so we only pay once per Q-id. + entities = fetch_wikidata_entities(list(title_to_qid.values())) + + verified: list[dict[str, Any]] = [] + rejected: list[dict[str, Any]] = [] + for title, mentioned_year in candidate_pairs: + qid = title_to_qid.get(title) + if qid is None: + rejected.append({"title": title, "year": mentioned_year, "reason": "no_wikidata_match"}) + continue + entity = entities.get(qid) + if entity is None: + rejected.append({"title": title, "year": mentioned_year, "reason": "entity_fetch_failed"}) + continue + if not is_film_instance(entity): + rejected.append({"title": title, "year": mentioned_year, "reason": "not_a_film", "qid": qid}) + continue + if not is_cast_member(entity, actor_qid): + rejected.append({"title": title, "year": mentioned_year, "reason": "actor_not_in_cast", "qid": qid}) + continue + wikidata_year = extract_publication_year(entity) + canon_title = canonical_label(entity) or title + verified.append( + { + "title": canon_title, + "mentioned_title": title, + "year": wikidata_year if wikidata_year is not None else int(mentioned_year), + "wikidata_qid": qid, + } + ) + + print( + f" verified={len(verified)} rejected={len(rejected)}", + file=sys.stderr, + ) + + # Ground-truth graph. + person_id = f"1:{actor_name}" + vertices: list[dict[str, Any]] = [ + { + "label": "Person", + "type": "vertex", + "id": person_id, + "properties": {"name": actor_name}, + } + ] + edges: list[dict[str, Any]] = [] + seen_movie_ids: set[str] = set() + for f in verified: + movie_id = f"2:{f['title']}" + if movie_id not in seen_movie_ids: + vertices.append( + { + "label": "Movie", + "type": "vertex", + "id": movie_id, + "properties": {"title": f["title"], "year": f["year"]}, + } + ) + seen_movie_ids.add(movie_id) + edges.append( + { + "label": "ACTED_IN", + "type": "edge", + "outV": person_id, + "inV": movie_id, + "outVLabel": "Person", + "inVLabel": "Movie", + "properties": {}, + } + ) + + chunks = chunk_extract(extract, target_chunks=3) + return ActorRecord( + name=actor_name, + qid=actor_qid, + wikipedia_title=canonical_title, + wikipedia_revid=revid, + extract=extract, + chunks=chunks, + vertices=vertices, + edges=edges, + verified_films=verified, + rejected_mentions=rejected, + ) + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--output", + default="hugegraph-llm/src/tests/data/public_actor_corpus.json", + help="Destination JSON path (relative to repo root).", + ) + args = parser.parse_args() + + repo_root = Path(__file__).resolve().parent.parent + out_path = repo_root / args.output + + records: list[ActorRecord] = [] + for actor in ACTORS: + try: + record = build_actor_record(actor) + except Exception as e: + print(f"[{actor}] ERROR: {type(e).__name__}: {e}", file=sys.stderr) + continue + if record is None: + continue + records.append(record) + time.sleep(3.0) # polite spacing between actors + + total_vertices = sum(len(r.vertices) for r in records) + total_edges = sum(len(r.edges) for r in records) + print( + f"\nBuilt {len(records)} actor corpora: {total_vertices} vertices, {total_edges} edges", + file=sys.stderr, + ) + + payload = { + "meta": { + "schema_source": "Person + Movie + ACTED_IN (matches live_benchmark schema)", + "text_source": "Wikipedia lead extract (via /w/api.php action=query prop=extracts exintro)", + "ground_truth_source": ( + "Wikidata P161 (cast member) verified per film via wbgetentities. " + "Only films whose Wikidata entity is an instance-of (P31) a film class AND " + "whose P161 claims include the actor's Q-id are admitted." + ), + "built_at_utc": datetime.now(timezone.utc).isoformat(), + "actor_count": len(records), + "total_vertices": total_vertices, + "total_edges": total_edges, + }, + "corpora": [ + { + "name": r.name, + "actor_qid": r.qid, + "wikipedia_title": r.wikipedia_title, + "wikipedia_revision": r.wikipedia_revid, + "wikipedia_url": (f"https://en.wikipedia.org/w/index.php?oldid={r.wikipedia_revid}"), + "text": r.extract, + "chunks": r.chunks, + "ground_truth": {"vertices": r.vertices, "edges": r.edges}, + "verified_films": r.verified_films, + "rejected_mentions": r.rejected_mentions, + } + for r in records + ], + } + out_path.parent.mkdir(parents=True, exist_ok=True) + out_path.write_text(json.dumps(payload, ensure_ascii=False, indent=2), encoding="utf-8") + print(f"Wrote corpus to {out_path}", file=sys.stderr) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/graph_extract_live_benchmark.py b/scripts/graph_extract_live_benchmark.py new file mode 100644 index 000000000..eb16bcbf5 --- /dev/null +++ b/scripts/graph_extract_live_benchmark.py @@ -0,0 +1,475 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +"""Live-LLM benchmark for the schema-based graph extraction strategies. + +Runs the baseline and enhanced extraction pipelines against a real LLM +(DeepSeek Chat via LiteLLM) on a public, externally-authored corpus, and +reports quality (F1 vs. Wikidata-verified ground truth), latency +(wall-clock), LLM call count, prompt/completion tokens, and an +approximate USD cost per run. + +This script is deliberately kept OUT of the pytest test tree — it costs +real money and requires network + DEEPSEEK_API_KEY. It is only invoked +manually to produce the "Live LLM Benchmark" section of the effect +report. + +Usage: + + # DEEPSEEK_API_KEY must be set (loaded from .env.local by default). + python scripts/graph_extract_live_benchmark.py \\ + --corpus hugegraph-llm/src/tests/data/public_actor_corpus.json \\ + --runs 3 + +The ``--corpus`` argument is required. Build a corpus with +``scripts/build_public_actor_corpus.py``. No hand-authored corpus is +embedded in this script — that would defeat the reproducibility goal +(text and ground truth must both be third-party-authored so a reviewer +can audit them). + +Output: + + * Human-readable table to stdout. + * Full run record (per-strategy metrics + LLM I/O) to + ``--output`` (defaults to ``.workflow/deepseek_live_run.json``). + +Cost estimates use DeepSeek Chat pricing as published at +https://api-docs.deepseek.com/quick_start/pricing at the time of the +run. Rates are hard-coded below; update if pricing changes. +""" + +from __future__ import annotations + +import argparse +import json +import math +import os +import statistics +import sys +import time +from dataclasses import asdict, dataclass, field +from pathlib import Path +from typing import Any + +# Ensure the local hugegraph-llm package is importable when invoked from repo root. +_REPO_ROOT = Path(__file__).resolve().parent.parent +sys.path.insert(0, str(_REPO_ROOT / "hugegraph-llm" / "src")) + +from hugegraph_llm.operators.llm_op.property_graph_extract import PropertyGraphExtract # noqa: E402 +from hugegraph_llm.operators.llm_op.property_graph_extract_enhanced import ( # noqa: E402 + GraphExtractionEvaluator, + GraphSchemaIndex, +) + +# DeepSeek Chat pricing as of 2026-07 (per 1M tokens, cache-miss standard rate). +# Update if pricing shifts. Source: https://api-docs.deepseek.com/quick_start/pricing +DEEPSEEK_INPUT_USD_PER_M = 0.27 +DEEPSEEK_OUTPUT_USD_PER_M = 1.10 +DEEPSEEK_MODEL_LITELLM = "deepseek/deepseek-chat" + + +SCHEMA: dict[str, Any] = { + "propertykeys": [ + {"name": "name", "data_type": "TEXT", "cardinality": "SINGLE"}, + {"name": "age", "data_type": "INT", "cardinality": "SINGLE"}, + {"name": "title", "data_type": "TEXT", "cardinality": "SINGLE"}, + {"name": "year", "data_type": "INT", "cardinality": "SINGLE"}, + {"name": "role", "data_type": "TEXT", "cardinality": "SINGLE"}, + ], + "vertexlabels": [ + { + "id": 1, + "name": "Person", + "id_strategy": "PRIMARY_KEY", + "primary_keys": ["name"], + "properties": ["name", "age"], + "nullable_keys": ["age"], + }, + { + "id": 2, + "name": "Movie", + "id_strategy": "PRIMARY_KEY", + "primary_keys": ["title"], + "properties": ["title", "year"], + "nullable_keys": ["year"], + }, + ], + "edgelabels": [ + { + "name": "ACTED_IN", + "source_label": "Person", + "target_label": "Movie", + "properties": ["role"], + }, + ], +} + + +EXTRACT_PROMPT_HEADER = """You are extracting a property graph from text. + +Output ONLY a JSON object with two keys: "vertices" and "edges". +Every vertex has: label, type: "vertex", id, properties (object). +Every edge has: label, type: "edge", outV, inV, outVLabel, inVLabel, properties (object). + +Do not include any text outside the JSON. +""" + + +@dataclass +class LLMCallRecord: + prompt_tokens: int + completion_tokens: int + total_tokens: int + latency_ms: float + + +@dataclass +class StrategyRunMetrics: + strategy: str + predicted: dict[str, Any] + total_latency_ms: float + call_count: int + prompt_tokens_total: int + completion_tokens_total: int + tokens_total: int + cost_usd_estimate: float + calls: list[dict[str, Any]] = field(default_factory=list) + + +class TrackedDeepSeekLLM: + """LLM adapter that captures per-call token usage + wall-clock latency.""" + + def __init__(self, api_key: str, model: str = DEEPSEEK_MODEL_LITELLM) -> None: + from litellm import completion # imported lazily so the module imports without litellm on non-live paths + + self._completion = completion + self._api_key = api_key + self._model = model + self.calls: list[LLMCallRecord] = [] + + def generate(self, prompt: str | None = None, messages: list[dict[str, Any]] | None = None, **_) -> str: + if messages is None: + assert prompt is not None + messages = [{"role": "user", "content": prompt}] + start = time.perf_counter() + response = self._completion( + model=self._model, + messages=messages, + api_key=self._api_key, + temperature=0.0, + max_tokens=2048, + ) + elapsed_ms = (time.perf_counter() - start) * 1000.0 + usage = response.usage + self.calls.append( + LLMCallRecord( + prompt_tokens=int(getattr(usage, "prompt_tokens", 0) or 0), + completion_tokens=int(getattr(usage, "completion_tokens", 0) or 0), + total_tokens=int(getattr(usage, "total_tokens", 0) or 0), + latency_ms=elapsed_ms, + ) + ) + return response.choices[0].message.content + + +@dataclass +class SingleRunResult: + """One (corpus, strategy, run-index) trial with metrics + evaluation.""" + + corpus_name: str + strategy: str + run_index: int + predicted: dict[str, Any] + total_latency_ms: float + call_count: int + prompt_tokens_total: int + completion_tokens_total: int + tokens_total: int + cost_usd_estimate: float + calls: list[dict[str, Any]] = field(default_factory=list) + overall_f1: float = 0.0 + vertex_f1: float = 0.0 + edge_f1: float = 0.0 + property_match_rate: float = 0.0 + evaluation: dict[str, Any] | None = None + + +def _estimate_cost(prompt_tokens: int, completion_tokens: int) -> float: + return ( + prompt_tokens * DEEPSEEK_INPUT_USD_PER_M / 1_000_000.0 + + completion_tokens * DEEPSEEK_OUTPUT_USD_PER_M / 1_000_000.0 + ) + + +def _run_strategy_on_corpus( + strategy: str, + api_key: str, + schema: dict[str, Any], + chunks: list[str], + expected: dict[str, Any], + corpus_name: str, + run_index: int, + evaluator: GraphExtractionEvaluator, +) -> SingleRunResult: + """Runs one strategy against one corpus once; returns a SingleRunResult + with LLM metrics and evaluation baked in. + """ + llm = TrackedDeepSeekLLM(api_key=api_key) + extractor = PropertyGraphExtract( + llm=llm, + example_prompt=EXTRACT_PROMPT_HEADER, + extract_strategy=strategy, + ) + context = {"schema": schema, "chunks": list(chunks)} + start = time.perf_counter() + result = extractor.run(context) + total_latency_ms = (time.perf_counter() - start) * 1000.0 + + prompt_tokens_total = sum(c.prompt_tokens for c in llm.calls) + completion_tokens_total = sum(c.completion_tokens for c in llm.calls) + tokens_total = sum(c.total_tokens for c in llm.calls) + + predicted = {"vertices": result.get("vertices", []), "edges": result.get("edges", [])} + evaluation = evaluator.evaluate(predicted, expected) + + return SingleRunResult( + corpus_name=corpus_name, + strategy=strategy, + run_index=run_index, + predicted=predicted, + total_latency_ms=total_latency_ms, + call_count=int(result.get("call_count", 0) or len(llm.calls)), + prompt_tokens_total=prompt_tokens_total, + completion_tokens_total=completion_tokens_total, + tokens_total=tokens_total, + cost_usd_estimate=_estimate_cost(prompt_tokens_total, completion_tokens_total), + calls=[asdict(c) for c in llm.calls], + overall_f1=evaluation.overall_f1, + vertex_f1=evaluation.vertex_metrics.f1, + edge_f1=evaluation.edge_metrics.f1, + property_match_rate=evaluation.property_metrics.property_exact_match_rate, + evaluation=evaluation.to_dict(), + ) + + +def _agg_stats(values: list[float]) -> dict[str, float]: + """mean/std/min/max over a list of floats. std is 0 for n<2 (documented).""" + if not values: + return {"mean": 0.0, "std": 0.0, "min": 0.0, "max": 0.0, "n": 0} + return { + "mean": statistics.fmean(values), + "std": statistics.pstdev(values) if len(values) >= 2 else 0.0, + "min": min(values), + "max": max(values), + "n": len(values), + } + + +def _aggregate(runs: list[SingleRunResult]) -> dict[str, Any]: + """Aggregate a set of runs into mean/std/min/max on the headline metrics.""" + return { + "runs": len(runs), + "overall_f1": _agg_stats([r.overall_f1 for r in runs]), + "vertex_f1": _agg_stats([r.vertex_f1 for r in runs]), + "edge_f1": _agg_stats([r.edge_f1 for r in runs]), + "property_match_rate": _agg_stats([r.property_match_rate for r in runs]), + "latency_ms": _agg_stats([r.total_latency_ms for r in runs]), + "prompt_tokens_total": _agg_stats([float(r.prompt_tokens_total) for r in runs]), + "completion_tokens_total": _agg_stats([float(r.completion_tokens_total) for r in runs]), + "cost_usd_estimate": _agg_stats([r.cost_usd_estimate for r in runs]), + } + + +def _load_env_local() -> None: + """Loads .env.local into os.environ. Silently no-ops if the file is absent.""" + env_path = _REPO_ROOT / ".env.local" + if not env_path.exists(): + return + for raw in env_path.read_text(encoding="utf-8").splitlines(): + line = raw.strip() + if not line or line.startswith("#"): + continue + if "=" not in line: + continue + key, _, value = line.partition("=") + os.environ.setdefault(key.strip(), value.strip()) + + +def _load_corpora(corpus_path: str) -> list[dict[str, Any]]: + """Loads corpora from a public corpus JSON built by + ``scripts/build_public_actor_corpus.py``. + + A corpus is required — this script no longer accepts an implicit + hand-authored fallback. The whole point of the live benchmark is to + measure effect on externally-authored text + ground truth; embedding + a hand-authored corpus in the script would silently reintroduce the + selection bias the public-corpus track exists to remove. + """ + data = json.loads(Path(corpus_path).read_text(encoding="utf-8")) + corpora: list[dict[str, Any]] = [] + for c in data["corpora"]: + corpora.append( + { + "name": c["name"], + "chunks": c["chunks"], + "ground_truth": c["ground_truth"], + "source": { + "wikipedia_revision": c.get("wikipedia_revision"), + "wikipedia_url": c.get("wikipedia_url"), + "actor_qid": c.get("actor_qid"), + "verified_films": len(c.get("verified_films", [])), + }, + } + ) + return corpora + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--corpus", + required=True, + help=( + "Path to a public corpus JSON built by " + "scripts/build_public_actor_corpus.py. Required — no implicit " + "hand-authored fallback (deliberate; see module docstring)." + ), + ) + parser.add_argument( + "--runs", + type=int, + default=3, + help=( + "Number of runs per (corpus, strategy) for variance estimation. " + "Default 3. Set to 1 for a quick single-run smoke." + ), + ) + parser.add_argument( + "--output", + default=None, + help=("Where to archive the full run JSON. Defaults to .workflow/deepseek_live_run.json (kept out of git)."), + ) + args = parser.parse_args() + + _load_env_local() + api_key = os.environ.get("DEEPSEEK_API_KEY") + if not api_key: + print("ERROR: DEEPSEEK_API_KEY is not set (checked .env.local + environment).") + return 2 + + corpora = _load_corpora(args.corpus) + print(f"Loaded {len(corpora)} corpora (runs per strategy per corpus: {args.runs})") + evaluator = GraphExtractionEvaluator(GraphSchemaIndex(SCHEMA)) + + all_runs: list[SingleRunResult] = [] + for corpus in corpora: + for strategy in ("baseline", "enhanced"): + for run_i in range(args.runs): + print( + f" {corpus['name']} / {strategy} / run{run_i + 1}/{args.runs} ...", + flush=True, + ) + r = _run_strategy_on_corpus( + strategy=strategy, + api_key=api_key, + schema=SCHEMA, + chunks=corpus["chunks"], + expected=corpus["ground_truth"], + corpus_name=corpus["name"], + run_index=run_i, + evaluator=evaluator, + ) + all_runs.append(r) + print( + f" F1={r.overall_f1:.3f} vF1={r.vertex_f1:.3f} eF1={r.edge_f1:.3f} " + f"latency={r.total_latency_ms / 1000:.2f}s " + f"tokens={r.tokens_total} cost=${r.cost_usd_estimate:.6f}" + ) + + # ----- per-(corpus, strategy) aggregation ----- + by_key: dict[tuple[str, str], list[SingleRunResult]] = {} + for r in all_runs: + by_key.setdefault((r.corpus_name, r.strategy), []).append(r) + + per_pair_agg: dict[str, dict[str, Any]] = {} + for (corpus_name, strategy), runs in by_key.items(): + per_pair_agg[f"{corpus_name}|{strategy}"] = _aggregate(runs) + + # ----- per-strategy aggregation across all runs ----- + per_strategy: dict[str, dict[str, Any]] = {} + for strategy in ("baseline", "enhanced"): + strategy_runs = [r for r in all_runs if r.strategy == strategy] + per_strategy[strategy] = _aggregate(strategy_runs) + + baseline_f1 = per_strategy["baseline"]["overall_f1"]["mean"] + enhanced_f1 = per_strategy["enhanced"]["overall_f1"]["mean"] + f1_absolute_delta = enhanced_f1 - baseline_f1 + f1_relative_pct = (f1_absolute_delta / baseline_f1 * 100.0) if baseline_f1 > 0 else float("nan") + + # ----- summary table ----- + print("\n=== Summary (mean +/- std across runs and corpora) ===") + print( + f" baseline: F1={baseline_f1:.3f} +/- {per_strategy['baseline']['overall_f1']['std']:.3f} " + f"latency={per_strategy['baseline']['latency_ms']['mean'] / 1000:.2f}s " + f"cost=${per_strategy['baseline']['cost_usd_estimate']['mean']:.6f}" + ) + print( + f" enhanced: F1={enhanced_f1:.3f} +/- {per_strategy['enhanced']['overall_f1']['std']:.3f} " + f"latency={per_strategy['enhanced']['latency_ms']['mean'] / 1000:.2f}s " + f"cost=${per_strategy['enhanced']['cost_usd_estimate']['mean']:.6f}" + ) + if not math.isnan(f1_relative_pct): + print(f" delta F1: {f1_absolute_delta:+.3f} absolute ({f1_relative_pct:+.1f}% relative)") + + # ----- persist ----- + workflow_dir = _REPO_ROOT / ".workflow" + workflow_dir.mkdir(exist_ok=True) + out_path = Path(args.output) if args.output else workflow_dir / "deepseek_live_run.json" + payload = { + "model": DEEPSEEK_MODEL_LITELLM, + "pricing": { + "input_usd_per_m": DEEPSEEK_INPUT_USD_PER_M, + "output_usd_per_m": DEEPSEEK_OUTPUT_USD_PER_M, + }, + "corpus_source": args.corpus, + "runs_per_strategy_per_corpus": args.runs, + "schema": SCHEMA, + "corpora": [ + { + "name": c["name"], + "chunks": c["chunks"], + "ground_truth": c["ground_truth"], + "source": c.get("source"), + } + for c in corpora + ], + "runs": [asdict(r) for r in all_runs], + "per_corpus_strategy_aggregation": per_pair_agg, + "per_strategy_aggregation": per_strategy, + "delta": { + "f1_absolute": f1_absolute_delta, + "f1_relative_percent": f1_relative_pct, + }, + } + out_path.parent.mkdir(parents=True, exist_ok=True) + out_path.write_text(json.dumps(payload, ensure_ascii=False, indent=2), encoding="utf-8") + print(f"\nFull run archived to {out_path}") + return 0 + + +if __name__ == "__main__": + sys.exit(main())