feat: topic graph quality scoring and auto-prune pipeline

docs/cli/topic-inspect.md

@@ -19,6 +19,8 @@ The command auto-detects whether the file is a tree (JSONL) or graph (JSON) and
 | `--all` | `-a` | Show the complete tree structure |
 | `--format FMT` | `-f` | Output format: `tree` (default), `table`, or `json` |
 | `--uuid` | `-u` | Show UUID/topic_id for each node |
+| `--score-report PATH` |  | Score report JSON from `topic score` (graph files only) |
+| `--show-pruned MODE` |  | Overlay flagged/pruned nodes from `--score-report`: `all` or `flagged-only` |
 
 ## Exploring by Level
 
@@ -120,6 +122,29 @@ deepfabric topic inspect topics.json --level 2 --uuid
 !!! tip "Use with Prune"
     UUIDs discovered here can be passed to `deepfabric topic prune --uuid` to remove specific branches. See [topic prune](topic-prune.md).
 
+## Prune Overlay
+
+For graph JSON files, you can overlay score output to preview which nodes would be pruned:
+
+```bash title="Inspect with prune overlay"
+deepfabric topic inspect topics.json --all \
+  --score-report topics_score_report.json \
+  --show-pruned all
+```
+
+Show only affected branches:
+
+```bash
+deepfabric topic inspect topics.json --all \
+  --score-report topics_score_report.json \
+  --show-pruned flagged-only
+```
+
+Overlay legend:
+
+- yellow: directly flagged by thresholds
+- red: would be pruned (including descendants of flagged nodes)
+
 ## Format Auto-Detection
 
 The inspect command determines the file format automatically:

docs/cli/topic-optimize-thresholds.md

@@ -0,0 +1,38 @@
+# topic optimize-thresholds
+
+The `topic optimize-thresholds` command searches for better GTD/LTD cutoff values on an existing graph.
+
+It does **not** regenerate the graph. It re-scores the same graph across multiple threshold combinations.
+
+## Usage
+
+```bash
+deepfabric topic optimize-thresholds topic_graph.json
+```
+
+By default this runs random search with 40 trials and writes:
+`<input>_threshold_optimization.json`
+
+## Example
+
+```bash
+deepfabric topic optimize-thresholds topic_graph.json \
+  --search random \
+  --trials 40 \
+  --depth1-gtd-min 0.10 --depth1-gtd-max 0.50 \
+  --gtd-neg-min -0.10 --gtd-neg-max 0.10 \
+  --ltd-min 0.10 --ltd-max 0.50 \
+  --output-report best_thresholds.json
+```
+
+## Constraints
+
+You can constrain acceptable solutions:
+
+```bash
+deepfabric topic optimize-thresholds topic_graph.json \
+  --max-removed-ratio 0.35 \
+  --max-internal-removed 120
+```
+
+If no trial satisfies constraints, the command falls back to the best unconstrained result and records this in the output report.

docs/cli/topic-score.md

@@ -0,0 +1,53 @@
+# topic score
+
+The `topic score` command evaluates graph quality using embedding-based drift metrics:
+
+- **GTD (Global Topic Drift)**: cosine similarity to the root topic
+- **LTD (Local Topic Drift)**: max cosine similarity to parent topic(s)
+
+It writes a JSON report with per-node metrics, flagged nodes, and estimated removal impact.
+
+## Usage
+
+```bash
+deepfabric topic score topic_graph.json
+```
+
+By default, this writes `<input>_score_report.json`.
+
+## Options
+
+| Option | Description |
+|--------|-------------|
+| `--output-report, -o` | Output path for report JSON |
+| `--depth1-gtd` | Flag depth-1 nodes with GTD below this threshold |
+| `--gtd-neg` | Flag nodes with GTD below this threshold |
+| `--ltd` | Flag nodes with LTD below this threshold |
+| `--embedding-key` | Metadata key used for node embeddings |
+| `--embedding-model` | SentenceTransformer model used if embeddings are missing |
+
+## Example
+
+```bash
+deepfabric topic score topic_graph.json \
+  --depth1-gtd 0.25 \
+  --gtd-neg 0.0 \
+  --ltd 0.25 \
+  --output-report report.json
+```
+
+## Report Contents
+
+The report includes:
+
+- `summary`
+  - node counts (original, flagged, estimated removed, estimated remaining)
+  - GTD/LTD distribution stats
+  - depth distribution
+  - threshold config used
+- `metrics_per_node` with GTD/LTD values
+- `flagged_nodes` with reasons
+- `removed_node_ids` estimated by subtree propagation from flagged nodes
+
+!!! note
+    If embeddings are not present in node metadata, DeepFabric attempts to generate them with `sentence-transformers`.

docs/cli/topic.md

@@ -22,6 +22,22 @@ The `topic` command group provides tools for inspecting and managing topic struc
 
     [:octicons-arrow-right-24: Reference](topic-prune.md)
 
+-   :material-chart-line: **topic score**
+
+    ---
+
+    Score topic graphs with GTD/LTD quality metrics
+
+    [:octicons-arrow-right-24: Reference](topic-score.md)
+
+-   :material-tune-variant: **topic optimize-thresholds**
+
+    ---
+
+    Search for better GTD/LTD cutoff values on an existing graph
+
+    [:octicons-arrow-right-24: Reference](topic-optimize-thresholds.md)
+
 </div>
 
 ## Workflow
@@ -35,14 +51,17 @@ deepfabric generate config.yaml --topic-only
 # 2. Inspect the result
 deepfabric topic inspect topics.json --level 1 --expand
 
-# 3. Prune if needed (preview first)
+# 3. Score graph quality
+deepfabric topic score topics.json
+
+# 4. Prune if needed (preview first)
 deepfabric topic prune topics.json --level 2 --dry-run
 deepfabric topic prune topics.json --level 2 -o refined_topics.json
 
-# 4. Verify the pruned structure
+# 5. Verify the pruned structure
 deepfabric topic inspect refined_topics.json --all
 
-# 5. Generate dataset from refined topics
+# 6. Generate dataset from refined topics
 deepfabric generate config.yaml --topics-load refined_topics.json
 ```
 

mkdocs.yml

@@ -44,6 +44,8 @@ nav:
       - cli/topic.md
       - topic inspect: cli/topic-inspect.md
       - topic prune: cli/topic-prune.md
+      - topic score: cli/topic-score.md
+      - topic optimize-thresholds: cli/topic-optimize-thresholds.md
     - upload-hf: cli/upload-hf.md
     - upload-kaggle: cli/upload-kaggle.md
     - upload: cli/upload.md
@@ -131,4 +133,3 @@ extra:
       link: https://github.com/always-further/deepfabric
     - icon: fontawesome/brands/python
       link: https://pypi.org/project/deepfabric/
-

pyproject.toml

@@ -27,6 +27,7 @@ dependencies = [
     "ollama>=0.6.1",
     "componentize-py>=0.19.3",
     "spin-sdk>=3.4.1",
+    "sentence-transformers>=5.2.3",
 ]
 
 [project.optional-dependencies]

tests/unit/test_topic_inspector.py

@@ -74,6 +74,28 @@ def graph_json_file():
     Path(path).unlink(missing_ok=True)
 
 
+@pytest.fixture
+def score_report_file():
+    """Create a temporary topic-score report JSON file."""
+    content = {
+        "summary": {
+            "thresholds": {"depth1_gtd": 0.25, "gtd_neg": 0.0, "ltd": 0.25},
+        },
+        "flagged_nodes": [
+            {
+                "node_id": "1",
+                "reasons": ["DEPTH1_LOW_GTD"],
+            }
+        ],
+        "removed_node_ids": ["1", "2"],
+    }
+    with tempfile.NamedTemporaryFile(mode="w", suffix=".json", delete=False) as f:
+        json.dump(content, f)
+        path = f.name
+    yield path
+    Path(path).unlink(missing_ok=True)
+
+
 @pytest.fixture
 def empty_file():
     """Create an empty file."""
@@ -83,6 +105,50 @@ def empty_file():
     Path(path).unlink(missing_ok=True)
 
 
+@pytest.fixture
+def graph_overlay_json_file():
+    """Create a graph file with one unaffected branch for flagged-only tests."""
+    content = {
+        "nodes": {
+            "0": {
+                "id": 0,
+                "topic": "Root",
+                "children": [1, 2, 3],
+                "parents": [],
+                "metadata": {"uuid": "overlay-uuid-0"},
+            },
+            "1": {
+                "id": 1,
+                "topic": "Flagged Node",
+                "children": [],
+                "parents": [0],
+                "metadata": {"uuid": "overlay-uuid-1"},
+            },
+            "2": {
+                "id": 2,
+                "topic": "Pruned Node",
+                "children": [],
+                "parents": [0],
+                "metadata": {"uuid": "overlay-uuid-2"},
+            },
+            "3": {
+                "id": 3,
+                "topic": "Safe Node",
+                "children": [],
+                "parents": [0],
+                "metadata": {"uuid": "overlay-uuid-3"},
+            },
+        },
+        "root_id": 0,
+        "metadata": {"provider": "openai", "model": "gpt-4", "temperature": 0.7},
+    }
+    with tempfile.NamedTemporaryFile(mode="w", suffix=".json", delete=False) as f:
+        json.dump(content, f)
+        path = f.name
+    yield path
+    Path(path).unlink(missing_ok=True)
+
+
 class TestDetectFormat:
     """Tests for format detection."""
 
@@ -259,6 +325,49 @@ def test_inspect_level_with_expand_depth(self, cli_runner, tree_jsonl_file):
         assert "Subtree from Level 1" in result.output
         assert "1 sublevel(s)" in result.output
 
+    def test_inspect_graph_with_prune_overlay(self, cli_runner, graph_json_file, score_report_file):
+        """Graph inspect can render flagged/pruned overlay from score report."""
+        result = cli_runner.invoke(
+            cli,
+            [
+                "topic",
+                "inspect",
+                graph_json_file,
+                "--all",
+                "--score-report",
+                score_report_file,
+                "--show-pruned",
+                "all",
+            ],
+        )
+        assert result.exit_code == 0
+        assert "Flagged Nodes" in result.output
+        assert "Would Prune" in result.output
+        assert "DEPTH1_LOW_GTD" in result.output
+        assert "(pruned)" in result.output
+
+    def test_inspect_graph_with_prune_overlay_flagged_only(
+        self, cli_runner, graph_overlay_json_file, score_report_file
+    ):
+        """flagged-only overlay should hide unaffected branches."""
+        result = cli_runner.invoke(
+            cli,
+            [
+                "topic",
+                "inspect",
+                graph_overlay_json_file,
+                "--all",
+                "--score-report",
+                score_report_file,
+                "--show-pruned",
+                "flagged-only",
+            ],
+        )
+        assert result.exit_code == 0
+        assert "Flagged Node" in result.output
+        assert "Pruned Node" in result.output
+        assert "Safe Node" not in result.output
+
     def test_inspect_with_uuid_flag(self, cli_runner, tree_jsonl_file):
         """--uuid flag shows UUIDs for leaf nodes."""
         result = cli_runner.invoke(cli, ["topic", "inspect", tree_jsonl_file, "--all", "--uuid"])