perf(neo4j): fix O(N²) edge inserts via labeled MATCH + UNWIND batching - #2258
Open
kingjin94 wants to merge 1 commit into
Open
perf(neo4j): fix O(N²) edge inserts via labeled MATCH + UNWIND batching#2258kingjin94 wants to merge 1 commit into
kingjin94 wants to merge 1 commit into
Conversation
Two compounding issues made push_to_neo4j catastrophically slow on large
graphs (5+ hours for ~40k nodes/edges):
1. Per-row round-trips: each node/edge was a separate session.run()
auto-commit transaction (~77k Bolt round-trips for a medium repo).
Fixed by grouping nodes/edges by label/rel-type and sending UNWIND
batches of 500 via execute_write().
2. Label-less edge MATCH (the critical one): the edge query was
`MATCH (a {id: row.src}), (b {id: row.tgt})` with no label, so Neo4j
ignored all uniqueness constraints and scanned the full node store for
every row — O(N²) regardless of indexes.
Fixed by building an id→label map from the node pass and grouping edges
by (src_label, tgt_label, rel), so the MATCH becomes
`MATCH (a:Code {id: row.src}), (b:Document {id: row.tgt})` which hits
the per-label b-tree index.
Benchmark (ephemeral Neo4j 5, no pre-existing data):
10k nodes / 20k edges:
baseline (per-row): 88s
UNWIND + constraint, no label: 71s (barely better)
UNWIND + constraint + label: 1.5s (50x faster)
Production result: 212k nodes + 398k edges pushed in ~24s.
kingjin94
marked this pull request as ready for review
July 28, 2026 09:26
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
push_to_neo4jwas catastrophically slow on large graphs (5+ hours for ~40k nodes/edges in production). Two compounding issues:Per-row round-trips: each node/edge was a separate
session.run()auto-commit transaction (~77k Bolt round-trips for a medium repo).Label-less edge MATCH (the critical one): the edge query was:
With no label, Neo4j ignores all uniqueness constraints and scans the full node store for every row — O(N²) regardless of indexes or UNWIND batching.
Fix
(src_label, tgt_label, rel), send as UNWIND batches of 500 viaexecute_write().id→labelmap during the node pass:idper label, which callers should create before pushing).Benchmark
Ephemeral Neo4j 5, no pre-existing data:
Production: 212k nodes + 398k edges pushed in ~24s.
Notes
CREATE CONSTRAINT label_id_unique IF NOT EXISTS FOR (n:Label) REQUIRE n.id IS UNIQUEfor each label before pushing to get the full benefit.