Repository navigation
Iterate a vertex's neighbours instead of scanning for them - #8
Merged
Merged
Conversation
N_adj built a vertex's neighbourhood by walking the whole vertex set and testing
G[x,y] one entry at a time, which is Θ(n) whatever the matrix underneath is, and
on a SparseMatrixCSC adds a binary search per test. So a sparse input was 1.8x
slower than a dense one rather than faster.
Introduce a `neighbors(G, x)` that reads column x, with a generic method that
scans the column -- O(n), which is that column's whole size, so exactly what a
dense matrix deserves -- and a SparseMatrixCSC method that walks the stored
entries in O(deg x). A new representation is one more method. Reading columns
rather than rows also stops the dense path striding across a column major array,
which is most of why it too got faster.
N_non is gone. Its only caller wanted X split into the neighbours and
non-neighbours of x, which is a flag array split costing O(|X| + deg x) for
either representation; materializing the non-neighbours is Θ(n) by construction
and can never suit a sparse graph.
symgraph_factorizing_permutation on a graph with m ≈ 3n:
n Matrix, before Sparse, before Sparse, after
800 0.0058s 0.0106s 0.0033s
3200 0.1347s 0.1958s 0.0470s 4.2x
Sparse input is now faster than dense input on a sparse graph, which is the
point. Dense input improved too, 0.4200s to 0.3298s at n = 3200, and a dense
graph handed over as a sparse matrix went from 1.9142s to 0.2626s.
This does not make the sparse path linear, and it is worth recording why not.
refine! visits every part of the partition on every call, so it costs Θ(n) per
pivot where the operation is defined to cost O(|S|). Counting elements visited
against elements that needed visiting, on m ≈ 3n:
n refine! calls Σ|S| needed Σ|X| scanned waste
400 489 2,923 195,600 66.9x
800 985 5,868 788,000 134.3x
1600 1,969 11,818 3,150,400 266.6x
3200 3,920 23,615 12,544,000 531.2x
The waste doubles as n doubles. Locating the parts that S meets through S's own
elements, rather than by scanning the partition, is the next step and is worth
more than anything else left.
Output is unchanged: 4000 graphs and digraphs, every factorizing permutation and
every tree identical to master, and identical between dense and sparse input.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013w7XdHqa7ynpy1nN1j19Qp
This was referenced Aug 24, 2026
StefanKarpinski
added a commit
that referenced
this pull request
Aug 24, 2026
refine! looked for the parts a pivot set S splits by walking the whole partition and testing every part against S. That costs Θ(n) per pivot where Habib & Paul define the operation to cost O(|S|), and it was what kept the sparse path quadratic after #8 made neighbourhoods cheap to enumerate. Elements visited against elements that needed visiting, on a graph with m ≈ 3n: n Σ|S| needed visited before visited now 400 2,914 195,600 4,194 800 5,790 788,000 8,350 1600 11,733 3,150,400 16,885 3200 23,449 12,544,000 33,817 531x, and 1.44x, doubling constant The partition is now an ordered partition whose parts are consecutive ranges of one array of vertices. Splitting a part rearranges only that part's range, so no other part moves and no part's position shifts. That buys three things at once: two parts compare by position in O(1), which is what the between test needs; the part holding a vertex is one lookup, so center in X and x in X stop being Θ(|X|) scans; and a split needs no insert! into a list of parts. refine! then buckets S by part and touches nothing else. The pivot and module lists hold part numbers rather than copies of part contents, so the findfirst(isequal(X), ...) searches are gone; entries are removed by clearing a position index and skipping the superseded entry on the way out. New part numbers go to the smaller side of each split, so relabelling costs O(min(k, |X|-k)). symgraph_factorizing_permutation on a graph with m ≈ 3n, as a SparseMatrixCSC, and cost per word of that representation: n before #8 after #8 now ns/word now 800 0.0106s 0.0033s 0.0006s 100.9 3200 0.1958s 0.0470s 0.0023s 104.0 12800 - - 0.0110s 122.3 Flat from n = 200 to n = 12800, a 64x range: linear in the size of a sparse input, which it has not been before. Dense input is unchanged, still flat at about 30ns per word of an n² matrix. As agreed on #3 this returns a different factorizing permutation than before. Costing O(|S|) means never looking at the non-members of a part, so the half that splits off is in the pivot set's order rather than the part's, which changes which vertex leads a part and so which pivot gets chosen. Every permutation is still a factorizing permutation and the trees are unchanged: * strong module trees identical to master over 3000 random graphs and digraphs, and dense and sparse input agree on every one; * 5,670,000 random starting orders over uniformly random graphs, twin-planted graphs and graphs with several planted modules at n = 8 … 14, every permutation modular; * every graph on n ≤ 6 against every starting order, 23,594,424 permutations, every one modular. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Step 1 of the plan in #3: abstract over the input representation so each one is
served in time proportional to the part of itself that gets touched.
The change
N_adjbuilt a vertex's neighbourhood by walking the whole vertex set andtesting
G[x,y]one entry at a time. That is Θ(n) whatever is underneath, andon a
SparseMatrixCSCit adds a binary search per test — which is why handingthe current code a sparse matrix made it 1.8× slower rather than faster.
The generic method scans a dense column in O(n), which is that column's entire
size, so it is exactly what a dense matrix deserves; the sparse method is
O(deg x). Supporting another representation is one more method, and nothing has
to commit to a representation. Reading columns rather than rows also stops the
dense path striding across a column-major array, which is most of why dense
input got faster too.
N_nonis gone. Its only caller wantedXsplit into the neighbours andnon-neighbours of
x, which is a flag-array split costing O(|X| + deg x) foreither representation. Materializing the non-neighbours is Θ(n) by construction
and can never suit a sparse graph.
Effect
symgraph_factorizing_permutation, graph with m ≈ 3n:MatrixbeforeSparsebeforeSparseafterSparse input is now faster than dense input on a sparse graph, which was the
whole point. Dense input improved as well — 0.4200s → 0.3298s at n = 3200 — and
a dense graph handed over as a sparse matrix went 1.9142s → 0.2626s.
What this does not do
It does not make the sparse path linear, and the reason is worth recording.
refine!visits every part of the partition on every call, so it costs Θ(n) perpivot where the operation is defined to cost O(|S|). Counting elements visited
against elements that needed visiting, on m ≈ 3n:
The waste doubles as n doubles. Reaching the parts that
Smeets throughS'sown elements, instead of by scanning the partition, is now the single biggest
thing left — bigger than the cutter loops I had put next in the plan, so the
ordering there should change.
Verification
Output is unchanged. Over 4000 random graphs and digraphs, every factorizing
permutation and every tree is identical to
master, and identical betweendense and sparse input — a representation must never change the answer, and it
does not.
🤖 Generated with Claude Code
https://claude.ai/code/session_013w7XdHqa7ynpy1nN1j19Qp