Repository navigation
Reach the parts a pivot set meets instead of walking the partition - #9
Merged
Merged
Conversation
refine! looked for the parts a pivot set S splits by walking the whole partition and testing every part against S. That costs Θ(n) per pivot where Habib & Paul define the operation to cost O(|S|), and it was what kept the sparse path quadratic after #8 made neighbourhoods cheap to enumerate. Elements visited against elements that needed visiting, on a graph with m ≈ 3n: n Σ|S| needed visited before visited now 400 2,914 195,600 4,194 800 5,790 788,000 8,350 1600 11,733 3,150,400 16,885 3200 23,449 12,544,000 33,817 531x, and 1.44x, doubling constant The partition is now an ordered partition whose parts are consecutive ranges of one array of vertices. Splitting a part rearranges only that part's range, so no other part moves and no part's position shifts. That buys three things at once: two parts compare by position in O(1), which is what the between test needs; the part holding a vertex is one lookup, so center in X and x in X stop being Θ(|X|) scans; and a split needs no insert! into a list of parts. refine! then buckets S by part and touches nothing else. The pivot and module lists hold part numbers rather than copies of part contents, so the findfirst(isequal(X), ...) searches are gone; entries are removed by clearing a position index and skipping the superseded entry on the way out. New part numbers go to the smaller side of each split, so relabelling costs O(min(k, |X|-k)). symgraph_factorizing_permutation on a graph with m ≈ 3n, as a SparseMatrixCSC, and cost per word of that representation: n before #8 after #8 now ns/word now 800 0.0106s 0.0033s 0.0006s 100.9 3200 0.1958s 0.0470s 0.0023s 104.0 12800 - - 0.0110s 122.3 Flat from n = 200 to n = 12800, a 64x range: linear in the size of a sparse input, which it has not been before. Dense input is unchanged, still flat at about 30ns per word of an n² matrix. As agreed on #3 this returns a different factorizing permutation than before. Costing O(|S|) means never looking at the non-members of a part, so the half that splits off is in the pivot set's order rather than the part's, which changes which vertex leads a part and so which pivot gets chosen. Every permutation is still a factorizing permutation and the trees are unchanged: * strong module trees identical to master over 3000 random graphs and digraphs, and dense and sparse input agree on every one; * 5,670,000 random starting orders over uniformly random graphs, twin-planted graphs and graphs with several planted modules at n = 8 … 14, every permutation modular; * every graph on n ≤ 6 against every starting order, 23,594,424 permutations, every one modular. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013w7XdHqa7ynpy1nN1j19Qp
This was referenced Aug 24, 2026
StefanKarpinski
added a commit
that referenced
this pull request
Aug 25, 2026
After #9 made the factorizing permutation linear in a sparse input, building the tree from it was 99% of what a sparse symmetric graph cost, and the cutter search was 99% of that. It scans the permutation for the first and last vertex that is related differently to the two ends of a consecutive pair, which looks at every place before the cutter whether or not anything is there: Θ(n) per pair and Θ(n²) over the permutation, on an input of size Θ(n + m). Those cutters are the vertices where the two ends' relations differ, so listing each vertex's relations in the order the permutation puts them and merging the two lists finds the first and last one without looking anywhere else. Sorting all the lists by place is one counting sort, and the merges are Θ(n + m). That is a method per representation, not a replacement, and the difference is large in both directions. The merge has to build the lists first, and that build costs the size of the representation -- Θ(n²) on a dense matrix, against a scan that on a graph with edges everywhere usually stops at the first place it looks. Using the merge for everything is 1170x slower on a dense random graph at n = 3200: 0.0011s to 1.2903s. So the scan stays for matrices that store every pair, and the merge is for matrices that store only the relations there are. Each is linear in the representation it serves. Building the strong module tree of a graph with m ≈ 3n, given as a SparseMatrixCSC, with cost per word of that representation: n before after ns/word before ns/word after 400 0.0026s 0.0006s 954.7 199.5 1600 0.0333s 0.0022s 2981.9 195.1 6400 0.5040s 0.0098s 11255.6 218.3 25600 8.6379s 0.0425s 48204.9 237.1 ~n^2.0 ~n^1.0 203x at n = 25600, and flat per word over a 64x range: linear in a sparse input, which tree construction has never been. Dense input is untouched, 0.0011s at n = 3200 before and after. The symmetric path is now linear in its input end to end for both representations. Verified: * the cutters found are identical to the ones the scan finds -- op, cl, lc and uc all equal -- over 8000 (graph, representation) pairs spanning symmetric graphs, digraphs and 0/1/2 labelled two-structures; * dense and sparse now take different methods, so they must agree, and they do over 20,000 random graphs and exhaustively over every symmetric graph on n ≤ 6, which is 33,864 of them; * trees identical to master over 3000 random graphs and digraphs; * 900,000 random digraph permutations and 5,670,000 symmetric ones, all modular. Exhaustively over 4 vertex digraphs, dense and sparse disagree on 2 of 4096. Those two are not from this change -- master disagrees on exactly the same two, and the cause is that issymmetric returns true for those matrices when stored sparse and false when stored dense, so graph_factorizing_permutation takes the symmetric branch for a digraph. Reported separately. test/factorizing.jl gains a check that the two methods agree, verified to catch a cutter that ignores one of the two directions. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013w7XdHqa7ynpy1nN1j19Qp
StefanKarpinski
added a commit
that referenced
this pull request
Aug 25, 2026
After #9 made the factorizing permutation linear in a sparse input, building the tree from it was 99% of what a sparse symmetric graph cost, and the cutter search was 99% of that. It scans the permutation for the first and last vertex that is related differently to the two ends of a consecutive pair, which looks at every place before the cutter whether or not anything is there: Θ(n) per pair and Θ(n²) over the permutation, on an input of size Θ(n + m). Those cutters are the vertices where the two ends' relations differ, so listing each vertex's relations in the order the permutation puts them and merging the two lists finds the first and last one without looking anywhere else. Sorting all the lists by place is one counting sort, and the merges are Θ(n + m). That is a method per representation, not a replacement, and the difference is large in both directions. The merge has to build the lists first, and that build costs the size of the representation -- Θ(n²) on a dense matrix, against a scan that on a graph with edges everywhere usually stops at the first place it looks. Using the merge for everything is 1170x slower on a dense random graph at n = 3200: 0.0011s to 1.2903s. So the scan stays for matrices that store every pair, and the merge is for matrices that store only the relations there are. Each is linear in the representation it serves. Building the strong module tree of a graph with m ≈ 3n, given as a SparseMatrixCSC, with cost per word of that representation: n before after ns/word before ns/word after 400 0.0026s 0.0006s 954.7 199.5 1600 0.0333s 0.0022s 2981.9 195.1 6400 0.5040s 0.0098s 11255.6 218.3 25600 8.6379s 0.0425s 48204.9 237.1 ~n^2.0 ~n^1.0 203x at n = 25600, and flat per word over a 64x range: linear in a sparse input, which tree construction has never been. Dense input is untouched, 0.0011s at n = 3200 before and after. The symmetric path is now linear in its input end to end for both representations. Verified: * the cutters found are identical to the ones the scan finds -- op, cl, lc and uc all equal -- over 8000 (graph, representation) pairs spanning symmetric graphs, digraphs and 0/1/2 labelled two-structures; * dense and sparse now take different methods, so they must agree, and they do over 20,000 random graphs and exhaustively over every symmetric graph on n ≤ 6, which is 33,864 of them; * trees identical to master over 3000 random graphs and digraphs; * 900,000 random digraph permutations and 5,670,000 symmetric ones, all modular. Exhaustively over 4 vertex digraphs, dense and sparse disagree on 2 of 4096. Those two are not from this change -- master disagrees on exactly the same two, and the cause is that issymmetric returns true for those matrices when stored sparse and false when stored dense, so graph_factorizing_permutation takes the symmetric branch for a digraph. Reported separately. test/factorizing.jl gains a check that the two methods agree, verified to catch a cutter that ignores one of the two directions. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Step 2 of the plan in #3, and the one the measurement in #8 said was worth more
than everything else left.
The problem
refine!found the parts a pivot setSsplits by walking the whole partitionand testing every part against
S. That is Θ(n) per pivot, where Habib & Pauldefine the operation to cost O(|S|) — and it is what kept the sparse path
quadratic even after #8 made neighbourhoods cheap to enumerate. Elements visited
against elements that actually needed visiting, on a graph with m ≈ 3n:
The change
The partition becomes an ordered partition whose parts are consecutive ranges of
one array of vertices. Splitting a part rearranges only that part's range, so no
other part moves and no part's position shifts. Three things fall out at once:
betweentest needs;
center in Xandx in Xstopbeing Θ(|X|) scans;
insert!into a list of parts.refine!then bucketsSby part and touches nothing else.The pivot and module lists (Habib & Paul's L and K) hold part numbers rather
than copies of part contents, so the
findfirst(isequal(X), ...)searches aregone; an entry is removed by clearing a position index and skipping the
superseded entry on the way out, which is O(1). New part numbers go to the
smaller side of each split, so relabelling costs O(min(k, |X|−k)).
Effect
symgraph_factorizing_permutationon a graph with m ≈ 3n, given as aSparseMatrixCSC, with cost per word of that representation:Flat from n = 200 to n = 12800, a 64× range — linear in the size of a sparse
input, which this has never been. Dense input is unchanged: still flat at about
30 ns per word of an n² matrix, slope 1.0.
The permutation changes, as agreed on #3
Costing O(|S|) means never looking at the non-members of a part, so the half that
splits off comes out in the pivot set's order rather than the part's. That
changes which vertex leads a part, hence which pivot is chosen, hence which
factorizing permutation is returned. Every permutation returned is still a
factorizing permutation, and the trees are unchanged:
masterover 3000 random graphs anddigraphs, and dense and sparse input agree on every one;
graphs, and graphs with several planted modules at n = 8 … 14 — every
permutation modular;
every one modular.
Where that leaves #3
The symmetric path is now linear in its input for both representations, up to
the O(n + m log n) the algorithm is designed for. Remaining, in order:
intersect_permutationon the digraph path, the cutter loops, node type bycounting cross edges, and modules by tree node rather than materialised leaf
sets.
🤖 Generated with Claude Code
https://claude.ai/code/session_013w7XdHqa7ynpy1nN1j19Qp