Repository navigation
Remove the avoidable quadratic work in tree construction and partition refinement - #7
Merged
Merged
Conversation
Two steps of building the tree from a factorizing permutation did work
proportional to the size of a subtree for each node of it, which is quadratic
on a deep tree and was avoidable in both cases.
classify_nodes called first_leaf on both operands of every comparison it makes,
and first_leaf walks to the bottom of a subtree, so a node with k children cost
O(k² · depth) rather than O(k²). It now classifies the children first and takes
each one's first leaf from the recursive call, which already knows it. Removing
the walk also removes a dynamic dispatch per comparison, since the leaves now
live in a Vector{T} rather than being pulled out of a Vector{Union{...}}.
The pass that drops non-module nodes asked, for each node, for the smallest
lower cutter and largest upper cutter over the node's whole span, and answered
by rescanning lc and uc. Since the nodes are properly nested, each open paren
can instead accumulate the extremes of everything seen since it opened and fold
them into its parent as it closes, which is O(1) per node.
On the phase that changed, at n = 4000: a graph whose modules nest all the way
down goes from 0.1699s to 0.0023s, and one complete node with 4000 children from
1.6342s to 0.0728s. End to end for the whole construction that is 3.0x and 10x,
and a uniformly random graph now holds flat at 0.34us per vertex.
Output is unchanged: over 1000 graphs across four shapes, both with an explicit
permutation and through the full StrongModuleTree(G) path, every tree prints
identically to before, with identical node types, kinds, edges, leaves and
strong module sets.
This does not make the package linear time. What is left in tree construction
is reading the graph: finding each pair's first and last cutter scans a row
until it finds one, and classifying a complete or linear node compares every
pair of children. Both are quadratic because the graph arrives as a dense
matrix, and getting past that needs the adjacency list representation the
linear time algorithms assume, not a local change. Separately,
symgraph_factorizing_permutation dominates the runtime of everything here by
orders of magnitude; see the issue.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013w7XdHqa7ynpy1nN1j19Qp
refine! computed X ∩ S and then X \ (X ∩ S) for every part X of the partition.
Each of those walks S as well as X, so refining by one pivot cost O(|P|·|S|)
where the operation is supposed to be O(|S|) -- Habib & Paul 2009, lemma 10.
That is what made the whole thing cubic. Marking the pivot set in a flag array
once per call and splitting each part against the flags in a single pass brings
it to O(Σ|X|), and the neighbourhood minus the pivot part is now built in one
scan rather than materialized and then hashed against.
symgraph_factorizing_permutation on a random graph, with the measured exponent:
n before after
100 0.0084s 0.0003s
200 0.0682s 0.0013s
400 0.4564s 0.0054s
800 3.4150s 0.0210s 163x
~n^2.9 ~n^2.05
and the whole decomposition of a random graph at n = 800 goes from 3.4628s to
0.0217s, 160x, with the exponent down from 2.74 to 1.96.
The remaining n² is the dense matrix: each pivot scans a row for its
neighbourhood, finding a pair's cutters scans a row, and classifying a complete
node compares every pair of children. None of that gets better without the
adjacency list representation the linear time algorithms assume.
Output is unchanged over 4000 random symmetric graphs, twin-planted graphs and
digraphs -- every factorizing permutation and every tree identical. The sweep
from #5 still passes: 5.67M random starting orders, plus every graph on n ≤ 6
against every starting order, all modular.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013w7XdHqa7ynpy1nN1j19Qp
StefanKarpinski
force-pushed
the
sk/linear-tree-construction
branch
from
August 20, 2026 12:21
79fabc4 to
311e065
Compare
Welcome to Codecov 🎉Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests. Thanks for integrating Codecov - We've got you covered ☂️ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses the second half of #3 — "parts of strong module tree construction
from a factorizing permutation that may be super-linear unnecessarily" — and,
along the way, the step that turned out to dominate everything else.
Nothing here changes what the package computes. Every factorizing permutation
and every tree is identical to
masteracross 4000 random symmetric graphs,twin-planted graphs and digraphs, with identical node types, kinds, edges,
leaves and strong module sets.
Building the tree
Two steps did work proportional to the size of a subtree for each node of it.
classify_nodescalledfirst_leafon both operands of every pairwisecomparison it makes, and
first_leafwalks to the bottom of a subtree, so anode with
kchildren costO(k² · depth)instead ofO(k²). It nowclassifies the children first and takes each one's first leaf from the
recursive call, which already knows it. That also drops a dynamic dispatch per
comparison, since the leaves now live in a
Vector{T}rather than being pulledout of a
Vector{Union{...}}.The pass that drops non-module nodes asked, for each node, for the smallest
lower cutter and largest upper cutter over the node's whole span, and answered
by rescanning
lcanduc. The nodes are properly nested, so each open parencan instead accumulate the extremes of everything seen since it opened and fold
them into its parent as it closes —
O(1)per node.On the phase that changed, at n = 4000:
The step that actually dominated
With that done, profiling the whole decomposition put essentially all of the
time in
symgraph_factorizing_permutation, at roughlyn^2.9. The hot line wasin
refine!. Both of those walkSas well asX, and they run for every partof the partition, so refining by one pivot costs
O(|P|·|S|)— where the wholepoint of the operation is that it should cost
O(|S|)(Habib & Paul 2009,lemma 10). Marking the pivot set in a flag array once per call and splitting
each part against the flags in one pass makes it
O(Σ|X|); the neighbourhoodminus the pivot part is likewise built in a single scan instead of being
materialized and then hashed against.
End to end, decomposing a random graph at n = 800 goes from 3.4628s to 0.0217s,
160×, with the measured exponent down from 2.74 to 1.96.
What is still not linear
The package is faster and better behaved, but it is not linear time, and the
rest is not a local change:
pair's cutters scans a row, and classifying a complete or linear node compares
every pair of children. All three are quadratic because the graph arrives as
an
AbstractMatrix. The linear-time algorithms assume adjacency lists, sogetting past this is an API change.
intersect_permutation, on the digraph path, is now the bottleneck there:0.2251s of the 0.2804s a digraph at n = 800 takes, against 0.0004s for
overlap components. It looks modules up with
X in Ms— a linear scancomparing whole vectors — calls
parent_node, which recomputesleavesas itdescends, and builds the nesting with
child in parentandfilter!overvectors. None of that needs to be superlinear.
So #3 should stay open after this; both things it names are now done, but its
headline claim is not yet true.
Verification
masteron 4000 random graphs and digraphs, checked asprinted trees and again as node types, kinds, edges, leaves and strong module
sets.
starting orders across three graph families at n = 8 … 14, plus every graph on
n ≤ 6 against every starting order — 23,594,424 permutations — all modular.
🤖 Generated with Claude Code
https://claude.ai/code/session_013w7XdHqa7ynpy1nN1j19Qp