Skip to content

Remove the avoidable quadratic work in tree construction and partition refinement - #7

Merged
StefanKarpinski merged 2 commits into
masterfrom
sk/linear-tree-construction
Aug 20, 2026
Merged

StefanKarpinski merged 2 commits into
masterfrom
sk/linear-tree-construction

Conversation

@StefanKarpinski

Copy link
Copy Markdown
Owner

Addresses the second half of #3 — "parts of strong module tree construction
from a factorizing permutation that may be super-linear unnecessarily" — and,
along the way, the step that turned out to dominate everything else.

Nothing here changes what the package computes. Every factorizing permutation
and every tree is identical to master across 4000 random symmetric graphs,
twin-planted graphs and digraphs, with identical node types, kinds, edges,
leaves and strong module sets.

Building the tree

Two steps did work proportional to the size of a subtree for each node of it.

classify_nodes called first_leaf on both operands of every pairwise
comparison it makes, and first_leaf walks to the bottom of a subtree, so a
node with k children cost O(k² · depth) instead of O(k²). It now
classifies the children first and takes each one's first leaf from the
recursive call, which already knows it. That also drops a dynamic dispatch per
comparison, since the leaves now live in a Vector{T} rather than being pulled
out of a Vector{Union{...}}.

The pass that drops non-module nodes asked, for each node, for the smallest
lower cutter and largest upper cutter over the node's whole span, and answered
by rescanning lc and uc. The nodes are properly nested, so each open paren
can instead accumulate the extremes of everything seen since it opened and fold
them into its parent as it closes — O(1) per node.

On the phase that changed, at n = 4000:

graph before after
modules nested all the way down 0.1699s 0.0023s 74×
one complete node, 4000 children 1.6342s 0.0728s 22×

The step that actually dominated

With that done, profiling the whole decomposition put essentially all of the
time in symgraph_factorizing_permutation, at roughly n^2.9. The hot line was

Xₐ = X ∩ S
...
Xₙ = X \ Xₐ

in refine!. Both of those walk S as well as X, and they run for every part
of the partition, so refining by one pivot costs O(|P|·|S|) — where the whole
point of the operation is that it should cost O(|S|) (Habib & Paul 2009,
lemma 10). Marking the pivot set in a flag array once per call and splitting
each part against the flags in one pass makes it O(Σ|X|); the neighbourhood
minus the pivot part is likewise built in a single scan instead of being
materialized and then hashed against.

n before after
100 0.0084s 0.0003s
200 0.0682s 0.0013s
400 0.4564s 0.0054s
800 3.4150s 0.0210s 163×
~n^2.9 ~n^2.05

End to end, decomposing a random graph at n = 800 goes from 3.4628s to 0.0217s,
160×, with the measured exponent down from 2.74 to 1.96.

What is still not linear

The package is faster and better behaved, but it is not linear time, and the
rest is not a local change:

  • The dense matrix. Each pivot scans a row for its neighbourhood, finding a
    pair's cutters scans a row, and classifying a complete or linear node compares
    every pair of children. All three are quadratic because the graph arrives as
    an AbstractMatrix. The linear-time algorithms assume adjacency lists, so
    getting past this is an API change.
  • intersect_permutation, on the digraph path, is now the bottleneck there:
    0.2251s of the 0.2804s a digraph at n = 800 takes, against 0.0004s for
    overlap components. It looks modules up with X in Ms — a linear scan
    comparing whole vectors — calls parent_node, which recomputes leaves as it
    descends, and builds the nesting with child in parent and filter! over
    vectors. None of that needs to be superlinear.

So #3 should stay open after this; both things it names are now done, but its
headline claim is not yet true.

Verification

  • Output identical to master on 4000 random graphs and digraphs, checked as
    printed trees and again as node types, kinds, edges, leaves and strong module
    sets.
  • The correctness sweep from symgraph_factorizing_permutation can return a non-modular permutation #5 still passes on this branch: 5,670,000 random
    starting orders across three graph families at n = 8 … 14, plus every graph on
    n ≤ 6 against every starting order — 23,594,424 permutations — all modular.

🤖 Generated with Claude Code

https://claude.ai/code/session_013w7XdHqa7ynpy1nN1j19Qp

StefanKarpinski and others added 2 commits August 20, 2026 12:20
Two steps of building the tree from a factorizing permutation did work
proportional to the size of a subtree for each node of it, which is quadratic
on a deep tree and was avoidable in both cases.

classify_nodes called first_leaf on both operands of every comparison it makes,
and first_leaf walks to the bottom of a subtree, so a node with k children cost
O(k² · depth) rather than O(k²). It now classifies the children first and takes
each one's first leaf from the recursive call, which already knows it. Removing
the walk also removes a dynamic dispatch per comparison, since the leaves now
live in a Vector{T} rather than being pulled out of a Vector{Union{...}}.

The pass that drops non-module nodes asked, for each node, for the smallest
lower cutter and largest upper cutter over the node's whole span, and answered
by rescanning lc and uc. Since the nodes are properly nested, each open paren
can instead accumulate the extremes of everything seen since it opened and fold
them into its parent as it closes, which is O(1) per node.

On the phase that changed, at n = 4000: a graph whose modules nest all the way
down goes from 0.1699s to 0.0023s, and one complete node with 4000 children from
1.6342s to 0.0728s. End to end for the whole construction that is 3.0x and 10x,
and a uniformly random graph now holds flat at 0.34us per vertex.

Output is unchanged: over 1000 graphs across four shapes, both with an explicit
permutation and through the full StrongModuleTree(G) path, every tree prints
identically to before, with identical node types, kinds, edges, leaves and
strong module sets.

This does not make the package linear time. What is left in tree construction
is reading the graph: finding each pair's first and last cutter scans a row
until it finds one, and classifying a complete or linear node compares every
pair of children. Both are quadratic because the graph arrives as a dense
matrix, and getting past that needs the adjacency list representation the
linear time algorithms assume, not a local change. Separately,
symgraph_factorizing_permutation dominates the runtime of everything here by
orders of magnitude; see the issue.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013w7XdHqa7ynpy1nN1j19Qp
refine! computed X ∩ S and then X \ (X ∩ S) for every part X of the partition.
Each of those walks S as well as X, so refining by one pivot cost O(|P|·|S|)
where the operation is supposed to be O(|S|) -- Habib & Paul 2009, lemma 10.
That is what made the whole thing cubic. Marking the pivot set in a flag array
once per call and splitting each part against the flags in a single pass brings
it to O(Σ|X|), and the neighbourhood minus the pivot part is now built in one
scan rather than materialized and then hashed against.

symgraph_factorizing_permutation on a random graph, with the measured exponent:

        n     before      after
      100    0.0084s    0.0003s
      200    0.0682s    0.0013s
      400    0.4564s    0.0054s
      800    3.4150s    0.0210s     163x
             ~n^2.9     ~n^2.05

and the whole decomposition of a random graph at n = 800 goes from 3.4628s to
0.0217s, 160x, with the exponent down from 2.74 to 1.96.

The remaining n² is the dense matrix: each pivot scans a row for its
neighbourhood, finding a pair's cutters scans a row, and classifying a complete
node compares every pair of children. None of that gets better without the
adjacency list representation the linear time algorithms assume.

Output is unchanged over 4000 random symmetric graphs, twin-planted graphs and
digraphs -- every factorizing permutation and every tree identical. The sweep
from #5 still passes: 5.67M random starting orders, plus every graph on n ≤ 6
against every starting order, all modular.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013w7XdHqa7ynpy1nN1j19Qp
@StefanKarpinski
StefanKarpinski force-pushed the sk/linear-tree-construction branch from 79fabc4 to 311e065 Compare August 20, 2026 12:21
@codecov

codecov Bot commented Aug 20, 2026

Copy link
Copy Markdown

Welcome to Codecov 🎉

Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests.

Thanks for integrating Codecov - We've got you covered ☂️

@StefanKarpinski
StefanKarpinski merged commit d2181d0 into master Aug 20, 2026
17 checks passed
@StefanKarpinski
StefanKarpinski deleted the sk/linear-tree-construction branch August 20, 2026 18:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant