Skip to content

Reach the parts a pivot set meets instead of walking the partition - #9

Merged
StefanKarpinski merged 1 commit into
masterfrom
sk/targeted-refinement
Aug 24, 2026
Merged

StefanKarpinski merged 1 commit into
masterfrom
sk/targeted-refinement

Conversation

@StefanKarpinski

Copy link
Copy Markdown
Owner

Step 2 of the plan in #3, and the one the measurement in #8 said was worth more
than everything else left.

The problem

refine! found the parts a pivot set S splits by walking the whole partition
and testing every part against S. That is Θ(n) per pivot, where Habib & Paul
define the operation to cost O(|S|) — and it is what kept the sparse path
quadratic even after #8 made neighbourhoods cheap to enumerate. Elements visited
against elements that actually needed visiting, on a graph with m ≈ 3n:

n Σ|S| needed visited before visited now
400 2,914 195,600 4,194
800 5,790 788,000 8,350
1600 11,733 3,150,400 16,885
3200 23,449 12,544,000 33,817
531×, doubling 1.44×, constant

The change

The partition becomes an ordered partition whose parts are consecutive ranges of
one array of vertices. Splitting a part rearranges only that part's range, so no
other part moves and no part's position shifts. Three things fall out at once:

  • two parts compare by position in O(1), which is exactly what the between
    test needs;
  • the part holding a vertex is one lookup, so center in X and x in X stop
    being Θ(|X|) scans;
  • a split needs no insert! into a list of parts.

refine! then buckets S by part and touches nothing else.

The pivot and module lists (Habib & Paul's L and K) hold part numbers rather
than copies of part contents, so the findfirst(isequal(X), ...) searches are
gone; an entry is removed by clearing a position index and skipping the
superseded entry on the way out, which is O(1). New part numbers go to the
smaller side of each split, so relabelling costs O(min(k, |X|−k)).

Effect

symgraph_factorizing_permutation on a graph with m ≈ 3n, given as a
SparseMatrixCSC, with cost per word of that representation:

n before #8 after #8 now ns/word now
800 0.0106s 0.0033s 0.0006s 100.9
3200 0.1958s 0.0470s 0.0023s 104.0
12800 — — 0.0110s 122.3

Flat from n = 200 to n = 12800, a 64× range — linear in the size of a sparse
input
, which this has never been. Dense input is unchanged: still flat at about
30 ns per word of an n² matrix, slope 1.0.

The permutation changes, as agreed on #3

Costing O(|S|) means never looking at the non-members of a part, so the half that
splits off comes out in the pivot set's order rather than the part's. That
changes which vertex leads a part, hence which pivot is chosen, hence which
factorizing permutation is returned. Every permutation returned is still a
factorizing permutation, and the trees are unchanged:

  • strong module trees identical to master over 3000 random graphs and
    digraphs, and dense and sparse input agree on every one;
  • 5,670,000 random starting orders over uniformly random graphs, twin-planted
    graphs, and graphs with several planted modules at n = 8 … 14 — every
    permutation modular;
  • every graph on n ≤ 6 against every starting order, 23,594,424 permutations —
    every one modular.

Where that leaves #3

The symmetric path is now linear in its input for both representations, up to
the O(n + m log n) the algorithm is designed for. Remaining, in order:
intersect_permutation on the digraph path, the cutter loops, node type by
counting cross edges, and modules by tree node rather than materialised leaf
sets.

🤖 Generated with Claude Code

https://claude.ai/code/session_013w7XdHqa7ynpy1nN1j19Qp

refine! looked for the parts a pivot set S splits by walking the whole partition
and testing every part against S. That costs Θ(n) per pivot where Habib & Paul
define the operation to cost O(|S|), and it was what kept the sparse path
quadratic after #8 made neighbourhoods cheap to enumerate. Elements visited
against elements that needed visiting, on a graph with m ≈ 3n:

        n   Σ|S| needed   visited before   visited now
      400         2,914          195,600         4,194
      800         5,790          788,000         8,350
     1600        11,733        3,150,400        16,885
     3200        23,449       12,544,000        33,817
                                 531x, and       1.44x,
                                  doubling      constant

The partition is now an ordered partition whose parts are consecutive ranges of
one array of vertices. Splitting a part rearranges only that part's range, so no
other part moves and no part's position shifts. That buys three things at once:
two parts compare by position in O(1), which is what the between test needs; the
part holding a vertex is one lookup, so center in X and x in X stop being Θ(|X|)
scans; and a split needs no insert! into a list of parts. refine! then buckets S
by part and touches nothing else.

The pivot and module lists hold part numbers rather than copies of part
contents, so the findfirst(isequal(X), ...) searches are gone; entries are
removed by clearing a position index and skipping the superseded entry on the
way out. New part numbers go to the smaller side of each split, so relabelling
costs O(min(k, |X|-k)).

symgraph_factorizing_permutation on a graph with m ≈ 3n, as a SparseMatrixCSC,
and cost per word of that representation:

        n     before #8      after #8         now    ns/word now
      800       0.0106s       0.0033s     0.0006s          100.9
     3200       0.1958s       0.0470s     0.0023s          104.0
    12800             -             -     0.0110s          122.3

Flat from n = 200 to n = 12800, a 64x range: linear in the size of a sparse
input, which it has not been before. Dense input is unchanged, still flat at
about 30ns per word of an n² matrix.

As agreed on #3 this returns a different factorizing permutation than before.
Costing O(|S|) means never looking at the non-members of a part, so the half
that splits off is in the pivot set's order rather than the part's, which
changes which vertex leads a part and so which pivot gets chosen. Every
permutation is still a factorizing permutation and the trees are unchanged:

  * strong module trees identical to master over 3000 random graphs and
    digraphs, and dense and sparse input agree on every one;
  * 5,670,000 random starting orders over uniformly random graphs, twin-planted
    graphs and graphs with several planted modules at n = 8 … 14, every
    permutation modular;
  * every graph on n ≤ 6 against every starting order, 23,594,424 permutations,
    every one modular.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013w7XdHqa7ynpy1nN1j19Qp
@StefanKarpinski
StefanKarpinski merged commit 5debaea into master Aug 24, 2026
15 checks passed
@StefanKarpinski
StefanKarpinski deleted the sk/targeted-refinement branch August 24, 2026 20:17
StefanKarpinski added a commit that referenced this pull request Aug 25, 2026
After #9 made the factorizing permutation linear in a sparse input, building the
tree from it was 99% of what a sparse symmetric graph cost, and the cutter
search was 99% of that. It scans the permutation for the first and last vertex
that is related differently to the two ends of a consecutive pair, which looks
at every place before the cutter whether or not anything is there: Θ(n) per pair
and Θ(n²) over the permutation, on an input of size Θ(n + m).

Those cutters are the vertices where the two ends' relations differ, so listing
each vertex's relations in the order the permutation puts them and merging the
two lists finds the first and last one without looking anywhere else. Sorting
all the lists by place is one counting sort, and the merges are Θ(n + m).

That is a method per representation, not a replacement, and the difference is
large in both directions. The merge has to build the lists first, and that build
costs the size of the representation -- Θ(n²) on a dense matrix, against a scan
that on a graph with edges everywhere usually stops at the first place it looks.
Using the merge for everything is 1170x slower on a dense random graph at
n = 3200: 0.0011s to 1.2903s. So the scan stays for matrices that store every
pair, and the merge is for matrices that store only the relations there are.
Each is linear in the representation it serves.

Building the strong module tree of a graph with m ≈ 3n, given as a
SparseMatrixCSC, with cost per word of that representation:

        n      before       after   ns/word before   ns/word after
      400     0.0026s     0.0006s             954.7           199.5
     1600     0.0333s     0.0022s            2981.9           195.1
     6400     0.5040s     0.0098s           11255.6           218.3
    25600     8.6379s     0.0425s           48204.9           237.1
                                            ~n^2.0          ~n^1.0

203x at n = 25600, and flat per word over a 64x range: linear in a sparse input,
which tree construction has never been. Dense input is untouched, 0.0011s at
n = 3200 before and after.

The symmetric path is now linear in its input end to end for both
representations.

Verified:

  * the cutters found are identical to the ones the scan finds -- op, cl, lc and
    uc all equal -- over 8000 (graph, representation) pairs spanning symmetric
    graphs, digraphs and 0/1/2 labelled two-structures;
  * dense and sparse now take different methods, so they must agree, and they do
    over 20,000 random graphs and exhaustively over every symmetric graph on
    n ≤ 6, which is 33,864 of them;
  * trees identical to master over 3000 random graphs and digraphs;
  * 900,000 random digraph permutations and 5,670,000 symmetric ones, all
    modular.

Exhaustively over 4 vertex digraphs, dense and sparse disagree on 2 of 4096.
Those two are not from this change -- master disagrees on exactly the same two,
and the cause is that issymmetric returns true for those matrices when stored
sparse and false when stored dense, so graph_factorizing_permutation takes the
symmetric branch for a digraph. Reported separately.

test/factorizing.jl gains a check that the two methods agree, verified to catch
a cutter that ignores one of the two directions.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013w7XdHqa7ynpy1nN1j19Qp
StefanKarpinski added a commit that referenced this pull request Aug 25, 2026
After #9 made the factorizing permutation linear in a sparse input, building the
tree from it was 99% of what a sparse symmetric graph cost, and the cutter
search was 99% of that. It scans the permutation for the first and last vertex
that is related differently to the two ends of a consecutive pair, which looks
at every place before the cutter whether or not anything is there: Θ(n) per pair
and Θ(n²) over the permutation, on an input of size Θ(n + m).

Those cutters are the vertices where the two ends' relations differ, so listing
each vertex's relations in the order the permutation puts them and merging the
two lists finds the first and last one without looking anywhere else. Sorting
all the lists by place is one counting sort, and the merges are Θ(n + m).

That is a method per representation, not a replacement, and the difference is
large in both directions. The merge has to build the lists first, and that build
costs the size of the representation -- Θ(n²) on a dense matrix, against a scan
that on a graph with edges everywhere usually stops at the first place it looks.
Using the merge for everything is 1170x slower on a dense random graph at
n = 3200: 0.0011s to 1.2903s. So the scan stays for matrices that store every
pair, and the merge is for matrices that store only the relations there are.
Each is linear in the representation it serves.

Building the strong module tree of a graph with m ≈ 3n, given as a
SparseMatrixCSC, with cost per word of that representation:

        n      before       after   ns/word before   ns/word after
      400     0.0026s     0.0006s             954.7           199.5
     1600     0.0333s     0.0022s            2981.9           195.1
     6400     0.5040s     0.0098s           11255.6           218.3
    25600     8.6379s     0.0425s           48204.9           237.1
                                            ~n^2.0          ~n^1.0

203x at n = 25600, and flat per word over a 64x range: linear in a sparse input,
which tree construction has never been. Dense input is untouched, 0.0011s at
n = 3200 before and after.

The symmetric path is now linear in its input end to end for both
representations.

Verified:

  * the cutters found are identical to the ones the scan finds -- op, cl, lc and
    uc all equal -- over 8000 (graph, representation) pairs spanning symmetric
    graphs, digraphs and 0/1/2 labelled two-structures;
  * dense and sparse now take different methods, so they must agree, and they do
    over 20,000 random graphs and exhaustively over every symmetric graph on
    n ≤ 6, which is 33,864 of them;
  * trees identical to master over 3000 random graphs and digraphs;
  * 900,000 random digraph permutations and 5,670,000 symmetric ones, all
    modular.

Exhaustively over 4 vertex digraphs, dense and sparse disagree on 2 of 4096.
Those two are not from this change -- master disagrees on exactly the same two,
and the cause is that issymmetric returns true for those matrices when stored
sparse and false when stored dense, so graph_factorizing_permutation takes the
symmetric branch for a digraph. Reported separately.

test/factorizing.jl gains a check that the two methods agree, verified to catch
a cutter that ignores one of the two directions.


Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant