Skip to content

Iterate a vertex's neighbours instead of scanning for them - #8

Merged
StefanKarpinski merged 1 commit into
masterfrom
sk/neighbor-iteration
Aug 24, 2026
Merged

StefanKarpinski merged 1 commit into
masterfrom
sk/neighbor-iteration

Conversation

@StefanKarpinski

Copy link
Copy Markdown
Owner

Step 1 of the plan in #3: abstract over the input representation so each one is
served in time proportional to the part of itself that gets touched.

The change

N_adj built a vertex's neighbourhood by walking the whole vertex set and
testing G[x,y] one entry at a time. That is Θ(n) whatever is underneath, and
on a SparseMatrixCSC it adds a binary search per test — which is why handing
the current code a sparse matrix made it 1.8× slower rather than faster.

neighbors(G::AbstractMatrix, x) = (y for y in axes(G,1) if y != x && !iszero(G[y,x]))
neighbors(G::SparseMatrixCSC, x) = ...   # walks the stored entries of column x

The generic method scans a dense column in O(n), which is that column's entire
size, so it is exactly what a dense matrix deserves; the sparse method is
O(deg x). Supporting another representation is one more method, and nothing has
to commit to a representation. Reading columns rather than rows also stops the
dense path striding across a column-major array, which is most of why dense
input got faster too.

N_non is gone. Its only caller wanted X split into the neighbours and
non-neighbours of x, which is a flag-array split costing O(|X| + deg x) for
either representation. Materializing the non-neighbours is Θ(n) by construction
and can never suit a sparse graph.

Effect

symgraph_factorizing_permutation, graph with m ≈ 3n:

n Matrix before Sparse before Sparse after
800 0.0058s 0.0106s 0.0033s
3200 0.1347s 0.1958s 0.0470s 4.2×

Sparse input is now faster than dense input on a sparse graph, which was the
whole point. Dense input improved as well — 0.4200s → 0.3298s at n = 3200 — and
a dense graph handed over as a sparse matrix went 1.9142s → 0.2626s.

What this does not do

It does not make the sparse path linear, and the reason is worth recording.
refine! visits every part of the partition on every call, so it costs Θ(n) per
pivot where the operation is defined to cost O(|S|). Counting elements visited
against elements that needed visiting, on m ≈ 3n:

n refine! calls Σ|S| needed Σ|X| scanned waste
400 489 2,923 195,600 66.9×
800 985 5,868 788,000 134.3×
1600 1,969 11,818 3,150,400 266.6×
3200 3,920 23,615 12,544,000 531.2×

The waste doubles as n doubles. Reaching the parts that S meets through S's
own elements, instead of by scanning the partition, is now the single biggest
thing left — bigger than the cutter loops I had put next in the plan, so the
ordering there should change.

Verification

Output is unchanged. Over 4000 random graphs and digraphs, every factorizing
permutation and every tree is identical to master, and identical between
dense and sparse input — a representation must never change the answer, and it
does not.

🤖 Generated with Claude Code

https://claude.ai/code/session_013w7XdHqa7ynpy1nN1j19Qp

N_adj built a vertex's neighbourhood by walking the whole vertex set and testing
G[x,y] one entry at a time, which is Θ(n) whatever the matrix underneath is, and
on a SparseMatrixCSC adds a binary search per test. So a sparse input was 1.8x
slower than a dense one rather than faster.

Introduce a `neighbors(G, x)` that reads column x, with a generic method that
scans the column -- O(n), which is that column's whole size, so exactly what a
dense matrix deserves -- and a SparseMatrixCSC method that walks the stored
entries in O(deg x). A new representation is one more method. Reading columns
rather than rows also stops the dense path striding across a column major array,
which is most of why it too got faster.

N_non is gone. Its only caller wanted X split into the neighbours and
non-neighbours of x, which is a flag array split costing O(|X| + deg x) for
either representation; materializing the non-neighbours is Θ(n) by construction
and can never suit a sparse graph.

symgraph_factorizing_permutation on a graph with m ≈ 3n:

        n      Matrix, before      Sparse, before       Sparse, after
      800             0.0058s             0.0106s             0.0033s
     3200             0.1347s             0.1958s             0.0470s   4.2x

Sparse input is now faster than dense input on a sparse graph, which is the
point. Dense input improved too, 0.4200s to 0.3298s at n = 3200, and a dense
graph handed over as a sparse matrix went from 1.9142s to 0.2626s.

This does not make the sparse path linear, and it is worth recording why not.
refine! visits every part of the partition on every call, so it costs Θ(n) per
pivot where the operation is defined to cost O(|S|). Counting elements visited
against elements that needed visiting, on m ≈ 3n:

        n   refine! calls   Σ|S| needed   Σ|X| scanned    waste
      400             489         2,923        195,600    66.9x
      800             985         5,868        788,000   134.3x
     1600           1,969        11,818      3,150,400   266.6x
     3200           3,920        23,615     12,544,000   531.2x

The waste doubles as n doubles. Locating the parts that S meets through S's own
elements, rather than by scanning the partition, is the next step and is worth
more than anything else left.

Output is unchanged: 4000 graphs and digraphs, every factorizing permutation and
every tree identical to master, and identical between dense and sparse input.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013w7XdHqa7ynpy1nN1j19Qp
@StefanKarpinski
StefanKarpinski merged commit 0169836 into master Aug 24, 2026
15 checks passed
@StefanKarpinski
StefanKarpinski deleted the sk/neighbor-iteration branch August 24, 2026 18:53
StefanKarpinski added a commit that referenced this pull request Aug 24, 2026
refine! looked for the parts a pivot set S splits by walking the whole partition
and testing every part against S. That costs Θ(n) per pivot where Habib & Paul
define the operation to cost O(|S|), and it was what kept the sparse path
quadratic after #8 made neighbourhoods cheap to enumerate. Elements visited
against elements that needed visiting, on a graph with m ≈ 3n:

        n   Σ|S| needed   visited before   visited now
      400         2,914          195,600         4,194
      800         5,790          788,000         8,350
     1600        11,733        3,150,400        16,885
     3200        23,449       12,544,000        33,817
                                 531x, and       1.44x,
                                  doubling      constant

The partition is now an ordered partition whose parts are consecutive ranges of
one array of vertices. Splitting a part rearranges only that part's range, so no
other part moves and no part's position shifts. That buys three things at once:
two parts compare by position in O(1), which is what the between test needs; the
part holding a vertex is one lookup, so center in X and x in X stop being Θ(|X|)
scans; and a split needs no insert! into a list of parts. refine! then buckets S
by part and touches nothing else.

The pivot and module lists hold part numbers rather than copies of part
contents, so the findfirst(isequal(X), ...) searches are gone; entries are
removed by clearing a position index and skipping the superseded entry on the
way out. New part numbers go to the smaller side of each split, so relabelling
costs O(min(k, |X|-k)).

symgraph_factorizing_permutation on a graph with m ≈ 3n, as a SparseMatrixCSC,
and cost per word of that representation:

        n     before #8      after #8         now    ns/word now
      800       0.0106s       0.0033s     0.0006s          100.9
     3200       0.1958s       0.0470s     0.0023s          104.0
    12800             -             -     0.0110s          122.3

Flat from n = 200 to n = 12800, a 64x range: linear in the size of a sparse
input, which it has not been before. Dense input is unchanged, still flat at
about 30ns per word of an n² matrix.

As agreed on #3 this returns a different factorizing permutation than before.
Costing O(|S|) means never looking at the non-members of a part, so the half
that splits off is in the pivot set's order rather than the part's, which
changes which vertex leads a part and so which pivot gets chosen. Every
permutation is still a factorizing permutation and the trees are unchanged:

  * strong module trees identical to master over 3000 random graphs and
    digraphs, and dense and sparse input agree on every one;
  * 5,670,000 random starting orders over uniformly random graphs, twin-planted
    graphs and graphs with several planted modules at n = 8 … 14, every
    permutation modular;
  * every graph on n ≤ 6 against every starting order, 23,594,424 permutations,
    every one modular.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant