Skip to content

Answer "which node is above this set" from an index - #10

Merged
StefanKarpinski merged 1 commit into
masterfrom
sk/indexed-tree-queries
Aug 24, 2026
Merged

StefanKarpinski merged 1 commit into
masterfrom
sk/indexed-tree-queries

Conversation

@StefanKarpinski

Copy link
Copy Markdown
Owner

Step 3 of the plan in #3: intersect_permutation, which #9 left as the digraph
path's bottleneck.

The problem

intersect_permutation asks two questions of a strong module tree, once for
each set in a family the size of the vertex set:

  • which node sits just above this set — parent_node, which walks down the
    tree materializing the leaf set of every child it passes;
  • is this set exactly one of the tree's modules — X in Ms, a linear scan
    over the whole list of modules comparing vectors.

Both are linear in the tree per question, so the pair costs Θ(n) per set and
Θ(n²) over the family.

Both are the same LCA question

The node wanted is the lowest one whose leaves cover the set; a set's cover is
fixed by its lowest and highest leaf alone; and the set is a module exactly when
that node's leaves are no more numerous than the set. So TreeIndex records,
in one pass over the tree, where each leaf falls in the leaf order, which range
of that order each node spans, and each node's parent. A query is then
O(|S| + depth) and touches no leaf set at all.

The result dictionary is keyed by node numbers rather than by the nodes
themselves, which hash their entire subtree.

And the tail that builds the tree

That part searched each parent for each child and filtered the child's leaves
out of it — O(|parent|·|child|) per pair, paid once per leaf of the child. The
sets are laminar, so sorting them by decreasing size makes the sets holding a
given leaf that leaf's ancestors, outermost first, which gives every set its
depth in one pass. Walking the tree from there is O(Σ|node|).

Effect

intersect_permutation on digraphs built by substitution:

n before after
200 0.0142s 0.0007s
400 0.0628s 0.0014s
800 0.2464s 0.0027s
1600 0.9518s 0.0057s 167×
3200 — 0.0132s
~n^1.9 ~n^1.0

End to end the digraph path goes 7.2510s → 2.6981s at n = 3200. Where its time
goes now:

n = 3200
G|G', G&G' 0.1220s
two symgraph_factorizing_permutation 1.4797s
two StrongModuleTree 0.4629s
intersect_permutation 0.0131s
StrongModuleTree(H, p) 0.2335s
sort_leaves! 0.4015s

intersect_permutation is 0.5% of it, rather than most of it. What dominates
now is the two factorizations, quadratic because G .| G' and G .& G' are
dense matrices — which is the same "linear in the input you were handed" story
as the symmetric path, since a dense matrix is n² words.

Verification

  • Trees identical to master over 3000 random graphs and digraphs.
  • TreeIndex agrees with parent_node, and its exactness flag agrees with
    X in Ms, on all 49,612 (tree, subset) pairs — every subset of every graph in
    a random sample.
  • 6000 digraph permutations checked against brute-force strong modules, over
    uniformly random and substituted digraphs, dense and sparse.
  • Every digraph on n ≤ 4 vertices — 4,164 of them — exhaustively.

🤖 Generated with Claude Code

https://claude.ai/code/session_013w7XdHqa7ynpy1nN1j19Qp

intersect_permutation asks two questions of a strong module tree, once per set
in a family the size of the vertex set: which node sits just above the set, and
whether the set is exactly one of the tree's modules. parent_node answered the
first by walking down the tree and materializing the leaf set of every child it
passed, and `X in Ms` answered the second by scanning the whole list of modules
comparing vectors. Both are linear in the tree per question, so the pair cost
Θ(n) per set and Θ(n²) over the family, and together they were most of the
digraph path.

Both are the same lowest-common-ancestor question. The node wanted is the lowest
one whose leaves cover the set, a set's cover is fixed by its lowest and highest
leaf alone, and the set is a module exactly when that node's leaves are no more
numerous than the set. TreeIndex records where each leaf falls in the leaf
order, which range each node spans, and each node's parent, in one pass over the
tree; a query is then O(|S| + depth) and touches no leaf set at all.

The result dictionary is keyed by node numbers rather than by the nodes, which
hash their entire subtree.

The tail that nests the resulting sets into a tree is rebuilt too. It was
searching each parent for each child and filtering the child's leaves out of it,
O(|parent|·|child|) per pair and once per leaf of the child. The sets are
laminar, so sorting them by decreasing size makes the sets holding a leaf that
leaf's ancestors, outermost first, which gives every set its depth in one pass;
walking the tree from there is O(Σ|node|).

That walk has to take a node's own leaves before any of its children, which is
not cosmetic and cost a round of CI to learn. A strong module can fail to be one
of the laminar sets and still have to come out contiguous, and the only thing
holding such a module together is that the leaves belonging to a node directly
are not separated by that node's children. The old code did this incidentally,
appending each child after filtering its leaves out of the parent. Emitting
children where their leaves fall instead splits such modules: on the five vertex
digraph now in test/factorizing.jl the module {1,4} came out at positions 1 and
4.

intersect_permutation on digraphs built by substitution:

        n      before       after
      200     0.0142s     0.0007s
      400     0.0628s     0.0014s
      800     0.2464s     0.0027s
     1600     0.9518s     0.0057s    167x
     3200           -     0.0132s
              ~n^1.9      ~n^1.0

End to end that is 7.2510s to 2.6981s at n = 3200. intersect_permutation is now
0.5% of the digraph path rather than most of it; what dominates now is the two
symgraph_factorizing_permutation calls on the dense matrices G|G' and G&G',
which is quadratic because those matrices are.

Trees are unchanged: identical to master over 3000 random graphs and digraphs.
The permutations remain factorizing permutations: 900,000 random digraphs over
six seeds, every one modular, where the bug above shows up about once in 50,000
and the same sweep reports failures against the code without the fix; plus every
digraph on n ≤ 4 exhaustively. TreeIndex agrees with parent_node, and its
exactness flag with `X in Ms`, on all 49,612 (tree, subset) pairs over every
subset of every graph in a random sample.

test/factorizing.jl gains the counterexample and a seeded sweep sized to the
rate the bug occurs at rather than to what is quick.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013w7XdHqa7ynpy1nN1j19Qp
@StefanKarpinski
StefanKarpinski force-pushed the sk/indexed-tree-queries branch from 82812b1 to 2672a9c Compare August 24, 2026 20:47
@StefanKarpinski

Copy link
Copy Markdown
Owner Author

Force-pushed: CI caught a real bug in the first version of this, and it is worth
recording what it was.

Walking the laminar family has to take a node's own leaves before any of its
children. I had emitted children where their leaves fall in the node's list,
reasoning that any depth-first order makes every set contiguous — which is true
of the sets themselves and false of what actually matters. A strong module can
fail to be one of the laminar sets and still have to come out contiguous, and
the only thing holding such a module together is that the leaves belonging
directly to a node are not separated by that node's children. The old
searching-and-filtering code did this incidentally, by appending each child
after filtering its leaves out of the parent.

The counterexample, now a regression test:

G = [0 0 1 0 0
     0 0 0 0 0
     0 1 0 0 0
     0 0 1 0 0
     0 1 1 0 0]

{1,4} is a module. The laminar family here is {1,2,3,4,5} ⊃ {1,2,4,5} ⊃ {2,5} — {1,4} is not in it. Grouping the own leaves of {1,2,4,5} gives
1 4 [2 5]; emitting the child in place gives 1 [2 5] 4, and the module comes
out at positions 1 and 4.

The verification I ran was too small to find this, which is the part I should
have caught myself.
The failure occurs in about one random digraph in 50,000
and I checked 6,000 — an expected yield of 0.11. I had made the same mistake
earlier in this issue's work, on a stress test that passed happily against the
code it was meant to guard, and did not carry the lesson over.

Now:

  • 900,000 random digraphs over six independent seeds, every permutation modular;
    the same sweep reports failures against the code without the fix, so it does
    detect the defect rather than merely passing;
  • every digraph on n ≤ 4 exhaustively;
  • trees still identical to master over 3,000 graphs and digraphs.

test/factorizing.jl gains the counterexample and a seeded sweep of 25,000
digraphs at n = 4…7, which reports two failures against the unfixed code and
runs in about a second and a half. Both were checked against a pre-fix copy of
the source before being committed.

@StefanKarpinski
StefanKarpinski merged commit 68f580b into master Aug 24, 2026
15 checks passed
@StefanKarpinski
StefanKarpinski deleted the sk/indexed-tree-queries branch August 24, 2026 21:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant