perf(#5954): derive the directed adjacency once as a CSR and share it (slices 5+6)

[cajasmota]

What

The directed adjacency is now derived once, as a compact CSR (int32 offsets + int32 neighbours + float64 weights), built directly from the relationship slice in BuildGraph. sampledBetweenness and the in-repo Louvain both consume it instead of walking the gonum graph and boxing ~1.33M edges into interfaces.

The directed gonum graph stays — PageRank and IdentifyArticulationPoints still use it. It is now materialised from the CSR rather than alongside it, which is what makes this a wall-time win as well as a memory one (see below).

Deleting the intermediate simple.WeightedUndirectedGraph — which existed only to be walked straight back out into Louvain's own CSR — turned out to be a bigger win than the shared CSR itself.

Measured

Fixture buildHeavyTailedGraph(60000, 4, 5), ~236k relationships / ~205k edges. The reference corpus is ~6.5x more edges, so the corpus-scale saving is unmeasured and deliberately not extrapolated — prior estimates in this epic were wrong three times, in both directions.

	main	HEAD
ComputeCommunities	177.71 MB/op, 1,004,964 allocs	36.62 MB/op, 108,848 allocs
sampledBetweenness (K=512)	34.11 MB/op, 313 allocs	8.17 MB/op, 278 allocs
whole pass churn	499.3 MB/op	335.9 MB/op
whole pass peak HeapInuse	229.6–241.7 MB	190.1–191.6 MB
whole pass wall	1150 ms	857 ms (−25%)

The regression that the benchmark could not see

Review caught this and it is the most useful thing in the PR. The first version computed each relationship's edgeWeight(rels[i].PropsSnapshot()) twice — once in BuildGraph's own loop, once in the CSR pass. PropsSnapshot() builds a fresh map[string]string each call.

That cost was invisible in the benchmark, because csrBenchFixture builds relationships with no properties at all, so PropsSnapshot() returned nil and edgeWeight short-circuited. The real corpus carries callsite_count/confidence on essentially every edge. A fixture that structurally cannot exhibit the cost of the thing being measured is worse than no benchmark.

Measured, 3x10 runs back-to-back, BuildGraph ms/op:

	main	two-walk version	HEAD
rels with props	130.8	156.9 (+20%)	119.6 (−8.5%)
rels without props	115.3	127.5 (+10.6%)	105.6 (−8.4%)

Fixed by inverting the order rather than threading a weight slice through (which would have cost a len(rels)-sized float slice, ~15 MB at corpus scale, in a memory-reduction epic): build the CSR first, materialise the gonum edge set from it. One property read per relationship, and gonum also loses its lookup/remove/re-add chain for parallel edges because the CSR arrives pre-collapsed. BenchmarkBuildGraph_WithProps and csrBenchFixtureWithProps are added, and csrBenchFixture's doc now states outright that it cannot exhibit property cost.

That restructure made two tests tautological — with g derived from the CSR, TestDirectedCSRCollapsesParallelEdgesWithAccumulatedWeight and TestDirectedCSRHighFanOutRow were comparing the CSR against itself. This was caught because mutation M9 stopped killing them. Both now compare against a verbatim legacyBuildGraph oracle, and TestDirectedCSRMatchesGonumAdjacencyBitForBit checks the CSR and g against it. M9 kills again.

Equivalence

BuildGraph's three load-bearing behaviours are reproduced exactly and pinned individually: a node for every entity including isolated ones, self-loops dropped, parallel edges collapsed. The collapse packs (target, arrival) into an int64 sort key so the float accumulation order matches gonum's w += existing.Weight() chain.

Numerical equivalence is asserted, not assumed. TestSampledBetweenness_BitIdenticalToLegacy pins the raw values — it is what catches hoisting (sigma[p]/sigma[w]) * (1+delta[w]), which is algebraically equal but not bit-identical. TestComputeCommunitiesMatchesLegacyGonumPath deep-equals the entity→community map, community sizes, TopEntities, the denoise count, and per-community modularity bit-for-bit across 4 fixtures x 3 legacy repetitions. Community membership is structurally immune to the sum reordering: the partition comes from louvainPartitionFromCSR(ucsr), and ucsr is asserted bit-identical to the legacy projection.

20 mutations introduced, 20 caught. Two further candidates were dropped as genuinely equivalent rather than uncaught, with the algebra shown: v <= u → v < u in the modularity sweep (buildCSRFromDirected cannot emit v == u), and removing that filter entirely (uniform doubling of I, K and m2 leaves q invariant and preserves the degree ordering).

Hardening from review

• The int32 bound now binds. It previously checked only n, never the edge count, while the doc claimed both — and the degraded path returned a structurally valid, all-zero-offset CSR that newBCScratch accepted, yielding all-zero betweenness and n singleton communities. A silent wrong answer, persisted as scores. Both bounds are now enforced (edge count accumulated in int64 so the check cannot itself wrap) and it panics. csrIndexLimit became a var only so the test can lower it and drive the real BuildGraph — a bound reachable only through a helper is the "guard that does not bind the path that runs" this codebase keeps producing. Independently verified: disabling the edge-count check fails TestDirectedCSRIndexLimitIsEnforced/edge_count.
• directedCSR now has no writer at all after BuildGraph returns; the undirectedDerivations counter moved to nodeIndex. That makes the invariant justifying newBCScratch's aliasing literal rather than "read-only except one field".
• High-fan-out fixture added — a 60k-out-degree hub with a second interleaved wave of parallel edges over every third target (duplicates far apart in arrival order) plus a back-edge. This is the one shape a pathological degree distribution, rather than sheer size, changes behaviour for; mutation M16 (scratch sized by the wrong bound) is caught only by this test.
• buildCSRFromUndirected and louvainPartitionWithSweeps now have no production caller and survive as test adapters, so the entire Louvain quality suite runs against the adapter. That is sound because TestUndirectedProjectionFromCSRMatchesGonumProjection proves the two projections bit-identical — and both functions now carry a comment naming that test as the single bridge, and stating that weakening it silently disconnects the quality suite from production.

Also fixes the red main

TestComputeCentrality_ByteIdenticalOnSampledPath has failed on main since it was introduced by #6017 — its legacy oracle pre-seeded a zero per entity while #6011 had deliberately made the centrality map sparse (legacy=4000 new=852). The expectation is now a literal mirror of the production loop at algorithms.go:983-989, fixing the key set while leaving values compared bit-for-bit via math.Float64bits.

This should have been a preceding standalone commit — an author repairing the oracle that guards their own change is a pattern a reviewer should never have to take on trust. It was instead verified independently: by provenance, by matching the expectation to production, and by three mutations (perturbed value, dropped pivot, corrupted shared adj) that each kill the repaired test.

Verification

go build ./... && go vet ./... && gofmt -l . clean. internal/graph 428 pass; -race -count=2 556 pass. Full repo exit 0, 228 packages, zero failures — confirmed via rtk proxy go test rather than the rtk summary line, which truncates above 10 MiB and reports a green summary over a red run.

Known limits

Corpus-scale saving unmeasured. Equivalence fixtures top out at 3000 entities apart from the 60k hub. The int32 bound is enforced but tested only via a lowered limit, never at true scale.

Remaining known boxing: IdentifyArticulationPoints still walks g.Edges() once — the last boxed edge iteration in the pass, and the obvious next slice.

Independently adversarially reviewed.

Closes #6024
Refs #5954