Measure the memory hierarchy and threadgroup memory, which reorders the target model

[AndreSlavescu]

meTile recorded one bandwidth number — 120.6 GB/s streaming from DRAM — and it isn't the whole story.
The gap surfaced by accident while scoping attention projections: a 786 KB weight read at 254 GB/s, twice
the recorded ceiling, so something above DRAM was serving it.

The hierarchy

working set	GB/s	vs DRAM
≤ 2 MB	2386	19.8x
4 MB	555	4.6x
8 MB	192	1.6x
32 MB	134	1.1x
≥ 64 MB	121–128	1.0x

The knee at 2→4 MB is a factor of four in one step. That makes fitting a working set the largest lever
in the target model by some distance — choosing the matrix unit over scalar is worth 2.4–3.7x, and
instruction scheduling is capped at 1.09x and measured unreachable above MSL. A tiling that fits and one
that misses are not the same kernel, and until now the compiler had no way to know. read_bandwidth_gbps,
resident and tiling_gain give a pass something to consult.

Threadgroup memory is barely faster, and much more fragile

The shared-memory passes all assume it's the fast place to put data. Measured against a resident device
read over the same 32 KB:

stride	shared	device
16 B	3361	2749
128 B	1216	2079
144 B	2322	2341
256 B	605	2004
512 B	437	1641

Contiguous, staging wins 1.22x — a far smaller margin than the usual scratchpad assumption, so no pass
can justify staging on bandwidth alone. And across strides the shared arm spreads 7.69x against the
device arm's 1.80x, which inverts the habit of treating shared memory as forgiving and device memory as
the thing needing careful access. Here device memory is the forgiving one.

144 bytes is one vector larger than 128 and reads nearly twice as fast, so the collapses are bank
alignment, not distance — 32 banks of 4 bytes puts every lane on the same bank at 128.

This confirms _optimal_pad

It was written from a stated model of 32 four-byte banks and never measured. It pads to an odd stride,
odd strides don't collapse, so the pass was right for the reason it claimed. A test ties the two together
so they can't drift apart.

Three controls, and two wrong turns

• Loop collapse: the pass loop re-reads the same addresses, so the backend could fold it into a
  multiply. Tripling traffic triples elapsed time at every size; doubling passes doubles it at every
  stride (1.88–2.00x). It isn't folded.
• Saturation: the sweep must reach the known 120.6 GB/s at large sizes or the thread count is too low
  and everything is a lower bound. It reaches 120.5, and the probe reports the check rather than assuming.
• Monotonicity: a larger working set is never served faster. This caught the resident regime twice —
  256 KB reading slower than 512 KB, then 512 KB slower than 1 MB. That's the pass loop's bookkeeping
  taking a bigger share at small sizes, so the regime is recorded as one number (2386, a floor) rather
  than four that would claim resolution the probe doesn't have.
• A wrong diagnosis: a span of 8 vectors measured half speed, so I added a padded arm to break the
  conflict. It got worse — the unpadded span was 7 and +1 moved it onto 128 bytes. Padding is an
  arithmetic result, not a direction.

686 pass. Lint and vulture clean.

🤖 Generated with Claude Code