Shuffle less for SSE4.1#137
Conversation
Which |
| pack8(7, 6, 5, 4, 3, 2, 1, 0)}; | ||
| // We will read from bswap at offsets for fixed format output. Accomodate | ||
| // the necessary offset calculations by using char*. | ||
| // This is aligned to 64bits, so we won't cross a cache line when reading, |
There was a problem hiding this comment.
Good eyes! Will correct
The second one which reads the data immediately written before. This dominated the profile. I will commit a simplification of the evaluation of the new stuff in float_layout tonight. It's probably not surprising that the AI stuff was a bit too roundabout. I will then split this in, probably, three patches which can be benchmarked in sequence: 1. Move writing |
|
@vitaut so I made the mistake of benchmarking not only with gcc but also with clang while reworking the patch: the whole thing is a bit of a mirage, with clang this is actually a pessimization, and the speed up with gcc only brings it to up to par with the pesssmized clang. So, I think this patch is misguided, and the apparent gain is actually due to gcc being led away from some questionable choices it makes in the original version. OTOH thinking about this, it became clear that I should just do what I had considered but dismissed, namely put together the string and put the decimal point in the middle while still inside the SSE register, and only do a single store to memory (+ extra_digit and last_digit). This seems to work well, and I hope to have it ready for submission later this week. |
|
Thanks for the detailed analysis. This sounds like a good plan and I look forward to the updated version. |
|
This takes a bit longer than planned because I went down a nano-optimization rabbit-hole on this and got distracted by the other patch I just posted. I wanted to finish this today, but due to a sick child it will have to wait until Monday. |
|
No rush and hope your child gets better soon. |
d5c1463 to
d2ceb6a
Compare
|
I'm sorry, I had written a long comment about the changes, but seem to not have submitted it 🫤 Sorry about that, will recover it later. |
|
I still had the edit window open on this PC, so I could restore the message: This series became more interesting than I'd hoped, and so it took a bit longer to converge on something. Everything I tried beyond this -- if it made any change at all -- made clang regress and gcc improve. Seems like compilers don't like producing optimal code. Here are the numbers on my Zen4:
You will note that I added a new benchmark. This exercises the whole exponent range for the fixed numbers. Canada is situated in a geographical situation that limits the range of numbers encountered, and small numbers exercise a slightly different code path (the initial zeros are used), and so it's worth making sure that there is no regression. You will also notice that the benchmark numbers are within the 2-3% speedup range except for clang's zmij/canada test where a massive gain obtained. This is the result of the third patch in the series, which does something very simple (sorry about the AI generated commit message, I only noticed as I'm writing this that I didn't turn it into something readable). This patch moves the to_digits call on the fixed path after the memcpy which puts the zeros in place. I thought that a What the patch does is not very complicated: instead of I will not bore you with a recount of all the things I tried, as you can tell from the impact of the reordering, it can be quite unpredictable how things will benefit. |
|
Thanks for the update! The new benchmark makes sense but please move it to a separate PR and also rebase this one since there were some changes to the shuffle table. |
Instead of inserting the decimal point by writing to memory once
and then shifting everything past the point over, we create space
by shuffling the simd register and then fix up the extra digit
which have moved past the end of the register. The shuffle is
added to the fixed_layout_table.
Mechanics: per dec_exp, fixed_layout stores two 16-byte pshufb shuffle
tables (one per extra_digit) that place the natural-order BCD bytes
returned by to_digits into their final output positions, with the byte
at the decimal-point position (when it falls inside the 16-byte
vector) replaced by a pshufb zero-marker. The fast path is:
- one shuffle-table load + one pshufb + one 16-byte store
- byte store of BCD[15] at buffer[16] (only meaningful for
extra_digit == 1 with 0 <= dec_exp <= 14, where that byte sits
outside the vector window; in every other case the slot is
overwritten below by '.' or last_digit, so the write is
unconditional and harmless)
- byte store of '.'
- byte store of last_digit
Compared to main on -march=x86-64-v2, RelWithDebInfo, taskset -c 0
(16 repetitions, ns/double):
bench gcc clang
-------- ---- ----
zmij/fixed 7.41 -> 6.88 6.99 -> 6.60 (-7% / -5%)
zmij/canada 7.54 -> 7.01 7.29 -> 7.08 (-7% / -3%)
zmij (mixed) flat within stddev on both compilers
Two changes that together give clang ~5–7% on the fixed-path canada benchmark, neutral on fixed_range and on the scientific path, and are ~neutral on gcc (combined with the preceding select() commit): * Push `auto dig = to_digits(...)` into the two branches that consume it. In the fixed-path arm this scopes the result to that branch, which lets clang interleave the tail of to_digits (specifically the final `por '0'` ASCII offset) with the surrounding scalar work instead of forcing it on the critical path before the dec_exp range check. Counted once per call regardless of branch, so no extra work; the duplication is purely a scheduling hint. * Move the unconditional `buffer[bcd_size] = pextrb(digits, 15)` write ahead of the `pshufb`/`memcpy` pair. This lets the extract retire in parallel with the shuffle instead of after it.
e7e8b47 to
bcb7890
Compare
|
Confirmed the improvement on gcc on the first commit (rebased): and no effect on gcc on the second commit: |
|
I'm going to close this PR for now and for clang changes I suggest opening a new PR with a bit more detail about the improvements and how to repro them. Thanks! |
|
Thanks! Also for looking at the diffs separately. I've found it remarkably hard to get reliable speed measurements. Not only do different microarchitectures have different eprformance characteristics, but there is also less predictable stuff. E.g., with clang19 there's sometimes a massive speed regression on zen5 that I cannot at all understand from the assembly. Opus is hypothesizing that it's about the code being so small that more than one iteration of the benchmark fits in the out-of-order buffer, and one thus sees unrealistic and unpredictable pressure on the multiplication and shuffle ports. I've no idea if that holds, it also asserts a lot of things that are easy to disprove, but given the things that I saw with integer conversion #40 I'm willing to assume that CPUs indeed look at a lot of code simultaneously. Anyhow, I will make sure to include more details of the benchmarks going forward. |
2 participants
In the current code, we shuffle to reverse the byte order and we shuffle to shift the bits. This combines the two. It also replaces a slow
memmovefor the decimal digits with another shuffle + store, so that the data is never re-read from memory.This is my first AI-assisted patch to zmij, and the AI's main task was to figure out the special cases when moving the writing of the last digit behind the shuffles. Previously, the last_digit only sometimes was part of the memmove and could thus end up in a different position. The AI suggested to keep a table of where it goes, and so I did that. A table-free approach made the compiler add a branch and lose time.
The way I avoided to increase the table size for the various bswap + shift variations and thus messing up alignments or introducing padding is quite simple: what we want to do is to reverse the order and drop the first N digits, where N depends on whether its the integral part or the decimals, and it depends on the exponent. We don't care what is added in the end past the actual digits. So, it doesn't actually matter what the last few entries in the shuffler are, and so we just keep the same bswap shuffle table, read it at an offset and don't care what the entries at the end end up being. This only works because we reverse at the same time, so this cannot be carried to the NEON code. The additional bytes are loaded from the same struct, so it's not uninitialized memory and ASAN is happy, but I'm not sure what the standard says about this. I don't think the SIMD memory access is in its scope. BTW I changed the type of
bswaptochar *in order to get simpler index algebra and less casts.Finally, in order to keep everything branch-free, in the case of
dec_exp < 0where no decimal point is inserted between the digits, the identical shuffle + store operation is simply repeated. This turned out to be measurably faster than conditionally jumping past it. Note that the zmij/canada benchmark, given Canada's geography, doesn't contain any numbers < 1, so I created a separate benchmark (not included) that uses random fixed numbers. To my surprise it was actually faster than the processing of the Canada data. A possible explanation is that it triggered thelast_digitfar less often but I didn't check this in detail.On my Zen 4 this saves 5%, on my Tiger Lake it is performance neutral.
I also asked Claude to do try the same thing for NEON but none of the variations it tried came out faster than what is there right now.
ps I just saw that I didn't clean this up fully. Hence the non-squashed three-patch series with one unrelated change that I didn't mean to commit as part of this: I moved the
alignas(64)annotation from thestatic_datadeclaration to thestruct datadeclaration. The idea being that this way all the pointers that we use remember the alignment. Turns out the assembly with and without this change is the same on x64 https://godbolt.org/z/Mxz1fbKjY and ARM64 https://godbolt.org/z/ecnPzTWhK, but I think it's cleaner, so I'm not going to remove it right now even though it is unrelated.