Initial riscv64 vector support (uses standard vector instrinsics for rvv 1.0. Presently VLEN=256 only.)

[mjosaarinen]

Summary:
rv64v support (risc-v vector extension 1.0, which is available on newer application-class silicon.)

Do you expect this change to impact performance: Yes/No
yes (risc-v only)

If yes, please provide local benchmarking results.
Roughly 2.5x perf on silicon with vector hardware.

[hanno-becker]

@mjosaarinen If you have nix setup, running autogen should hopefully resolve the linting issues.

[rod-chapman]

Note that on the "fastntt3" branch, there are layer-merged implementations of the NTT and INTT that are highly amenable to auto-vectorization with compilers like GCC 14. Benchmarks of that code on an RV64v target were encouraging, so might provide some inspiration for a fully vectorized, hand-written back-end.

[mjosaarinen]

Note that on the "fastntt3" branch, there are layer-merged implementations of the NTT and INTT that are highly amenable to auto-vectorization with compilers like GCC 14. Benchmarks of that code on an RV64v target were encouraging, so might provide some inspiration for a fully vectorized, hand-written back-end.

Yeah you can easily double the speed with autovectorization alone, and some Google folks were of the opinion that they wanted to rely on that entirely in BoringSSL (RISC-V Android etc), rather than maintain a hand-optimized version. The resulting code is pretty wild; I looked at that when considering RISC-V ISA extensions ( see slides 17 for example in https://mjos.fi/doc/20240325-rwc-riscv.pdf ). It was almost "too good" -- I suspect that Google has used those NTTs as a microbenchmark when developing LLVM autovectorizers :)

[mjosaarinen]

@mjosaarinen If you have nix setup, running autogen should hopefully resolve the linting issues.

Yeah, sorry for abusing your CI like that (I wasn't expecting it to be that extensive), I could have just read the documentation. I'll set up this nix thing.

[hanno-becker]

@mjosaarinen Sorry, we should have pointed that out earlier. With the nix environment, you should not need to waste anymore time making the linter happy. Just run format && autogen before pushing.

[hanno-becker]

@mkannwischer @mjosaarinen The RV functional tests in the ci-cross shell don't seem to pass.

% nix develop --extra-experimental-features 'nix-command flakes' .#ci-cross
% CROSS_PREFIX=riscv64-unknown-linux-gnu- make func OPT=1 AUTO=1 -j32
% EXEC_WRAPPER=qemu-riscv64 make run_func_512 -j32 OPT=1 AUTO=1
qemu-riscv64 test/build/mlkem512/bin/test_mlkem512
ERROR (test/test_mlkem.c,41)
ERROR (test/test_mlkem.c,225)
make: *** [Makefile:53: run_func_512] Error 1

Same for OPT=0.

Could you investigate?

[hanno-becker]

The RV64 cross tests in CI are failing. I don't know if this is an issue with the code or the CI setup, but it needs looking into.

[mjosaarinen]

The RV64 cross tests in CI are failing. I don't know if this is an issue with the code or the CI setup, but it needs looking into.

It looks like some things are running fine, and some things have build problems. I suspect that platform auto-detection code (in the C header files + makefies) is to blame (failing no_opt means "no optimization"?) It's a quite complicated CI -- can you tell me what would be the best way to debug this stage of the CI build process locally?

[hanno-becker]

It's a quite complicated CI -- can you tell me what would be the best way to debug this stage of the CI build process locally?

Sure. If you have nix installed, then it's exactly what I posted above: You open the nix shell with

% nix develop --extra-experimental-features 'nix-command flakes' .#ci-cross

Building using cross compiler:

% CROSS_PREFIX=riscv64-unknown-linux-gnu- make func AUTO=1 

And then running:

% qemu-riscv64 test/build/mlkem512/bin/test_mlkem512
% qemu-riscv64 test/build/mlkem768/bin/test_mlkem768
% qemu-riscv64 test/build/mlkem1024/bin/test_mlkem1024

The base command line used by AUTO=1 above is:

riscv64-unknown-linux-gnu-gcc -Wall -Wextra -Werror=unused-result -Wpedantic -Werror 
  -Wmissing-prototypes -Wshadow -Wpointer-arith -Wredundant-decls -Wno-long-long 
  -Wno-unknown-pragmas -Wno-unused-command-line-argument -O3 -fomit-frame-pointer 
  -std=c99 -pedantic -MMD  -march=rv64gcv_zvl256b 
  -DMLK_FORCE_RISCV64 -DMLK_CONFIG_USE_NATIVE_BACKEND_ARITH 
  -DMLK_CONFIG_USE_NATIVE_BACKEND_FIPS202 -DMLK_CONFIG_PARAMETER_SET=1024 

What is confusing to me is that this happens regardless of whether OPT=0 or OPT=1 (it defaults to OPT=0). If OPT=0, the new files aren't even compiled.

If you don't want be bothered by auto-settings, we can for now set AUTO=0 and manually specify everything on the command line using CC and CFLAGS:

% CFLAGS="-march=rv64gcv_zvl256b " CC=riscv64-unknown-linux-gnu-gcc make func AUTO=0
...
% qemu-riscv64 test/build/mlkem1024/bin/test_mlkem1024
ERROR (test/test_mlkem.c,49)
ERROR (test/test_mlkem.c,225)

[hanno-becker]

@mkannwischer @mjosaarinen

It looks like the issue is solely due to -march=rv64gcv_zvl256b. Even on main, if you do:

CFLAGS="-march=rv64gcv_zvl256b" CC=riscv64-unknown-linux-gnu-gcc make func -j32 AUTO=0 OPT=0
qemu-riscv64 test/build/mlkem512/bin/test_mlkem512

it'll fail with

ERROR (test/test_mlkem.c,41)
ERROR (test/test_mlkem.c,225)

Probably some configuration is missing in the invocation of qemu-riscv64? Nowhere do we say that we're emulating a system with 256-bit vector extension.

[hanno-becker]

qemu-riscv64 -cpu rv64,v=true,vlen=256 works. @mjosaarinen is that the right config?

[mjosaarinen]

qemu-riscv64 -cpu rv64,v=true,vlen=256 works. @mjosaarinen is that the right config?

Hmm. I'm not sure if qemu is able to pull the standard runtime and dynamic libraries that match with the "vector ABI" (the calling convention changes somewhat with vector registers). I had to link the test programs with -static when I was testing the code. I was testing with spike and vector silicon myself, sorry I have not looked yet at the qemu environment used by CI.

Anyway, the code assumes "vector 1.0" extension, 256-bit VLEN, and the standard "gc" stuff, so hat looks correct. I left the Keccak instruction stuff out.

[hanno-becker]

I adjusted the CI. Most tests pass, but there are some issues in the ACVP tests and the monobuild examples. The ACVP one seems to be an issue in the test script (not expecting parameters in the exec-wrapper), while the monobuild failure seems to be an architecture confusion. I'll take a look.

[hanno-becker]

You know, an appropriate thing would be to work out the actual bounds systematically -- for every step and (proven-correct) arithmetic "gadget" -- and always have the proven output properties as input assumptions further down the line.

Yes! For the record, this is exactly what the HOL-Light proofs for Arm are doing. You will see tight bounds in the HOL-Light specs that would be hard to verify on paper and impossible by eye. However, the frontend never relies on those bounds. The bounds required by the frontend can so far be audited by eye. This means, we go as far into lazy reduction as can easily be annotated and followed in the code, but not further. That may leave some unnecessary reduction in the code, but at least for the invNTT the returns of more and more lazy reduction diminish quickly.

[mjosaarinen]

@mjosaarinen Let's do as you say. I think one can follow the Arm backend here, but if that's so -- I will try out to be sure -- it can be a follow-up.

Similarly, you mentioned you aim for a VL agnostic version. Can this be a follow up too?

I think it can be a follow-up - there is always a fallback to the C back-end.

The thing about that is; while some parts can be easily made VLEN-agnostic using the "strip-mining" approach (discussed in the RISC-V manual), it seems to me that for efficient NTT/INTT, one would need to have separate versions for VLEN in { 128, 256, 512, 1024, perhaps others } due to the actual register allocation.

btw.. In RVV, the vector length VL is different fom VLEN -- VL holds the dynamic vector length (in elements, like bytes or floats) while VLEN is the hardware size (in bits) of each of the 32 vector registers. Vectors (VL) do not have to be a power-of-two in size and can span up to 8 vector registers (LMUL). The max hardware register size allowed by the spec is VLEN=65536 bits. Due to LMUL one can technically do a lot of stuff in a single instruction (operate on 8*256 = 2048 bits even with VLEN=256) so that's an another thing to explore. This code uses only LMUL=1 and LMUL=2.

[hanno-becker]

@mjosaarinen Very interesting, thank you. I should say "VLEN" then, not "VL".

What else needs doing then for this PR, from your perspective? I will aim to go through everything again, probably later or tomorrow.

Do you have input on my note above on runtime detection of vector extension support?

We should extend src/sys.h with the necessary system capability and guard the backend accordingly -- see e.g. how it's done for AVX2. Otherwise, one cannot target hosts with heterogeneous capabilities with a single build (e.g. RV64 CPU with/without vector extension, or with vector extension but different length). Probably we want a capability for "Vector Extension of VL=256". What's the correct name here? Once we have that, we should extend CI to test the library on a CPU a) without vector extension, b) with vector extension, but different VL -- in both cases, it should just fall back to the C code, of course. I can do that part, though.

[hanno-becker]

0-diff rebase to remove merge commits.

[hanno-becker]

Did some history cleanup, and re-added bounds assertions aiming to check that coefficients stay unsigned-canonical.

[mjosaarinen]

Did some history cleanup, and re-added bounds assertions aiming to check that coefficients stay unsigned-canonical.

It no longer even compiles (this file is needed.)

  CC      test/build/mlkem1024/unit/mlkem/src/native/riscv64/src/rv64v_poly.c.o
mlkem/src/native/riscv64/src/rv64v_poly.c:15:10: fatal error: rv64v_settings.h: No such file or directory
   15 | #include "rv64v_settings.h"
      |          ^~~~~~~~~~~~~~~~~~
compilation terminated.

[mjosaarinen]

@mjosaarinen Very interesting, thank you. I should say "VLEN" then, not "VL".

What else needs doing then for this PR, from your perspective? I will aim to go through everything again, probably later or tomorrow.

Well, I was thinking of running some benchmarks on real RVV hardware (SpecemiT X60) on Monday-Tuesday and perhaps tweaking some things a bit based on results.

Do you have input on my note above on runtime detection of vector extension support?

We should extend src/sys.h with the necessary system capability and guard the backend accordingly -- see e.g. how it's done for AVX2. Otherwise, one cannot target hosts with heterogeneous capabilities with a single build (e.g. RV64 CPU with/without vector extension, or with vector extension but different length). Probably we want a capability for "Vector Extension of VL=256". What's the correct name here? Once we have that, we should extend CI to test the library on a CPU a) without vector extension, b) with vector extension, but different VL -- in both cases, it should just fall back to the C code, of course. I can do that part, though.

From a practical viewpoint, it should be ok to have the existence of RVV (yes/no) as a build-time flag. After all, I understand that the Android and Ubuntu kernel & userland will be built for an RVA23-like processor profile and wouldn't work without Vector. Now, the physical vector length (VLEN) is another matter; it is intended to be dynamic and is supposed to be handled by the programs directly. RVV code can do that with 1 instruction (it's designed to be fast) and without syscalls etc.

[hanno-becker]

From a practical viewpoint, it should be ok to have the existence of RVV (yes/no) as a build-time flag. After all, I understand that the Android and Ubuntu kernel & userland will be built for an RVA23-like processor profile and wouldn't work without Vector. Now, the physical vector length (VLEN) is another matter; it is intended to be dynamic and is supposed to be handled by the programs directly. RVV code can do that with 1 instruction (it's designed to be fast) and without syscalls etc.

If that's the case, the PR needs further work to make the dispatch to the backend dynamic -- at the moment, it is static. If I understand you correctly, at compile-time we should not have MLK_SYS_RISCV64_V256 (currently in sys.h), but merely MLK_SYS_RISCV64_RVV indicating compile-time support for RVV. And the backend, one would use the "VLEN-instruction" you mentioned to dynamically dispatch to C or the backend code. Once that's done, the CI should be extended to exercise multiple VLENs (cf. multi-functest/action.yml).

Can you please make the corresponding adjustments?

Is there a similarly convenient way to infer without syscalls whether RVV itself is present?

[hanno-becker]

Current SpacemiT K1 8 (Banana Pi F3) benchmarks 👍

Benchmark suite	Current: 3fa9c60	Previous: 5aaca94	Ratio
ML-KEM-512 keypair	158993 cycles	227126 cycles	0.70
ML-KEM-512 encaps	169731 cycles	271033 cycles	0.63
ML-KEM-512 decaps	214114 cycles	345293 cycles	0.62
ML-KEM-768 keypair	266697 cycles	376324 cycles	0.71
ML-KEM-768 encaps	285106 cycles	433349 cycles	0.66
ML-KEM-768 decaps	348528 cycles	529898 cycles	0.66
ML-KEM-1024 keypair	402950 cycles	560159 cycles	0.72
ML-KEM-1024 encaps	433437 cycles	636793 cycles	0.68
ML-KEM-1024 decaps	516474 cycles	758317 cycles	0.68

[hanno-becker]

If buflen is not a multiple of vl23 * 2, then this function may leave bytes unprocessed in the input buffer, which would be incorrect. We should reject the input upfront if that's the case (thereby falling back to C), or add tail-handling in support of this case. As far as I understand the code, it should be simple to basically handle the tail in the same way as the bulk of the loop, adapting vl23.

This starts to be an issue with VLEN>=512, where vl23 = 48 = 3 * 16 which is not a multiple of the concrete buflens MLK_XOF_RATE and MLK_XOF_RATE * MLKEM_GEN_MATRIX_NBLOCKS this function is called with.

[hanno-becker]

The issue surfaces in the KATs as soon as one tests with VLEN=512.

[hanno-becker]

  while (ctr < len && pos < buflen)
  {
    const size_t vl = __riscv_vsetvl_e16m1((buflen - pos) * 8 / 12);
    const size_t vl23 = (vl * 24) / 32;

    const vuint16m1_t vid = __riscv_vid_v_u16m1(vl);
    const vuint16m1_t srl12v = __riscv_vmul_vx_u16m1(vid, 12, vl);
    const vuint16m1_t sel12v = __riscv_vsrl_vx_u16m1(srl12v, 4, vl);
    const vuint16m1_t sll12v = __riscv_vsll_vx_u16m1(vid, 2, vl);

   ...

seems to work. Experimented a bit with the other functions, too, and it looks like 1f2fce3 passes QEMU tests for all supported VLENs (128,256,512,1024), skipping [inv]NTT unless VLEN=256 for now.

It's not yet clear to me why the [inv]NTT doesn't work for VLEN > 256 even if in the code one forces vl = 16. Should that behave as if VLEN=256? understood

[hanno-becker]

@mjosaarinen At the moment, we compile with "CFLAGS += -march=rv64gcv_zvl256b". Is this still the right option if we want to compile for RVV 1.0 but without fixing the vector length? Or does this merely signify a minimal vector length?

EDIT: Looks like we want just -march=rv64gcv. See 1f2fce3

Btw, leaving aside #1037 (comment), it's pretty cool to see the length-agnostic implementation of rej_uniform 👍

[mjosaarinen]

So, I did some further benchmarks. It's well-known that LLVM has a really good autovectorizer for RISC-V (with large contributions from Google, I think. BoringSSL's MLKEM is not too slow on RISC-V because of that.)

gcc: riscv64-unknown-linux-gnu-gcc (GCC) 14.2.1 20250322
clang: clang version 22.0.0git (I built this from main on Sep 8, 2025)
Intrin: ml-kem native intrinsics enabled (-O3 -march=rv64gcv)
Autovec: plain C, just autovectorization (-O3 -march=rv64gcv)
No vec: plain c, no vector (-O3 -march=rv64gc)

The gcc intrin vs autovec numbers are within 1 or 2% of those previous benchmarks, probably down to measurement method (my runs were probably shorter).

		Intrin	Intrin	Autovec	Autovec	No vec	No vec
Algorithm	Function	clang	gcc	clang	gcc	clang	gcc
ML-KEM-512	Keypair	141480	160647	186322	226757	313511	311440
ML-KEM-512	Encaps	152256	170002	203173	273024	414068	422153
ML-KEM-512	Decaps	189825	215589	240965	347203	549109	566565
ML-KEM-768	Keypair	240108	271747	313054	376235	511832	510837
ML-KEM-768	Encaps	257651	287527	335101	436653	639801	649390
ML-KEM-768	Decaps	311529	353817	384829	535102	810516	833938
ML-KEM-1024	Keypair	363202	406382	471771	557870	744641	736261
ML-KEM-1024	Encaps	393483	434324	505658	636611	904048	908036
ML-KEM-1024	Decaps	464661	518650	567039	761398	1109203	1127421

Some observations:

• LLVM 22git autovectorization is 29% faster than GCC 14.2.1 autovectorization.
• Intrinsics give a 28% (clang) or 48% (gcc) speedup over autovectorization.
• Using intrinsics + autovectorization means 2.42x (clang) or 2.18x (gcc) speedup over not using vector instructions.
• With plain rv64gc ISA, the two compilers are roughly the same -- clang often slower!

[hanno-becker]

Thanks for the analysis @mjosaarinen! Looks like the move to intrinsics still brings a notably benefit over autovectorization.

Note that we omit poly_add/sub from the backend API because of the assumption that autovectorizers should have an easy time with them. mlk_rv64v_poly_add and mlk_rv64v_poly_sub can thus be removed from this PR.

[rod-chapman]

@mjosaarinen Thanks for those data. Could you re-run the "Autovec" set with clang 22 on the "fastntt3" branch please? I'm intrigued to see how clang does on that code, which was specifically written to be amenable to auto-vectorization. Results using GCC 14 on AArch64 were surprisingly good.

[mjosaarinen]

@mjosaarinen Thanks for those data. Could you re-run the "Autovec" set with clang 22 on the "fastntt3" branch please? I'm intrigued to see how clang does on that code, which was specifically written to be amenable to auto-vectorization. Results using GCC 14 on AArch64 were surprisingly good.

Unfortunately I already packed away my own SpacemiT X60 board as I am traveling tomorrow. (But I think there is one plugged into the CI somehow, perhaps one could try that one.)