Genuine OoO issue vs. CV32A65X's issue window, does HPD-Cache's MLP go underused?

[Nameer-Iqbal-Ansari]

I've been reading through CV32A65X's design docs alongside the CVA6S+ and "Ramping Up Open-Source RISC-V Cores" papers, and I have a question I haven't seen addressed directly.

CV32A65X issues in-order from what looks like a small (2-entry) issue window. HPD-Cache is built to support multiple outstanding misses. If the head of that window stalls on a miss, is there anywhere left for the front end to find independent, ready work to issue in the meantime, or does that memory-level parallelism mostly go unused until the front end unblocks?

I have a small 2-wide, genuinely out-of-order RV32I core (RAT-based renaming, 28-entry ROB, 12-entry reservation stations) that I built at MERL, and I'm trying to work out whether pairing something with a deeper, readiness-based issue window against HPD-Cache would show a real difference on memory-bound, irregular-access workloads, or whether CV32A65X already captures most of the realistic benefit for the workloads this family actually targets.

Genuinely not sure which way this goes, and would rather find out from people who've already measured it than assume.

[cainria]

(My two cents: As CVA6 is in-order, I do not think we can consider an issue window here: it does not choose an instruction in a window like an OoO core would, it issues in-order instead.)

TL;DR: I don't think so, here are some details about why (the load unit is blocking, not the issue stage)

First, let me recap the CVA6 pipeline stages here: IFetch | BPred | Decode | Issue | Execute | Commit
Loads are handled in the execute stage (there is no memory stage in CVA6), not in the issue stage.
Also, and this may be what makes CVA6 specific within in-order cores: it has a ReOrder Buffer called the scoreboard, allowing instructions to complete out of order (but it is not an out-of-order CPU as it does not read operands out-of-order) in independent functional units in the execution stage.

Consequently, if there is a miss load, the CPU continues issuing next instructions as long as there is no data (dependence) or structural hazards.
It means that there can be a multi-cycle instruction in the execute stage such as a load, followed by a short instruction such as an add, and the add can complete first.
Can it work with two loads?

1:  lw a0, 0(s0)  # miss
2:  lw a1, 0(s2)  # hit
3:  addi a2, a1, 42

Instructions 1 and 2 are independent so 2 can be issued before 1 completes (but they cannot be issued simultaneously because the load unit only has one issue bus)
Issue of instruction 3 is blocked by a data hazard with instruction 2 which did not produce a result yet. It also blocks next instructions as CVA6 is in-order.

Instruction 3 would benefit from having the result from 2 before 1 completes.
Now the question is: does the load unit send the second load to the cache before the first one completes?
I really don't think so, but it can be worth checking!

Maybe @cfuguet has an opinion on this? (Sorry to ping you again!)

[cfuguet]

Hello @cainria and @Nameer-Iqbal-Ansari,

Happy to jump in.

To answer your question @cainria, yes, the CVA6 can issue multiple loads, with multiple of them triggering a cache miss.

Of course, multiple conditions must be true for this to happen:

• Loads are independent of each other (the source address of one cannot depend on the result of another pending load or of any other pending operation). If this is the case, the issue stage will stall.
• The scoreboard is not full (the number of entries is defined by parameter NrScoreboardEntries).
• The load buffer in the load unit is not full (the number of entries is defined by parameter NrLoadBufEntries).

If these conditions are true, the CVA6 can issue multiple loads to the HPDcache (even without yet receiving the response to previous loads). Then the HPDcache will accept and process them if the internal buffers are not full (MSHRs, ReplayTable,...).

As an in-order issuing core, the opportunities to exploit memory-level parallelism (MLP) in the CVA6 are scarce, but still exist. Usual compiler optimizations such as loop unrolling or batch loads and stores help to improve the MLP. Of course, in desperate cases, you need to manually modify your code, or even write some subroutines in assembly to effectively exploit MLP.

Anyway, even if the core itself finds it hard to exploit MLP, the MLP of the HPDcache can be exploited by connecting a memory prefetcher (e.g., https://upcommons.upc.edu/server/api/core/bitstreams/6518bdf7-3d56-4c9b-a960-c8f1897b9297/content or https://ieeexplore.ieee.org/document/10769136).

[Nameer-Iqbal-Ansari]

Hi @cfuguet, appreciate you jumping in, and good to know the real gating conditions, scoreboard, and load-buffer depth, not a single-issue bus. That's a more precise picture than I had.

The part that's most useful to me is "the opportunities to exploit MLP are scarce, but still exist", and that getting them in practice leans on compiler reordering or hand-written code. That's the real distinction I was reaching for with the original question, just stated more precisely than I had it: CVA6 can expose MLP when something upstream has already arranged the code to make it visible; a readiness-based scheduler finds it automatically, at runtime, without that arrangement.

The prefetcher suggestion is a fair, real alternative, and I'll read both papers. Though I think it only closes part of the gap. Prefetching works well on regular, predictable access patterns, but not on genuinely data-dependent ones, a load whose address depends on the result of another in-flight load, where there's nothing to predict ahead of time. That's the specific case neither in-order issue nor prefetching can help with, and it's exactly what dynamic, readiness-based issue is built for.

If that distinction holds up, it seems like the fair comparison isn't "OoO vs. CVA6+prefetcher" in general, it's specifically irregular, data-dependent access patterns where prefetching has nothing to go on. Does that match how you'd think about it, or is there a reason that case is rarer in practice than I'm assuming for the workloads this family actually targets?

[cfuguet]

Of course, that is why OoO cores exist. They are much better exploiting MLP than InO cores. And the HPDcache shows all its potential with OoO cores. But OoO cores come with a (high) price in terms of energy and area that you only pay for General Purpose or HPC systems.

I have a not yet open-source gem5 model of the HPDcache, that I've associated with OoO cores and results are very good. Projects like Cinco Ranch chip from Barcelona Supercomputing Center (BSC) associate OoO RISC-V cores with the HPDcache shows also good results.

By the way, if you integrate the HPDcache with your OoO core I'll be very happy to play with !