[CV32A60X] Spike doesn't take interrupt in the right time

[AyoubJalali]

Hello,
This is an issue of spike (tandem mode), Spike doesn't take interrupt in the right time.
the problem was seen when I disable MIE bit related and there a pending interrupt waiting to be taken, spike doesn't take the interrupt but CVA6 toke it, after debug I understand that spike after every instruction retired sample Mstatus.MIE & MIP & MIE (not cycle accurate), but CVA6 has a pipeline so it know that was a pending interrupt waiting even you disable MIE he took the interrupt anyway.

So the problem here related to spike being a ref model with no pipeline of course, it's a limitation !!

[MikeOpenHWGroup]

Hi @AyoubJalali, this is a well known problem in verification environments that compare results generated by a reference model and an RTL model that use different abstractions of "time". In this case, the Spike ISS uses 'instruction time' and the RTL uses cycle (clock) time. This is not a problem for synchronous operation where 'instruction time' and 'cycle time' are in phase. This is not the case for asynchronous events such as interrupts and debug-requests because the ISS will always "take" the interrupt at instruction retirement time, and this may be different than when the RTL "sees" the interrupt.

There are three classic approaches to this problem:

1. Retime Async Events. This was done for the CV32E40P v1 and was called "Step-and-Compare-2.0" (which is a terrible name). It is well documented here.
2. Forced Synchronization. In this strategy, each time there is an asynchronous event, the reference model (ISS in this case), is forced to match the RTL, and a set of SV Assertions are used to ensure that the RTL has the correct behavior.
3. Multiple Predictions. In this strategy, the reference model produces a list of possible future results, and the test passes if one of the possible outcomes matches the actual outcome from the RTL. This is a common technique, but Spike is not well suited for this.

For the CVA6 (specifically the CV32A60X), we could try either either retiming or forced sync. This would be practical if this is the only known test failure due to async interrupts in CV32A60X regressions.

[zchamski]

By definition, async events can occur "between" the events naturally seen by the ISS, and it is precisely this property of "event occurred after previous commit and before the current one" that needs to be captured. The reference model has therefore to be more than just an ISS: it needs to interleave the usual processing of synchronous events (commits and synchronous exceptions) with the processing of asynchronous ones that occurred since the last synchronous event.

At first sight, the retiming approach seems the most appropriate, with the convention that:

• the RVFI monitor inserts an appropriate representation of all async events since last ISS commit into the event sequence presented to the reference model;
• the reference model processes the async event representation and alters the state of the ISS accordingly prior to stepping it.

[AyoubJalali]

@MikeOpenHWGroup I have some waveform to share if it's enough with some explanations :

case : where the test failed

the instruction 0x3040_1073 where I clear the MIE (csrw mie, zero), but before it's committed (clock cycle before) the CVA6 has a pending interrupt MIE.MTIE == 1 && MIP.MTIP == 1 && mstatus.MIE == 1, so even you disable all MIE bits the interrupt should be taken in this case, but Spike didn't as @zchamski explain why !!

case: where test passed

It's look the same, but in this case before commit the instruction 0x3040_1073 (csrw mie, zero) they was no pending interrupt, in cycle before we had MIE.MTIE == 1 && MIP.MTIP == 0 && mstatus.MIE == 1 so interrupt is taking as Spike does !!

[MikeOpenHWGroup]

Thanks for the detailed explanations @AyoubJalali and @zchamski. This aligns with my understand as well. Let me see if I can put it another way. In this example:

• The CVA6 (specifically, the CV32A60X) sees the interrupt happen one or more clock cycles after the instruction to clear MIE (actually, csrrw x0, mie, x0) is launched, but before it is retired. Therefore, the interrupt is taken.
• Spike launches and retires instructions in zero time and so the instruction to clear MIE retires at the time as the interrupt, so Spike does not take the interrupt.

Is the above correct? If so, we should be able to bind in a couple of SV Assertions to the RTL that explicitly provides coverage to see if this event happens and then to check that the correct behaviour occurred.

[AyoubJalali]

Thanks for the detailed explanations @AyoubJalali and @zchamski. This aligns with my understand as well. Let me see if I can put it another way. In this example:

• The CVA6 (specifically, the CV32A60X) sees the interrupt happen one or more clock cycles after the instruction to clear MIE (actually, csrrw x0, mie, x0) is launched, but before it is retired. Therefore, the interrupt is taken.
• Spike launches and retires instructions in zero time and so the instruction to clear MIE retires at the time as the interrupt, so Spike does not take the interrupt.

Is the above correct? If so, we should be able to bind in a couple of SV Assertions to the RTL that explicitly provides coverage to see if this event happens and then to check that the correct behaviour occurred.

we have SV assertion that provide coverage for that !

[MikeOpenHWGroup]

...what about check for correct behaviour?

[AyoubJalali]

...what about check for correct behaviour?

this is also implemented, normally the assertion failed if it didn't match RTL behavior, also I added some extra check in scoreboard

[zchamski]

Incidentally, Spike already has support for "spontaneous" events coming from the environment: host-to-target communication in (semi-)hosted configurations is implemented as changes in the content of hart memory. Prior to stepping the hart, the simulation engine checks if the memory at address fromhost changed to a non-zero value, and acts according to the value found.

For precise tandem simulation of interrupts we need a similar checker of interrupt notifications coming from the RVFI. In parallel, the current implementation of method Processor::step() (CVV: vendor/riscv/riscv-isa-sim/riscv/Proc.cc) needs a careful review wrt. its handling of incoming event information.

[MikeOpenHWGroup]

For precise tandem simulation of interrupts we need a similar checker of interrupt notifications coming from the RVFI. In parallel, the current implementation of method Processor::step() (CVV: vendor/riscv/riscv-isa-sim/riscv/Proc.cc) needs a careful review wrt. its handling of incoming event information.

I assume that is "future work", and outside the scope of CV32A60X.

[zchamski]

For precise tandem simulation of interrupts we need a similar checker [...]

I assume that is "future work", and outside the scope of CV32A60X.

Given the current workload yes, this is future work which (miracles excluded) will take place after TRL5 wrap-up of CV32A60X.

[JeanRochCoulon]

The gap between Spike and RTL is checked by assertions which are not yet pushed to GitHub.
To me, this issue does not impact cv32a60x TRL-5, but more future developments.
In mid-term, plan is to either:

• Publish these assertions
• Modify Spike