Multi-Core Architecture

[NikLeberg]

Hi there @stnolting,
I'd love to implement and experiment with a multi-core architecture based on your RISC-V core. That would then be a symmetric multiprocessing system or SMP.

Based on the privileged ISA specification there are numerous additional unratified specifications that define how this can be done and the necessary modules for that. For example one spec proposes a CLINT (Core Local Interrupter) or another a CLIC (Core Level Interrupt Controller). Also the FreeRTOS sources mention CLINT/CLIC but basically only use the mtime register used together with mtimecmp to produce the timer interrupt.

Do you know more about the state of RISC-V and SMP? Or do you have an opinion about the unratified specs and maybe some guidance on what version may be the best to implement? I'm asking here because I have noticed that up to the ancient version 1.3.0.0 (25.07.2020) you provided a CLIC module and most likely will have had more experience with the RISC-V specs than me. Maybe you also know how all this can come together.

The RISC-V port of FreeRTOS sadly is only single-core and only uses the memory mapped mtime together with mtimecmp as the timer interrupt. And the inter-processor interrupt functionality (that CLIC, CLINT etc. provides) for SMP is not used. Have you mabe encountered other RTOS that have the SMP already implemented? Zephyr looks like it has but the source code is a bit of a mess. Same for NuttiX.

Thanks,
Nik

[GideonZ]

Dear Nik, Although I am not @stnolting, I thought I'd drop my thoughts on your idea for SMP using the NeoRV32. Thank you for sharing this idea! There are various SMP implementations of Risc-V, for instance the Ox64 from Pine implements two clusters of two Risc-V processors, each cluster is SMP in itself. I don't know the exact details of the BL808 chip, however. What might be interesting to look into is that the FreeRTOS port for the ESP32 WiFi module, found in the (ESP32 SDK) is adapted for SMP, because the ESP32 is a dual core. The initial ESP32 modules are not based on Risc-V, however (it's Xtensa), but an idea of how to add SMP to FreeRTOS is certainly there and fully operational. Since the 'portable' layer of FreeRTOS is really really thin, it shouldn't be much of a hassle to combine the fragments from the Risc-V port and the ESP32 SMP port together! One other thing to keep in mind is that the NeoRV32 is *not* designed for performance. The focus is on size, correctness of handling illegal opcodes and exceptions, debug unit and documentation. I think Stephen did an EXTRAORDINARY job in this!! The reason that I mentioned performance is that you may get the same stuff done with a faster single core. Of course real time properties will be different, so there is definitely a use for having multiple slower cores. Speaking of performance, one of the first things I would address when improving it, would be to ditch the horrendous Wishbone bus. At this point the processor bus is not capable of issuing multiple transactions; each of them have to be finished before the next one can start. This eliminates the possibility to cleverly interleave I and D accesses to external memory. Another significant improvement would be the addition of burst support to these buses. Especially with the cache, I believe this would make a significant difference. Could you confirm @stnolting, that the external bus interface is still the same as some months ago? I might have overlooked some significant improvements in this area. Best regards, Gideon

…

On Mon, Apr 10, 2023 at 7:33 PM NikLeberg ***@***.***> wrote: Hi there @stnolting <https://github.com/stnolting>, I'd love to implement and experiment with a multi-core architecture based on your RISC-V core. That would then be a symmetric multiprocessing system or SMP. Based on the privileged ISA specification there are numerous additional unratified specifications that define how this can be done and the necessary modules for that. For example one spec proposes a CLINT (Core Local Interrupter) or another a CLIC (Core Level Interrupt Controller). Also the FreeRTOS sources mention CLINT/CLIC but basically only use the mtime register used together with mtimecmp to produce the timer interrupt. Do you know more about the state of RISC-V and SMP? Or do you have an opinion about the unratified specs and maybe some guidance on what version may be the best to implement? I'm asking here because I have noticed that up to the ancient version 1.3.0.0 (25.07.2020) you provided a CLIC module and most likely will have had more experience with the RISC-V specs than me. Maybe you also know how all this can come together. The RISC-V port of FreeRTOS sadly is only single-core and only uses the memory mapped mtime together with mtimecmp as the timer interrupt. And the inter-processor interrupt functionality (that CLIC, CLINT etc. provides) for SMP is not used. Have you mabe encountered other RTOS that have the SMP already implemented? Zephyr looks like it has but the source code is a bit of a mess. Same for NuttiX. Thanks, Nik — Reply to this email directly, view it on GitHub <#573>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACUFDSM7LBYUVRARHH6TIJLXAQ76VANCNFSM6AAAAAAWZG5UZQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[NikLeberg]

Hi there @GideonZ,
your thoughts are more than welcome, thanks!

There are various SMP implementations of Risc-V, for instance the Ox64 from
Pine implements two clusters of two Risc-V processors, each cluster is SMP
in itself. I don't know the exact details of the BL808 chip, however.

Interesting chip... SMP with RISC-V is certainly doable. But sadly the most open source cores are either written in Verilog or obscure high level languages like Chisel. I'd like to stay in the realms of VHDL. And there the only really good, and also really good documented, project is NEORV32. At least to my knowledge.

What might be interesting to look into is that the FreeRTOS port for the
ESP32 WiFi module, found in the (ESP32 SDK) is adapted for SMP, because the
ESP32 is a dual core.

There also seems to be an experimental branch of FreeRTOS that incorporates the changes Espressif did and makes them available for other architectures. There is no official porting guide but if I have seen it correctly then FreeRTOS only really needs one MTIME based interrupt on one core, and the rest of the cores with software interrupts (with CLINT for example). Additionally it requires atomic extension..

[stnolting]

Hey @NikLeberg!

A multi-core setup is something I also would like to play around with 😉

The hardware requirements are heavily defined by the actual software that is going to be executed on the cores. Just a few thought (from the RISC-V bubble):

• CLIC, CLINT, PLIC and the recent ACLINT (😅) are intended for rich operating systems like Unix/Linux. Anyway, it would be great to have a minimal PLIC/CLINT/whatever module that could be attached to a cluster of the cores providing spec-compatible interrupt handling. @drgrandios also mentioned this (Some XIRQ documentation oddities #568). Such a module would be nice as it presents a centralized instance for handling
  • MTIME interrupt per-core
  • software interrupt for inter-processor-interrupts
  • centralized handling, routing, prioritizing, .. of external interrupts
• Atomic memory access (A ISA extension) and cache operations (riscv-CMOs) might be relevant for a multi-core setup. Right now NEORV32 does not support any of them. 😅
• Zephyr is the only RTOS I know that also supports some kind of multi-core setup out of the box. But yeah, the code and especially the build system is quite heavy.

---

@GideonZ

What might be interesting to look into is that the FreeRTOS port for the
ESP32 WiFi module, found in the (ESP32 SDK) is adapted for SMP, because the
ESP32 is a dual core.

That's interesting. I was not aware of that. 👍

One other thing to keep in mind is that the NeoRV32 is not designed for
performance. The focus is on size, correctness of handling illegal opcodes
and exceptions, debug unit and documentation. I think Stephen did an
EXTRAORDINARY job in this!!

Haha, thank you 😉

This project cannot compete with other cores like VexRisc when it comes to performance as they use deep pipelines, smart prefetch logic and maybe even out-of-order/speculative execution. So size and especially "execution safety" (i.e. correct, precise and resumable trap handling) are the targeted "niche" of this project.

Speaking of performance, one of the first things I would address when
improving it, would be to ditch the horrendous Wishbone bus.

😆
Yes, this on my todo-list. Now that we have caches for data and instructions we really should add some kind of burst accesses as those bus accesses are the primary performance bottleneck right now.

I like Wishbone because it is damn simple, it allows to terminate pending transfers and it is completely royalty-free. However, AXI Lite is the way to go I think.

[NikLeberg]

A multi-core setup is something I also would like to play around with 😉

I did, and it's awesome 😁. I merely instantiated two cores and individually connected them through an icache and then both of them to one single dcache (using multiple 2:1 neorv32_busswitches). They happily run in the single address space and with the help of software that differentiates them with the hartid I have made some CPU specific gpios / LEDs blinking.

Now all the additional peripherals to synchronize them is required... thanks for the pointers. I think, initially a simple CLINT would be enough. But I really don't like that is uses 48 KB of address space for simply interrupts.

I like Wishbone because it is damn simple, it allows to terminate pending transfers and it is completely royalty-free. However, AXI Lite is the way to go I think.

I too really like Wishbone due to its simplicity. But I must also agree that currently the cache takes twice the time to load a cache line than it theoretically should as there is no burst support. You probably know but there are ways to make bursts with Wishbone possible. I.e. actual pipelined mode or WISHBONE Registered Feedback that uses the tag signals.
I don't like AXI as its super heavy in signal count and complexity... Have you heard of TileLink from SiFive (presentation, specs)? I think it's quite clever and the simplest form is supposed to be very Wishbone like but with the benefit of having bursts. It has additional the support for multi level caches and directly embedded ways to manage the cache coherency. That could become very interesting for the multi-core architecture.

Can the community support you @stnolting in any specific topic regarding the bus?

[stnolting]

I did, and it's awesome 😁. I merely instantiated two cores and individually connected them through an icache and then both of them to one single dcache (using multiple 2:1 neorv32_busswitches). They happily run in the single address space and with the help of software that differentiates them with the hartid I have made some CPU specific gpios / LEDs blinking.

That's cool! Is that open-sourced somewhere? 😉

I think, initially a simple CLINT would be enough. But I really don't like that is uses 48 KB of address space for simply interrupts.

But this would be something that is located "processor-externally", right?

Can the community support you @stnolting in any specific topic regarding the bus?

That's very kind of you. I really appreciate that!

I think we should first think about modifications to the processor-internal bus system so it supports burst transfers. All the peripherals should already support burst accesses (in theory). After that we can check out what else is required to boost the external Wishbone interface.

[NikLeberg]

That's cool! Is that open-sourced somewhere? 😉

Soon. ☺ First I want to replace the multiple levels of 2:1 bus muxes with a custom Wishbone crossbar. My idea is to use the fact that RAM in FPGAs is inherently dual-port and as such want to have dual-port access to the IMEM. So both CPUs could read IMEM at the same time.
I'll put it in my neorv32_soc project that you already have ⭐ed (thanks ❤!).

But this would be something that is located "processor-externally", right?

Yes, no CPU change required. In the toplevel the MTIME also needs to be disabled. Otherwise the timer interrupt of the external CLINT can't be connected. But I'll need to write my own toplevel anyway to instantiate two CPUs.

I think we should first think about modifications to the processor-internal bus system so it supports burst transfers

Do I understand it correctly that the RISC-V ISA has no instruction for memory bursts, right? So that means that the only peripherals that needs bursts are the caches and maybe a future DMA engine?

[stnolting]

I'll put it in my neorv32_soc project

Great! I'm looking forward to that.

Do I understand it correctly that the RISC-V ISA has no instruction for memory bursts, right?

As far as I know things like memory burst are never part of the ISA as they are just another implementation-specific microarchitecture design parameter (like pipelined vs. single-cycle execution).

So that means that the only peripherals that needs bursts are the caches and maybe a future DMA engine?

Right. The caches need to implement burst accesses as well as the external memory interface, the upcoming DMA and maybe also the XIP module.

[tmeissner]

@stnolting

I like Wishbone because it is damn simple, it allows to terminate pending transfers and it is completely royalty-free. However, AXI Lite is the way to go I think.

The Wishbone specification also contains definitions of burst accesses. Full AXI is a whole new level of complexity which is not that easy to implement correctly if you want all the fancy features.

Ah, I see @NikLeberg also answered to that specific topic. Next time I better read the comments correctly before post 😄

IIRC we have a wishbone verification component which implements classic bursts and registered feedback modes from the B3 spec. It was used to verify the wishbone module of NEORV32 during a master thesis of a student who I was in charge of. From my point of few, an enhancement of the current wishbone implementation to burst modes would be the easiest way. I would not remove Wishbone, as most of free IP supports it.

[stnolting]

IIRC we have a wishbone verification component which implements classic bursts and registered feedback modes from the B3 spec. It was used to verify the wishbone module of NEORV32 during a master thesis of a student who I was in charge of. From my point of few, an enhancement of the current wishbone implementation to burst modes would be the easiest way. I would not remove Wishbone, as most of free IP supports it.

👍

I agree. We should keep Wishbone (for now). 😉
I really need to check the Wishbone specs again... Could you provide a short summary what signals we would require to support simple burst (constant address increment)?

[NikLeberg]

I had a closer look at the Wishbone B4 specs. I now think that the registered feedback mode is overkill. It is defined for both pipelined- and classic mode and would introduce two new tag-like signals: CTI_IO() cycle time identifier and BTE_IO() burst type extension. In CTI a value of 000 indicates a classic access whereas 001 means constant address burst and 010 means incrementing address burst. The BTE would only be used for the incementing burst to indicate after how many beats the burst should wrap around to the start address.

Using the real pipelined mode of wishbone may be easier. Instead of asserting STB basically identical to CYC and requiring that the address and data bits do not change, on each clock where STB is asserted, new valid data and address are presented on the bus. The slave is expected to buffer them internally (if he needs wait states) and respond asynchronously with ACKs for each cycle. But now the slave also gets a new output signal STALL with which he tells the master that the current STB cycle will be ignored and the STB must be repeated/kept.
Slaves could always assume an incrementing access and just stall the master if the address is not the expected increment (and load the correct value on the next clock). Or the assumption of the address burst may be made on a per slave basis. A FIFO could assume a constant address access whereas a IMEM could assume incrementing?

The CPU internal bus does not support this directly though... it has no way of saying that a burst of multiple accesses are ongoing. Maybe the introduction of a signal similar to CYC would be necessary? What would you be comfortable to change about the internal bus design @stnolting?

[NikLeberg]

Me again 😊

We had a brief discussion about the removed lr and sc instructions from the A / atomics extension in #308. For multi core system this would be very much necessary. Even if only lr and sc instructions are supported the amos can all be emulated in software based on those. Instead of hacking together my own atomics external to the cpu I'd like to be somewhat standard compliant and add that functionality back to NEROV32.

Would you @stnolting be welcoming to such an pr? If so, I thought of two possible implementations:

• same as you had it: using a lock signal on the external Wishbone interface. If i understand the Wishbone specs correctly then the lock signal is to disable/freeze arbitration on the whole bus. I see this becoming a problem in larger crossbars.
• alternatively additional ports (separate from Wishbone) could be added to the cpu with signals reservation_o, reservation_addr_o and reservation_valid_i or similar. The outputs would be set on a lr call. The external memory system then may implement a reservation system (as you mentioned) that then observes accesses to that address. If no access is observed the input is just reservation_valid_i <= reservation_o. But if another write was observed, then the input gets set to zero.

I'm not certain about the implications of either of these options, that's why I'm asking here. :)

[stnolting]

Hey Nik!

We had a brief discussion about the removed lr and sc instructions from the A / atomics extension in #308. For multi core system this would be very much necessary.

I agree. But also a single-core setup could benefit from this extension. For example, image a RTOS running on the core - or just any interrupt-driven application: If the current thread is interrupted while updating a mutex or semaphore things can get ugly. In a bare-metal setup you can simply disable interrupts inside this critical sections. But what about a RTOS setup? Here, the A extension would be very handy.

Even if only lr and sc instructions are supported the amos can all be emulated in software based on those.

Thanks for the link! As lr and sc would be sufficient, let's focus on these two instructions only. 👍

Would you @stnolting be welcoming to such an pr?

Sure! But I have no idea how to implement this extension. The actual instruction decoding, address generation and stuff like that would be no problem (we already had that). But how to implement those reservation sets?

Btw, at first I would recommend to focus on the single-core processor level only. As mentioned above, atomic accesses would also be nice add-on for the default single-core setup.

same as you had it: using a lock signal

Locking the bus is quite easy to implement but I do see some problems here. If lr basically locks the bus so no other instance can access it before sc releases the bus again, the CPU might starve/deadlock if it cannot fetch further instructions before reaching the releasing sc instruction.

alternatively additional ports (separate from Wishbone) could be added to the cpu with signals reservation_o, reservation_addr_o and reservation_valid_i or similar. The outputs would be set on a lr call. The external memory system then may implement a reservation system (as you mentioned) that then observes accesses to that address. If no access is observed the input is just reservation_valid_i <= reservation_o. But if another write was observed, then the input gets set to zero.

lr: No need for an additional reservation_addr_o signal - we could use the general access address signal. If reservation_set_o is set during a load operation this should "set" (?) the reservation for the current address.

sc: If reservation_clr_o is set during a store operation this should "remove" (?) the reservation for the current address. This operation should also return some status signal to the core to check if the reservation was still intact before this.

Interrupt/exceptions/trap/you-name-it should also clear all outstanding reservations, right? The RISC-V A spec is way too high-level for me... 😅

Anyway, how would you/we/one implement this for the current single-core processor setup? A new module attached to the bus that is responsible for managing these reservations?

[NikLeberg]

Btw, at first I would recommend to focus on the single-core processor level only.

Sure, no problem. I'll consider solutions that are extendable for multi cores tough.

Locking the bus is quite easy to implement but I do see some problems here. [...] the CPU might starve/deadlock [...]

That's a very good point! I haven't even thought about the deadlock. My SMP playground system uses a crossbar which relaxes this quite a bit (concurrent IMEM an DMEM access).

No need for an additional reservation_addr_o signal - we could use the general access address signal.

My mistake, I was somehow thinking about the neorv32_top entity when writing and wanted to keep wishbone and the reservation system apart from each other. But of course we would implement it inside the toplevel i.e. on the neorv32_cpu entity level where we have d_bus_addr_o.

This assumes that only modules within neorv32_top are ever going to write to memory though! If the DMEM is accessed over external Wishbone interface then we can't guarantee this as other external masters may exist which we have no clue about. But to keep it simple we could focus on the internal system first.

This is how I would implement it:

• on execution of a lr the cpu sets the flag reservation_set_o and d_bus_addr_o contains the reserved address
• the reservation module registers the address internally
• cpu continues to work through instructions but keeps the flag active. It only resets the flag if [...] there is an intervening context switch on this hart, or if in the meantime the hart executed a privileged exception-return instruction. (taken from the unprivileged specs v2.2, page 41)
• in the meantime the reservation module snoops on the bus (i.e. in front of the dcache) for any access on that internally registered address
• as long as the module did not notice any access it routes (via registers?) the reservation_set_o flag back to an reservation_valid_i input of the cpu, if it did, the valid flag will be set to 0
• eventually the cpu will execute sc, it can simply check weather or not the valid flag is set
• as the cpu is now writing to the reserved address a last the module will reset the valid flag now

I see this working great on single core systems. For multi core systems though there is a small window between reading the valid flag and actually writing to the memory. You know the cpu by heart, how many cycles would such an sc need? You mention in the datasheet to require 4 execution cycles for a load/store. I assume it's identical for this? As what stage in these four cycles is the bus request triggered? the earlier the better...

[stnolting]

This assumes that only modules within neorv32_top are ever going to write to memory though! If the DMEM is accessed over external Wishbone interface then we can't guarantee this as other external masters may exist which we have no clue about. But to keep it simple we could focus on the internal system first.

Correct. The processor-internal memories and peripherals can and should only be accessible by the CPU core and by the (hopefully) upcoming DMA controller (-> #593). Accesses from "outside" are not planned. So if you need - for example - the IMEM to be accessible by several CPU cores you'll need to implement it outside of the default processor setup.

My SMP playground system uses a crossbar which relaxes this quite a bit (concurrent IMEM an DMEM access).

By the way, I never thought about a use case were only the actual NEORV32 CPU is instantiated into custom designs. 😅 My idea was to strip down the entire processor (using the generics) so only the core + basic bus infrastructure remains giving you a core complex with a single Wishbone interface (we did this for the LiteX Core Complex).

This is how I would implement it:

• on execution of a lr the cpu sets the flag reservation_set_o and d_bus_addr_o contains the reserved address

👍 This should be quite easy to implement.

• the reservation module registers the address internally

👍 For the single-core NEORV32 processor setup this reservation module could be implemented within the BUS KEEPER as this module is already responsible to monitor all bus accesses.

• cpu continues to work through instructions but keeps the flag active. It only resets the flag if [...] there is an intervening context switch on this hart, or if in the meantime the hart executed a privileged exception-return instruction. (taken from the unprivileged specs v2.2, page 41)

So when do we invalidate/remove a reservation? If sc is executed the reservation of the associated address is removed (at least this is how I understand the spec). But what about all those traps? Do we only remove the reservation when executing mret (return from trap handler) or do we invalidate them already when a trap handler is entered? The latter approach seems to be more straightforward. In this case the trap handler code cannot compromise any "pending" reservation (e.g. mutex access from inside am interrupt handler).

• in the meantime the reservation module snoops on the bus (i.e. in front of the dcache) for any access on that internally registered address

👍 What about instruction fetch access? Do they also alter any reservation states? I think I did not see anything about this in the latest version of the spec...

• eventually the cpu will execute sc, it can simply check weather or not the valid flag is set

👍

• as the cpu is now writing to the reserved address a last the module will reset the valid flag now

Same question as above: when to clear the reservation? Only on sc (and trapping) or also on "normal" load/store operation to the "reservated" address?

Another question: how many parallel reservation would we support? Just a single one?

I see this working great on single core systems. For multi core systems though there is a small window between reading the valid flag and actually writing to the memory.

I think this is something the memory system should take care of.

You know the cpu by heart, how many cycles would such an sc need? You mention in the datasheet to require 4 execution cycles for a load/store. I assume it's identical for this? As what stage in these four cycles is the bus request triggered? the earlier the better...

Basically, lr and sc would be handled as normal load/store operations. I would not worry too much about the latency of the actual bus request signals now. At first we need to find a general approach how to implement all this 😉

And yes, we should propagate those reservation signals (somehow) via the external bus interface.

[NikLeberg]

could be implemented within the BUS KEEPER

Makes sense!

Do we only remove the reservation when executing mret (return from trap handler) or do we invalidate them already when a trap handler is entered?

That part in the specs makes not much sense to me... I think it's only possible (or sensical?) to reach a mret after having done a context switch in the first place i.e. exeption / interrupt? Is it not? If it's too complicated to reset the flag in both cases I'd prefer it be reset by the context switch.

What about instruction fetch access? Do they also alter any reservation states? I think I did not see anything about this in the latest
version of the spec...

I don't read anything about that as well. But as long as the instruction reads are not on the same address the registration will still be valid. Besides those will be only reads. The reservation is only invalidated on writes.

Same question as above: when to clear the reservation? Only on sc (and trapping) or also on "normal" load/store operation to the "reservated" address?

Normal access i.e. a store must clear the reservation for sure. This case might be best handled by the reservation module i.e. BUS KEEPER. It observes a store to the reserved address and resets the flag. No need to handle that in the CPU itself.

Another question: how many parallel reservation would we support? Just a single one?

Thats entirely platform defined. The specs state multiple LR reservations might be active simultaneously for a single hart. which gives us the freedom to do as we like. I think we only really need one possible active reservation.

Just a thought: The specs describe a thing they call guarantee of eventual forward progress which basically means that in between a lr/scpair only a maximum of 16 _I_ extension instructions are allowed excluding jumps/branches and regular load/stores. The footnote states that this is defined such that the instructions may entirely reside on the same cache line and needn't be fetched from IMEM. Assuming NEORV32 is configured with an icache this would then give the rather _simple_ option to lock the bus for the duration of thelr/sc` pair. This could be simpler hardware wise but will create complications in software (the pairs must be aligned such that the fit in one cache line, A extension cannot be used without enabled icache, and so on.)
So this then leaves the option to thing about locking the bus again. 🤷‍♂️

By the way, I never thought about a use case were only the actual NEORV32 CPU is instantiated into custom designs. 😅 My idea was to strip down the entire processor (using the generics) so only the core + basic bus infrastructure remains giving you a core complex with a single Wishbone interface (we did this for the LiteX Core Complex).

I figured. 😊 But I did not see that going well because then each core complex would have it's own little internal memory space and a bigger shared external memory space. And especially IO modules could then not be shared with all cores.
It works for now.. 😋

[stnolting]

That part in the specs makes not much sense to me... I think it's only possible (or sensical?) to reach a mret after having done a context switch in the first place i.e. exeption / interrupt? Is it not? If it's too complicated to reset the flag in both cases I'd prefer it be reset by the context switch.

I have not read the spec again, but I would think that any outstanding reservations should be removed when starting a trap (handler) and also when leaving a trap (handler). 🤔 Seems liek I have to read the specs again.

I don't read anything about that as well. But as long as the instruction reads are not on the same address the registration will still be valid. Besides those will be only reads. The reservation is only invalidated on writes.

You are right. I did not think about this.

Normal access i.e. a store must clear the reservation for sure. This case might be best handled by the reservation module i.e. BUS KEEPER. It observes a store to the reserved address and resets the flag. No need to handle that in the CPU itself.

👍

Thats entirely platform defined. The specs state multiple LR reservations might be active simultaneously for a single hart. which gives us the freedom to do as we like. I think we only really need one possible active reservation.

Ok, so we should start with a single reservation set. Maybe we could add a generic or package constant to configure the maximum number of reservation sets.

Just a thought: The specs describe a thing they call guarantee of eventual forward progress

Yes, I have seen that but then my brain refused to continue reading 😅

We cannot lock the data bus for a certain number of instructions after executing lr. In this scenario the instruction fetch bus would not be able to get further instruction (potentially including the final sc) leading to a deadlock.

Does this forward progress means that active reservation automatically clear after N instructions? If this is the case, we could simplify this and clear a reservation after a simple cycle counter times out. 🤔

I figured. 😊 But I did not see that going well because then each core complex would have it's own little internal memory space and a bigger shared external memory space. And especially IO modules could then not be shared with all cores.
It works for now.. 😋

If you disabled all internal peripherals and memories the only things that are left are

• the CPU
• the bus infrastructure including the bus keeper
• the SYSINFO module
• and the Wishbone interface

In this case the entire address space is available for external components (except for the internal IO space, but that is just a really tiny fraction).

[stnolting]

To add support for the RISC-V A extension the CPU's data interface needs to be quipped with additional signals:

d_bus_rvset_o : out std_ulogic; -- set reservation for current address
d_bus_rvclr_o : out std_ulogic; -- clear reservation for current address
d_bus_rvrst_o : out std_ulogic; -- clear all reservations
d_bus_rvsta_i : in  std_ulogic; -- reservation state for current address

Adding just the lr and sc instruction to the execution unit should be no big deal. The entire extension should be configurable by a new generic CPU_EXTENSION_RISCV_A.

We would add this hardware immediately keeping it deactivated for now so we can further experiment with it.

What would be the required additions for the Wishbone interface? Is there something spec-defined for this purpose or do we have to implement those additional signals as custom tags?

[stnolting]

Hey @NikLeberg!

Currently, I am redesigning the processor's bus infrastructure replacing all the bus signals by two custom interface type:

  -- Internal Bus Interface: Request --------------------------------------------------------
  -- -------------------------------------------------------------------------------------------
  type bus_req_t is record
    addr : std_ulogic_vector(31 downto 0); -- access address
    data : std_ulogic_vector(31 downto 0); -- write data
    ben  : std_ulogic_vector(03 downto 0); -- byte enable
    we   : std_ulogic; -- write request (single-shot)
    re   : std_ulogic; -- read request (single-shot)
    src  : std_ulogic; -- access source (1=instruction fetch, 0=data access)
    priv : std_ulogic; -- set if privileged (machine-mode) access
  end record;

  -- Internal Bus Interface: Response -------------------------------------------------------
  -- -------------------------------------------------------------------------------------------
  type bus_rsp_t is record
    data : std_ulogic_vector(31 downto 0); -- read data
    ack  : std_ulogic; -- access acknowledge (single-shot)
    err  : std_ulogic; -- access error (single-shot)
  end record;

I hope this will make the bus structure a little bit cleaner. But for sure it will simplify the "wiring" overhead! 😉

I terms of the LR/SC implementation I think a single new (2-bit) signal added to the bus_req_t buses might be sufficient:

amo_lrsc

• 00 - do nothing
• 01 - set reservation for current address
• 10 - clear reservation for current address
• 11 - clear all reservations

As we have to route this signal through the d-cache and also through the bus muxes a simple approach is highly preferred 😅

I think we do not need an additional signal to return the reservation state back to the CPU (when executing sc). sw is basically just a store instruction, so all peripherals/memories should keep outputting all-zero on their read data buses. Hence, we could also use this standard read data mechanism to return the reservation state back to the CPU. What do you think about this?

edit

-> #607

[NikLeberg]

Currently, I am redesigning the processor's bus infrastructure replacing all the bus signals by two custom interface type:

Very nice! (And I love VHDL too 😄.) This makes it definitely cleaner.

I terms of the LR/SC implementation I think a single new (2-bit) signal added to the bus_req_t buses might be sufficient

Seems reasonable. Compared to the old approach of d_bus_rvset_o and d_bus_rvclr_o we have with this two bit approach the flexibility to support multiple concurrent reservations.

I think we do not need an additional signal to return the reservation state back to the CPU (when executing sc). sw [you meant sc, right?] is basically just a store instruction, so all peripherals/memories should keep outputting all-zero on their read data buses. Hence, we could also use this standard read data mechanism to return the reservation state back to the CPU. What do you think about this?

Hmm.. I don't think it's that easy. I agree that sc is basically just a sw and we shall reuse the same mechanisms. But the store is, well, conditional. So the cpu must be able to decide beforehand if it is allowed to trigger the write at all or not. How would that work if there is not an additional status signal?

Would we be better of when we use the err signal?

Although I like how simple the cpu side of the implementation could get if we do it the way you describe it. We could require the external reservation system to return the reservation state (either over the read data bus or err) when an sc is issued. The reservation system would also need to block the write to addresses that fail the conditional. So instead of the cpu not triggering the write, the cpu always triggers it but the res system blocks the write and returns the state over the read bus (or flags an error state)? Of course only on invalid reservations. For this purpose the res system could interpret the 10 state of the amo_lrsc instead of clear reservation as test & clear reservation.
What do you think?

I tried to implement the lr and sc as discussed earlier and based on your old implementation but failed miserably 😅. The cpu often would spontaneously stop running or do other funky stuff.
I'll try again after you have merged your soc_bus pr and in the way now envisioned.

[stnolting]

Hmm.. I don't think it's that easy. I agree that sc is basically just a sw and we shall reuse the same mechanisms. But the store is, well, conditional. So the cpu must be able to decide beforehand if it is allowed to trigger the write at all or not. How would that work if there is not an additional status signal?

You are right... I keep forgetting about this 🙈
So we do need an additional feedback signal, right? And this signal has to be checked by the CPU before issuing the actual memory write request?! 🤔

Otherwise, the memory system itself would have to manage the conditional part of sc. For the processor-internal memory system this would be feasible, but as soon as we are starting to propagate those atomic operation via the Wishbone interface we'll get some problem with this approach...

Would we be better of when we use the err signal?

I think that would violate the A specs. Setting the err signal will cause an exception, which is not intended by the RISC-V spec.

I tried to implement the lr and sc as discussed earlier and based on your old implementation but failed miserably 😅.

Oh dear 😅 Can I help you out somehow?

[NikLeberg]

Before we discuss the implementation further: If have read the specs again, which state: An SC must fail if there is another SC (to any address) between the LR and the SC in program order. So my initial interpretation that multiple lr/sc can be in flight at the same time was wrong, sorry about that. Executing an sc invalidates every previous lr (except the most recent lr).

So two additional signals in the bus_req_t structure should be enough. For example:

rvset : std_ulogic; -- set reservation for current address
rvclr : std_ulogic; -- test & clear reservation for current address

For the bus_rsp_t it depends on our decisions:

So we do need an additional feedback signal, right? And this signal has to be checked by the CPU before issuing the actual memory write request?! 🤔

Option the CPU does the thing then: Yes and yes.
Option the Memory system does the thing then: Maybe and no.

Option CPU:
Extending bus_rsp_t with an rvsta : std_ulogic; -- reservation state for current address could be enough. Signal shall be high as long as the current reservation is valid. (The simplest implementation setting it globally with rvset and resetting it with rvclr.) The CPU must check this signal before issuing the write.

Option Memory System:
As atomics are fundamentally very connected to the memory system I see no apparent problem in delegating it to the memory system.
The specs have a note about it (see note: More complex implementations might also implement AMOs at memory controllers) or some other processors have it (see AXI docs). Where it's basically implemented as memory near special ALUs. The CPU then may just issue a normal write and be notified of the outcome in some way.

We could have the rvsta signal but now instead of being active for the duration of the reservation it flags the state only while the ack in the response is set. So a request with rvclr set and a response with ack + rvsta would be a successful sc. A response with only an ack a failed one.
But here the usage of err could be a bit more elegant.

Setting the err signal will cause an exception, which is not intended by the RISC-V spec.

Is this fixed in the neorv32 design? Could we react differently to the bus error when it happens during the execution of an sc?

A lot of options... 😅

Oh dear 😅 Can I help you out somehow?

I think I can manage it. Again, the very excellent documentation, especially the cpu architecture diagram is tremendously helpful.
But don't expect anything super fast. If you want to get it done fast, then feel free. 🤗

[stnolting]

This is even more complicated than I thought 😅

Before we discuss the implementation further: If have read the specs again, which state: An SC must fail if there is another SC (to any address) between the LR and the SC in program order. So my initial interpretation that multiple lr/sc can be in flight at the same time was wrong, sorry about that. Executing an sc invalidates every previous lr (except the most recent lr).

So a single reservation set is all we need (and want) - at least for the processor-only setup 👍

Option Memory System:

From a functional point of view, this seems to be the easier solution. Plus we would be able to add AMO/read-modify-write operations (in the future). However, this would require another instance in the bus system as we need to "intercept" store requests to make them conditional (propagate the write enable signal to the memory only if the reservation is still valid (for sc.w)):

CPU -> (d-cache) -> bus-mux_0 -> busmux_1 -> atomic_handler -> actual bus -> memories, peripherals, ...

Problem here: what about the d-cache? 🙈
The cache would have to be aware of the current reservation's address in order to make a direct memory access without caching.

Option CPU:

From a hardware point of view the actual core logic wouldn't be too complex (nice example implementation -> https://github.com/munetomo-maruyama/mmRISC-1/blob/main/verilog/cpu/cpu_datapath.v#L736).

But I see some issues here. I think we would need a way to lock the bus system so the CPU executing the sc.w instruction has exclusive access. Why? Well, the CPU needs several cycles to complete such an instruction and we need to make sure that the reservation state does not change within this time. So the CPU would have to

1. lock the bus
2. sample the state of the reservation; if it is still valid the CPU issues a write request
3. release the lock after the CPU receives the ACK from the memory system

Without the lock there might be the change that - for instance - a DMA access breaks the reservation after the CPU has checked the reservations state signal but before the CPU has issues the write request.

Both concepts have their pros and cons so I am not sure which way to go. What so you think?

[stnolting]

For the records: atomic memory accesses / LR/SC instructions were implemented in #651.

[NikLeberg]

Super cool stuff!
Finally having the time for playing around again. Sorry for the dead silence... had to do my bachelor's thesis 😊.

I like how you have implemented it by gating the write request by the reservation station. My crooked implementation tried that differently directly in the cpu and the timing sequence got complicated.
With this in place I'm likely going back on track towards SMP.

I followed neorv32s progress closely and noticed that the internal bus was very much remodelled. Besides the obvious benefits, were there any long term goals in doing so? For example pipelining?

[stnolting]

Sorry for the dead silence... had to do my bachelor's thesis 😊.

No worries (and congrats 👍)!

I like how you have implemented it by gating the write request by the reservation station. My crooked implementation tried that differently directly in the cpu and the timing sequence got complicated.

Well, the basic idea evolved during our conversion. 😉

With this in place I'm likely going back on track towards SMP.

Cool! Keep me updated!
There is another user thinking about a multi-core setup: -> #644

I followed neorv32s progress closely and noticed that the internal bus was very much remodelled. Besides the obvious benefits, were there any long term goals in doing so? For example pipelining?

The bus itself did not really change - we just added another signal for identifying reservation set operations. All the other modifications were targeting the bus infrastructure. Now we are using a simplified and centralized address decoding that allows much easier customization (hopefully).

Pipelined/burst transfer are still not supported yet. 🙈 But this is something having high(er) priority now ... 😉

[NikLeberg]

Well hello @stnolting

Just letting you know, that some progress has been made:

• added memory mapped CLINT to handle msi interrupts (besides mtime and mti), allows to implement inter processor interrupts
  • If you can, I'd love if you would give some feedback on the implementation. It requires quite some LUTs (~300) and you seem to be an expert in reducing LUT usage. 😉
• adapted crt0.S to keep secondary harts (mhartid != 0) in an wfi loop until awakened by ipi
  • the secondary harts get a very bogus stack pointer, that is something I'll have to fix
• in the main.c all functionality comes together:
  • primary hart enables the other ones with ipi
  • each hart runs the same main function that blinks an LED on the neorv32_gpio
  • access to the gpio is guarded by an spinlock implemented thanks to your A extension.

There is a lot more to do but the hardware (except coherent caching) would now be ready for SMP. Now the software is missing. Even though FreeRTOS has a SMP branch, that one has not been implemented for the risc-v architecture so far. I'll probably go with zephyr as there SMP on risc-v is already existent (and even a neorv32 board implementation! 😊).

In the meantime I want to extend the dm debug module to support multiple harts. As you must have read the specification very carefully when you implemented it yourself, you maybe can answer me this question: Not every hart needs a dm right? A single dm can handle multiple harts i.e. reset and halt them? OpenOCD seems to support this. And except when one implements the hart array mode only one hart actually runs the debug program buffer the rest stays in the park loop? At least that is how I understand the specification...

[NikLeberg]

Related: Do I understand this correctly that the dm registers that the CPU can access are completely custom definer. I.e. no specs about their layout and functionality exists as this is entirely platform dependant, right?

[stnolting]

Wow! This is so cool!!

I just had a quick look at your code and it looks really good! Just one comment request for the CLIC: how about adding a generic to allow customization of the module's base address (I would like to use that module for a test setup over here 😅)?

I'll probably go with zephyr as there SMP on risc-v is already existent (and even a neorv32 board implementation! 😊).

Digging deeper into Zephyr is still on my to-do list... But it is nice to know that it supports SMP out of the box.

As you must have read the specification very carefully when you implemented it yourself, you maybe can answer me this question:

To be honest, the RISC-V debug spec. - even if written very well - is one of the most complicated specs I have ever read (at least inside the RISC-V universe). 🙈

Not every hart needs a dm right? A single dm can handle multiple harts i.e. reset and halt them? OpenOCD seems to support this. And except when one implements the hart array mode only one hart actually runs the debug program buffer the rest stays in the park loop? At least that is how I understand the specification...

You are right. Basically there are two ways to add debug support if you have multiple harts in your system:

• Add a DM for each hart and just daisy-chain the JTAG lines. openOCD will show a new tap for every core that can be accessed from GDB via unique ports.
• According to the debug spec a single DM can control up to 2^20 harts at the same time. Right now, this feature is not implemented by the NEORV32 DM. The dmcontrol register does not implement the hartsel bits and the dmstatus register only implements a single hart for the all* and any* status bits. Furthermore, the park loop code is designed for the a single hart only. But I think it shouldn't be too hard to extend this for a configurable number of harts.

Related: Do I understand this correctly that the dm registers that the CPU can access are completely custom definer. I.e. no specs about their layout and functionality exists as this is entirely platform dependant, right?

That's right. The interface between CPU and DM (at least on the hardware level) is entirely platform-specific.

[NikLeberg]

Just one comment request for the CLIC: how about adding a generic to allow customization of the module's base address

Not required, its just plain Wishbone and reacts as soon as the stb selection strobe is asserted. In my system the address decoding is external to the io modules (same as in neorv32 since a few commits ago). See my wb_crossbar and wb_mux modules that do the coarse decoding based on a wb_map_t memory map. The CLINT then internally decodes its 4 KB of memory it is claiming (which is huge but defined in the standard).

(I would like to use that module for a test setup over here 😅)?

Is there a repo somewhere? 😉

• Add a DM for each hart and just daisy-chain the JTAG lines. openOCD will show a new tap for every core that can be accessed from GDB via unique ports.

I think I have read that with that aproach two individual gdb sessions need to be started.

• According to the debug spec a single DM can control up to 2^20 harts at the same time. Right now, this feature is not implemented by the NEORV32 DM. The dmcontrol register does not implement the hartsel bits and the dmstatus register only implements a single hart for the all* and any* status bits. Furthermore, the park loop code is designed for the a single hart only. But I think it shouldn't be too hard to extend this for a configurable number of harts.

With this and an additional target <something> -rtos hwthread openocd supports the multiple harts with one single gdb debugger. See here.

That's right. The interface between CPU and DM (at least on the hardware level) is entirely platform-specific.

Thats perfect! Thanks for clarifiying. Then I know what to do! 😋

[stnolting]

Not required, its just plain Wishbone and reacts as soon as the stb selection strobe is asserted. In my system the address decoding is external to the io modules (same as in neorv32 since a few commits ago). See my wb_crossbar and wb_mux modules that do the coarse decoding based on a wb_map_t memory map. The CLINT then internally decodes its 4 KB of memory it is claiming (which is huge but defined in the standard).

Makes sense. 😅

Is there a repo somewhere?

Not yet. I am experimenting with something similar to ARM's bigLITTLE architecture. Time will show if this is something I can present on GitHub 😉

I think I have read that with that aproach two individual gdb sessions need to be started.

I do not like the concept of having individual DMs for each core as the park loop would require adaption for each DM (sine it is mapped to a different base address). A DM that supports a configurable number of harts would be nice to have, but I do not have any (good) idea how to implement that yet...

[NikLeberg]

Oh man, I actually can't believe it. It works 😋

Info : Examined RISC-V core; found 4 harts
Info :  hart 0: XLEN=32, misa=0x40801101
Info :  hart 1: currently disabled
Info :  hart 2: currently disabled
Info :  hart 3: currently disabled
Info : datacount=1 progbufsize=2

(gdb) info threads
  Id   Target Id                                                      Frame 
* 1    Thread 1 (Name: cycloneive.neorv32_cpu0, state: debug-request) 0x0000020c in delay_ms (time_ms=time_ms@entry=128) at src/main.c:30
  2    Thread 2 (Name: cycloneive.neorv32_cpu1, state: debug-request) 0x0000020c in delay_ms (time_ms=time_ms@entry=256) at src/main.c:30
  3    Thread 3 (Name: cycloneive.neorv32_cpu2, state: debug-request) __crt0_cpu_secondary_sleep () at ../lib/neorv32/sw/common/crt0.S:279
  4    Thread 4 (Name: cycloneive.neorv32_cpu3, state: debug-request) __crt0_cpu_secondary_sleep () at ../lib/neorv32/sw/common/crt0.S:279

I extended the DCI interface, adapted the park-loop to be hart aware, and implemented the hartsel selection in the DM.
The DM can be found here, and the ocd park-loop is here. Additionally there is a testbench that does DMI transactions more or less the same way OpenOCD does to examine harts.

A problem that I had was that if more than two harts were halted and as such executing the park loop they will hog the bus and the other harts have no chance in accessing the DM. At least in my bus system with fixed priorities (I have to change it to round-robin at some time...). As temporary fix I forced the i$ to cache the park-loop, which is usually not done as it is in the IO memory space.

I have to say @stnolting I really like how you implemented the DM! It is super intuitive, simplistic but ingenious. I now bolted a few more bits to it 😉.

I do not have any (good) idea how to implement that yet...

How would you have done it? There may be a lot of optimizations possible.

[stnolting]

Hey @NikLeberg!
This is so amazing! Really great work!! 👍

I thought adding multi-core support to the DM would require far more bits to be implemented but your DM has proven me wrong 😅 I would really like to add at least a dual-core setup for the base core. And with your work this should not be a big deal at all! Again, awesome work!!

How would you have done it? There may be a lot of optimizations possible.

I'll have a closer look at your code. But so far it really looks great!

[NikLeberg]

Revisiting my SMP shenanigans...

I've seen that now the SYSINFO module can be disabled. This triggered me to try out what you suggested long ago:

instead of instantiating multiple neorv32_cpu directly, use the neorv32_top with pretty much everything disabled to build a multi core system

I'm paraphrasing, as I could not find the relevant answer that I think to remember you @stnolting have given somewhere.

But this is not feasable currently. Mainly because while enabling the A extension, one simultaneously enables the extension in the CPU but also the generation of the neorv32_bus_reservation_set. The idea was to use the neorv32_top with everything but the i$ disabled and then instantiate that multiple times for an SMP system. The atomics needs to be enabled as well as it is very important in multi core. This leavess us with one reservation stations per hart which makes lr/sc non functional.

Solving the above problem could be as simple as allowing for yet another generic paramerter to individually disable the reservation station. To as complex as to modularize (put in entities) the contents of the neorv32_top which could then be used more efficiently in multi core setups. What do you think?

You mentioned above:

Not yet. I am experimenting with something similar to ARM's bigLITTLE architecture. Time will show if this is something I can present on GitHub 😉

Did something come of it? Is is on github? 😃 Or what are the design considerations you did there?

[NikLeberg]

Or even worse: what if the core runs out of prefetched instructions right before executing sc(note that the XBUS interface is used for accessing data and for fetching instructions) right before executing sc?

Oh, you are absolutely right... It worked on my SMP system because I have separated the I- and D-bus, so the D-bus could do whatever it likes to do while instructions could still be fetched. I forgot. Thanks for reminding me why I did it that way. 🙃

Can we just keep cyc high all the time? And can we trigger further accesses without de-asserting cyc? Maybe the Wishbone lock signal would be better suited.

Wishbone allows cyc to be high for as long as we want as long as we access the same slave / address range. Keeping cyc high has (in most designs) the effect of simply locking the arbitration of the current slave.

[stnolting]

Wishbone allows cyc to be high for as long as we want as long as we access the same slave / address range.

Unfortunately, this is something we cannot guarantee - especially when context switches kick in 🙈

The only solution I see right now is to provide the reservation set "snooping interface" as new top entity ports. 🤔

[NikLeberg]

Could you share how you did atomics it in your multi-core setup?

I first implemented something like a snooping system. That required quite a large bit of logic as for N cores N^2 address comparisons need to be made (sure that could be optimized and shared but it still required quite a bit).

I then came up with another way of doing it. If we multiplex i and d transactions through the xbus gateway, we cannot lock the wishbone bus with cyc as it would stall the cpu. So what if, we assume we can lock the bus, and if another (instruction) accesss is done, we free the bus and fail the later sc operation. If we have a large enough i-cache (minimum 32 words), it will eventually contain the instruction sequence from lr until sc and the operation can fully succeed.

It seems to work. :) I have to test all the edge-cases, but so far it is promising.

[stnolting]

That doesn't sound bad at first - even if I haven't quite understood the exact concept yet... 😅

So we start locking the (x)bus on lr and release the lock on sc, right? If there is an instruction fetch request in between, the lock is remove and the instruction request gets access to the (x)bus. This also means that the upcoming sc has to fail. Hence, your bus switch/gateway needs to remember the address of the original lr and also that the reservation has failed (due to the "interleaved" instruction fetch, correct? 🤔

[NikLeberg]

Yes, correct!

It sort of combines your reservastion station with the xbus gateway and generates an appropriate cyc signal that is kept asserted during lr/sc sequences to ensure atomicity. But sure, the address is checked too, otherwise any lr/sc would pass.