attempt at listing some system rules with examples (#401)

Something which came up in RVI discussions is how to build CHERI
systems, so this section adds some system guidelines.

---------

Signed-off-by: Andres Amaya Garcia <[email protected]>
Co-authored-by: Andres Amaya Garcia <[email protected]>
Co-authored-by: Alexander Richardson <[email protected]>

Author: 3 people

src/img/large_cheri_purecap_system.drawio.png

@@ -64,6 +64,8 @@ include::tid-ext.adoc[]
 
 include::instructions.adoc[]
 
+include::system.adoc[]
+
 include::tables.adoc[]
 
 ///////////////////////////////////////////////////////////////////////////////

src/img/large_cheri_system.drawio.png

@@ -30,10 +30,36 @@ @TechReport{cheri-v9-spec
   institution =  {University of Cambridge, Computer Laboratory},
   doi = 	 {10.48456/tr-987},
   issn = 	 {1476-2986},
-  issn = 	 {1476-2986},
   number = 	 {UCAM-CL-TR-987}
 }
 
+@misc{tagged-memory,
+  author = {
+            Alexandre Joannou,
+            Jonathan Woodruff,
+            Robert Kovacsics,
+            Simon W. Moore,
+            Alex Bradbury,
+            Hongyan Xia,
+            Robert N. M. Watson,
+            David Chisnall,
+            Michael Roe,
+            Brooks Davis,
+            Edward Napierala,
+            John Baldwin,
+            Khilan Gudka,
+            Peter G. Neumann,
+            Alfredo Mazzinghi,
+            Alex Richardson,
+            Stacey Son,
+            A. Theodore Markettos},
+
+  title = {Efficient Tagged Memory},
+  year = 	 2017,
+  month =  nov,
+  url = 	 {https://www.cl.cam.ac.uk/research/security/ctsrd/pdfs/201711-iccd2017-efficient-tags.pdf},
+}
+
 @misc{riscv-priv-spec,
   author = {RISC-V},
   title = {RISC-V Privileged Specification},

src/img/small_cheri_system.drawio.png

@@ -1107,6 +1107,7 @@ CHERI adds architectural guarantees that can prove to be microarchitecturally us
 Speculative-execution attacks can -- among other factors -- rely on instructions that fail CHERI permission checks not to take effect.
 When implementing any of the extensions proposed here, microarchitects need to carefully consider the interaction of late-exception raising and side-channel attacks.
 
+[#section_pma]
 === Physical Memory Attributes (PMA)
 
 Typically, the entire memory space need not support tagged data. Therefore, it

src/riscv-cheri.adoc

@@ -0,0 +1,81 @@
+[appendix]
+== CHERI System Implications
+
+CHERI processors need memory systems which support the capability validity tags in memory.
+
+There are, or will soon be, a wide range of CHERI systems in existence from tiny IoT devices up to server chips.
+
+There are two types of bus connections used in chips which contain CHERI CPUs:
+
+. Tag-aware busses, where the bus protocol is extended to carry the tag along with the data.  This is typically done using a user defined bit in the protocol.
+.. These busses will read tags from memory (if tags are present in the target memory) and return them to the requestor.
+.. These busses will write the tag to memory as an extension of the data write.
+. Non-tag aware busses, i.e. normal non-CHERI aware busses.
+.. Reads of tagged memory will not read the tag.
+.. Writes to tagged memory will clear the tag of any CLEN-aligned CLEN-wide memory location where any byte matches the memory write.
+
+The fundamental rule for any CHERI system is that the tag and data are always accessed atomically. For every naturally aligned CLEN-wide memory location, it must never be possible to:
+
+. Update any data bytes without also writing the tag
+.. This implies clearing the tag if a non-CHERI aware bus master overwrites a capability in memory
+. Read a tagged value with mismatched (stale or newer) data
+. Set the tag without also writing the data.
+
+NOTE: Clearing tags in memory does not necessarily require updating the associated data.
+
+=== Small CHERI system example
+
+[#small_cheri_system]
+.Example small CHERI system with local capability tag storage
+image::small_cheri_system.drawio.png[width=80%,align=center]
+
+This example shows a minimum sized system where only the local memory is extended to support capability tags.
+The tag-aware region is highlighted.
+All tags are created by the CHERI CPU, and only stored locally. The memory is shared with the system, probably via a secure DMA, which is not tag aware.
+
+Therefore the connection between CPU and memory is tag-aware, and the connection to the system is not tag aware.
+
+All writes from the system port to the memory must clear any memory tags to follow the rules from above.
+
+=== Large CHERI system example
+
+[#large_cheri_system]
+.Example large CHERI system with tag cache
+image::large_cheri_system.drawio.png[width=80%,align=center]
+
+In the case of a large CHERI SoC with caches, all the cached memory visible to the CHERI CPUs must support tags.
+All memory is backed up by DRAM, and standard DRAM does not offer 129-bit words and so a typical system will have a tag cache IP.
+
+A region of DRAM is reserved for CHERI tag storage.
+
+The tag cache sits on the boundary of the tag-aware and non-tag-aware memory domains, and it provides the bridge between the two.
+It stores tags locally in its cache, and if there is a miss, it will create an extra bus request to access the region of DRAM reserved for tag storage.
+Therefore in the case of a miss a single access is split into two - one to access the data and one to access the tag.
+
+The key property of the tag cache is to preserve the atomic access of data and tags in the memory system so that all CPUs have a consistent view of tags and data.
+
+The region of DRAM reserved for tag storage must be only accessible by the tag cache, therefore no bus initiators should be able to write to the DRAM without the transactions passing through the tag cache.
+
+Therefore the GPUs and peripherals cannot write to the tag storage in the DRAM, or the tagged memory data storage region.
+These constraints will be part of the design of the network-on-chip.
+It _is_ possible for the GPU and peripherals to read the tagged memory data storage region of the DRAM, if required.
+
+NOTE: It would be possible to allow a DMA to access the tagged memory region of the DRAM directly to allow swap to/from DRAM and external devices such as flash.
+ This will require the highest level of security in the SoC, as the CHERI protection model relies on the integrity of the tags, and so the root-of-trust will need to authenticate and encrypt the transfer, with anti-rollback protection.
+
+For further information on the tag cache see cite:[tagged-memory].
+
+<<<
+
+=== Large CHERI pure-capability system example
+
+[#large_cheri_purecap_system]
+.Example large CHERI system with only tag-aware bus masters
+image::large_cheri_purecap_system.drawio.png[width=80%,align=center]
+
+In this example every DRAM access passes through the tag cache, and so _all_ bus masters are tag-aware and can access the tagged memory if permitted by the network-on-chip.
+
+The system topology is simpler than in xref:large_cheri_system[xrefstyle=short].
+
+There is likely to be a performance difference between the two systems.
+The main motivation for xref:large_cheri_system[xrefstyle=short] is to avoid the GPU DRAM traffic needing to look-up every tag in the tag cache, potentially adding overhead to every transaction.