Push-Button-Calyx-FPGA Lab Notebook

[nathanielnrn]

This is a place where I will be tracking issues, progress, and thoughts on the effort to get FPGA execution to an easy-to-use state. Feedback, comments, questions, nits, and anything else is always super appreciated. Please don't hesitate to leave comments or reach out!

This work is building off of some of the work tracked in a previous notebook.

For a fantastic high-mid level overview see @sampsyo's writeup #1756

To briefly go over that writeup, planned steps are as follows:

1. Clean up the current Verilog AXI generation that we have. There are likely still a few bugs in our implementation which cause undesired behavior in programs. Some bugs will require immediate fixes, while other can be left as TODOs for the planned Calyx implementation (described below). Tracking down bugs will likely be accelerated by writing an AXI interface by hand. I will be working on this to improve my understanding of AXI.

The goal at the end of this stage is to have a basic one-button-flow that lets us execute (small) programs on our FPGAs using our Verilog AXI implementation.

2. Separate AXI generation from the Calyx compiler. To do this, a standard "Interface Description Language" would need to be created that described the input/outputs of an accelerator based on our programs.
   Our goal here is to have a concrete IDL that is expressive enough to use for AXI interface generation and readmemh wrapper generation.

3. Create a tool that would consume the IDL described above and generate an implementation of this interface.
   Our goal is to have tools that consume said IDL to generate AXI interfaces implemented in Calyx (using calyx-py).

Further, long-term steps can be seen in the writeup linked above.

[nathanielnrn]

Sep 4, 2023

Most of this week was spent setting up and refamiliarizing myself with AXI.

I was able to run our example dot products and vectorized addition on havarti's fpga, which was nice.

Having a better grasp of the tools we have, the next work I intend to pursue will be going through xilinx backends and looking through notes & XXXs & TODOs there. I seem to recall that we could only execute programs with less than 512 bytes of input/output but the memory is hazy. Hoping I'll be able to repro this issue quickly and see what can be done to fix this.

Fixing up our generated Verilog as stated above is all in the name of getting more familiar with implementing AXI so that we can then move on to creating a stand alone Calyx-AXI-Generator that takes in some description of interfaces generated by Calyx, then outputs more Calyx that implements an AXI interface based on that description. (Detailed issue forthcoming, for now see Phase 1 and #1084).

Some legwork will probably be needed before working on the above in earnest, namely separation of AXI and XML generation tools.

To Summarize

Action items seem to be

1. Reproduce too-large-program issue and implement a fix.
2. Separate AXI and XML generation tools.
3. Work on generating interface descriptions through Calyx.

As always, happy to hear thoughts!

cc @rachitnigam @sampsyo @andrewb1999

[rachitnigam]

Sweet progress @nathanielnrn! It's cool that things "just worked" on the havarti FPGAs; those designs can serve as a nice test suite for the new, Calyx-based AXI generator in the future. Couple of scattered thoughts on things:

• Making a wider set of designs work could be good short term work. It'll make this test suite bigger and allow you to have more confidence when building the AXI-based flow.
• I would recommend immediately crashing if you see memories that are not a power of two. I suspect that there might be annoying alignment requirements in AXI that are best dealt with once we have more confidence in our interface generator.
• Re: separating the AXI and XML generators. A recent PR did some work in separating them out into their own calyx-backend package. The issue now amounts to separating them further into a package that just generates binaries to run the AXI and XML gen binaries. I think looking at main.rs and using that as an example to see how those tools are invoked could be a good start.

[nathanielnrn]

I'll expand on this on my next update, but seems like I was mistaken. Having trouble running programs on Havarti's fpga at the moment.

[nathanielnrn]

Sept 11, 2023

This week was spent trying to get fixedpoint programs to work on fpgas (after manually compiling xclbins for them). See #1721. The goal here is to have a larger set of tests with which we can validate the envisioned calyx-axi-implementation. The concrete work likely involves refactoring a bit of the current json_to_dat functionality to some sort of abstract json_to_*, and we can then specify if we want to go to .dat or intend to send data to a PYNQ buffer.

Fortunately, the above is not blocking per say (although is definitely something to get done, especially given that it shouldn't take too long to implement). So plan is to work on above and also future work detailed below in parallel.

Spoke with @sampsyo today about concrete steps for creating a backend that outputs information we would need about programs. Seemed like there was agreement that we can be solely interested in memory names, width, and size, and leave off things such as types, which came up as an idea.

I have quite a bit to learn about compilers in general, and Calyx in particular. Plan on looking at first couple of 6120 lectures as an overview. Then, if there are any recommended resources in the vein of "Intro to Calyx backends" I'll look into those (cc @rachitnigam?). Even if such a thing doesn't exist, at a high level I don't think it sounds too complex to pattern match on existing backends and takes a representation of a program's memories and output them as a list. Although, the devil is often in the details.

As always, appreciate all comments/questions/concerns!

[sampsyo]

Awesome. Yeah, if it's helpful at all, I think Lesson 2, on representing programs, is likely the most helpful part of 6120. Not that this should stop you from exploring the rest of it, just for fun, of course. 😃

As far as interacting with the Calyx compiler and writing a new backend, I think the main place to look is indeed at existing backends. FWIW, the "big" main backend is the Verilog backend, but I wouldn't recommend starting there. The most instructive backend is probably the "Xilinx XML backend", which is a weird one: it's the "backend" that produces the kernel.xml file that the Xilinx toolchain expects to describe the hardware interfaces. So it's kind of similar to the JSON description we're hoping to produce—just in XML, and invented by Xilinx instead of us, so it has lower-level and more AXI-specific information than we need.

Salient aspects include that the main work happens in an emit function:
https://github.com/cucapra/calyx/blob/590c1945de6ce993aae0928770e692bd7b1d7a21/calyx-backend/src/xilinx/xml.rs#L121-L124

And that the backend works by traversing the structure of the Calyx program, such as this little chain that filters the cells in the top-level component to find memories:
https://github.com/cucapra/calyx/blob/590c1945de6ce993aae0928770e692bd7b1d7a21/calyx-backend/src/xilinx/xml.rs#L152-L155

Let us know if any questions at all come up while looking over this existing code!

[rachitnigam]

I would also recommend thinking of these as standalone tools and not backends. The XML and Xilinx "backends" are not really backends; they generate summaries from a given Calyx program. This is why we want to refactor them out as tools (#1084).

[sampsyo]

Yes, factoring out the Xilinx-stuff generation into standalone tools is an explicit goal enumerated in the proposal.

For a bit more context, the point at this stage @nathanielnrn is looking at is to generate one summary from which all of this will stem (not only the Xilinx XML and AXI wrapper code but also, eventually, the RTL testbench wrapper). The "goal state" here is that all of the current stuff becomes external tools that consume this summary. But at this stage, we need to construct this summary. The proposal here is that this be a backend (and the last (and eventually only) special-case backend). It should hopefully be super simple: just look for all the external memories and summarize them.

But @rachitnigam brings up the point that this proposal IDL-generating thing is not really a backend, so we shouldn't set it up as, like, calyx -b interface-summary or whatever. We could instead make this a totally separate executable, called calyx-interface-summary or something, that links the Calyx library and just produces this JSON file. It wouldn't be a backend; it would just be a separate executable. I think this seems like a little bit more hassle than it's worth, and this new thing should probably just be a "backend," despite not being a proper codegen backend, but it will be the last one (all other non-backend backends will turn into separate tools). @rachitnigam, does that seem like a bad idea to you?

[nathanielnrn]

This week's work largely consisted of going down the rabbit hole of running programs on actual FPGAs, which I had mistakenly previously thought was working as intended. This took a while to repro and comprehend.

Concretely, this week I intend to continue digging into bolstering our xclrun.py to work for hw-targeted-xclbins as mentioned in #1724 linked above.
Also intend to being implementing our "Idl" backend that turns calyx programs into json describing our memory interfaces. (Happy to get feedback on the name)

[sampsyo]

Good question about the name. 😃 While "IDL" is the general term for any language for describing interfaces, maybe we should make up a name for this specific IDL we are designing for this project? Here are some random ideas:

• CalyxIDL or CIDL are the obvious ones.
• FIDL harkens back to the old name (FuTIL) and also sounds like 🎻.
• Accelerator Interface Description (AID) is extremely generic/descriptive.
• We have considered changing the filename extension for Calyx code to yx (cf Define two binaries for compiler and deprecate futil in favor of calyx #1454 (comment)). In which case we could borrow from OCaml, which has .ml & .mli, and use .yx & .yxi? In which case we would just call our language YXI, pronounced "Calyx interface"?

[nathanielnrn]

This week we got a minimal working version of our IDL backend #1729. Some things left to do is small refactoring and renaming.
We have decided on the yxi name (pronounced icks-ee(?)) as a reference to the the Cal[yx i]nterface.

Going forward it seems like next steps are to get familiar with calyx-py through its tutorials. Goal is to make a small program with it as a means to get familiarized. Maybe basic matrix multiplier into an attempt at a systolic version? We will see.

The next week or so are quite busy for me so the going might be a bit slow. But looking forward to getting to AXI generation on the near future.

[nathanielnrn]

Been a bit since the last update since until now hasn't been a lot to talk about.
Currently, I'm still working on getting a manually written calyx implementation of an axi-wrapper working. We will then go forward with porting this to calyx-py.

Today a synchronous discussion with @sampsyo led to a better understanding of how we should move forward with our calyx implementation.
Currently there are 2 options. Current plan is to move forward with the first one, and if that proves difficult, fall back to second one.

Option 1:
A lot of AXI consists of what can be reduced to linear, non-branching FSMs. We start at some state A, wait for something (possibly an external event) to happen, go to state B, wait, go to state C, and possibly loop back to A. So A -> B -> C -> A. This maps beautifully to calyx's seq constructs. So the hope is that we can construct a small number of self contained groups that allow us to use seqs to implement the FSMs we need for AXI. For example, in AXI a transfer of a single unit of data requires a handshake to occur, meaning both *VALID and *READY signals are high. We can imagine two groups as follows

group block_read{ //stay here until we are ready to perform a transfer (single exchange of data, part of a wider transaction. Includes "if we are ready" logic. In this case, if reg is ready
        RREADY = ARESET ? 1'b0;
        RREADY = !ARESET ? data_received.done; //dones are asserted for a single cycle, so prob should be 1'b1;?
        block[done] = RREADY && RVALID ? 1'b1;
        block[done] = !(RREADY && RVALID) ? 1'b0; /good to be explicit and not rely on undefined behavior
      }

      //precondition: a handshake has occurred, i.e block[done] == 1'b1;
      //NOTE: a transfer is a single exchange of data within a transaction (still requires
      a handshake)
      group do_transfer{
          RREADY = 1'b0;
          data_received.write_en = 1'b1;
          data_received.in = RDATA;
          do_transfer[done] = data_received.done;
      }

And then have a control sequence

control {
  seq {
    block_read;
    do_transfer;
  }
}

That will perform a transfer. We can imagine surrounding this in a while statement that causes this this to occur until all of these transfers are done (which occurs as part of a single transaction, by AXI definitions).

Option 2:
Our other option is to use continuous assignment to achieve the equivalent of above. For example

wires {                                                                                                                                                                                     
      RREADY = ARESET ? 1'b0;                                                                                                                                                                 
      RREADY = !ARESET ? data_received.done;                                                                                                                                                  
      std_reg.write_en = RREADY && RVALID ? 1'b1;                                                                                                                                             
      std_reg.write_en = !(RREADY && RVALID) ? 1'b0;                                                                                                                                          
}                                                  

This can get a bit verbose and is probably not as nice to reason about. So this is intended to be a fallback if things don't go well with our break-up-into-groups approach doesn't work well for some reason. We will basically be writing verilog in calyx, as these continuous assignment map closely to verilog continuous assignments.

With these 2 options in mind, I think the way forward is getting read channel transfers to work, then read address channel transfers, then combine them into getting read transactions to work.
We will then move on to write channels, write-address channels, and write-response channels. We will then combine these into correct write transactions.

If any of this seems confusing, one thing that might be relevant to note is the difference between a transfer and a transaction:

AXI makes a distinction between transfers and transactions:

A transfer is a single exchange of information, with one VALID and READY handshake.
A transaction is an entire burst of transfers, containing an address transfer, one or more data transfers, and, for write sequences, a response transfer.

As always, happy to hear any thoughts on any of the above!

[sampsyo]

Awesome; thanks so much for the summary, @nathanielnrn!

For the benefit of anybody else reading this stuff opportunistically, I just wanted to highlight that this concept of a "wait until something happens" group is super interesting and different from what we have done with Calyx so far. To summarize, the idea is that a group exists purely to interface with the outside world and to define its done signal based on something that happens there. With that in place, the control program can activate that group as a kind of funky "await" statement, forcing an ordering on other stuff in the control program. Very nifty!!

[rachitnigam]

FWIW this kind of block group also shows up in @paili0628's @sync operator compilation!

[sampsyo]

Good point!