DLB: Deep Learning Blocks

'DLB (Deep Learning Blocks)' as a part of DPU (Deep Learning Processing Unit) is a collection of synthesizable Verilog modules for deep learning inference network.

All contents are provided as it is WITHOUT ANY WARRANTY and NO TECHNICAL SUPPORT will be provided for problems that might arise.

Table of contents

Click to expand table of contents

1. Overview
   1.1 Prerequisites
   1.2 Prepareing environment
2. Verification
   
3. Deep Learning Blocks
   3.1 Concatenation
   3.2 Convolution
   3.3 Linar
   3.4 Pooling
   3.5 Residual
4. Deep Learning Processing Unit
   
5. Projects
   5.1 Project: LeNet-5
   5.2 Project: Tiny Yolo V2
   5.3 Project: U-Net
6. Other things
   6.1 Acknowledgment
   6.2 Authors and contributors
   6.3 License
   6.4 Revision history

  

---

1. Overview

'DLB (Deep Learning Blocks)' as a part of DPU (Deep Learning Processing Unit) is a collection of synthesizable Verilog modules for deep learning inference network.

DLB blocks contain followings (some are under development and more will be added):

• Convolution layer (straightforward 2D convolution)
• Pooling layer (2D max pooling)
• Fully connected layer (1D linear vector-to-matric multiplication)
• Concat layer
• Residual layer (point-to-point adder)
• move, Fill, and so on.

The blocks have following highlights:

• Fully synthesizable Verilog code
• Parameterized to be adopted wide range of usages
• Test-benches for Verilog, C/C++, and PyTorch
• FPGA verified using Future Design Systems’ CON-FMC

1.1 Prerequisites

This program requires followings.

• GNU GCC: C++ compiler
• PyTorch / Conda / Python3
• HDL simulator: Xilinx Xsim (Xilinx Vivado 2019.1 or later)
  • Xilinx Vivado 2017 or earlier version does not support SystemVerilog features that are used in this project.
  • Xilinx Vivado Webpack will be fine, which does not require license.
• FPGA implementation: Xilinx Vivado
• HW-SW Co-Simulation Library for AMBA AXI BFM using DPI/VPI
• DLR: Deep Learning Routines (v1.4 for thie project)

1.2 Prepare environment

Refer to "How to Prepare DPU Development Environment".

  

---

2. Verification

The picture below shows how to verify DLB blocks.

| | | | | | |:---:| |:---:| |:---:| | DPU on board | | DPU on FPGA | | DPU HW/SW co-simulation |

  

As shown in the picture below, DLB blocks are verified along with HDL (Hardware Description Language), C/C++, and Python.

| || || | |:---:| |:---:| |:---:| | DLB with HDL | | DLB with C | | DLR with Python|

  

---

3. Deep Learning Bloks

It should be noted that the blocks are not optimized to get a higher performance since the blocks are for hardware implementation not for computation. In addition to this the blocks are only for inference not for training. For optimized version, contact at Future Design Systems.

3.1 Concatenation

To be added.

3.2 Convolution

To be added.

3.2 Deconvolution

To be added.

3.3 Linear (Fully connected)

To be added.

3.4 Pooling

To be added.

3.5 Residual

To be added.

  

---

4. Deep-learning Processing Unit

More details will be added.

Deep-learning processing unit

  

---

5. Projects

5.1 Project: LeNet-5 for MNIST

LeNet-5 is a popular convolutional neural network architecture for handwritten and machine-printed character recognition.

Following picture summarizes how to use DPU for LeNet-5 inference network, in which design flow consists of three major phases.

• Training phase: Training on PyTorch
  • It uses PyTorch framework for modeling and training the LeNet-5 to get trained weights.
• Verification phase: HW/SW co-simulation
  • It uses cosim framework for verifying functionality after preparing DPU RTL and SW.
• Inferencing phase: Running on FPGA
  • It uses CON-FMC frmework to run the model on FPGA along with SW through USB.

This project uses '32-bit floating point' data type and there is no need to manipulate trainined weights. This project does not have 'network compiler' that could generate C/C++/Python SW code to control DPU hardware.

5.2 Project: Tiny YOLO-V2

More details will be added.

5.3 Project: U-Net

More details will be added.

  

---

6. Other thins

6.1 Acknowledgment

The author(s) thanks to all who contributed on this package.

6.2 Authors and contributors

• [Ando Ki] - Initial work - Future Design Systems

6.3 License

DLR (Deep Learning Routines) and its associated materials are licensed with the 2-clause BSD license to make the program and library useful in open and closed source projects independent of their licensing scheme. Each contributor holds copyright over their respective contribution.

The 2-Clause BSD License

Copyright 2020-2021 Future Design Systems (http:://www.future-ds.com)

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.

2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

6.3 Revision history

• 2021.08.10: Started by Ando Ki (adki(at)future-ds.com)