Capstone project in University of Washington (EE526)

ASIC Implementation of Local Feature Detector in ORB-SLAM3

Image feature extraction and matching is a fundamental but computation intensivetask in machine vision. This project implements FAST feature point detection and BRIEF feature descriptor construction and matching.

The function of our design include FAST feature extraction and BRIEF feature matching.

The objective is (1) understanding the theories and code in ORB_SLAM. In folder slam_ex contain same simple practice while learning the math theories in VSLAM. (2) Speed up ORB feature detector in verilog to speed up. In folder RTL contain the verilog implement. In folder Simulation contain the simulation result.

Verification Dependencies For FAST

This project verification works based on the environment with the following dependencies.

• MATLAB with computer vision Toolbox(FAST Verification)

Verification Dependencies For BRIEF

This project verification works based on the environment with the following dependencies.

• Python 3.8.10
• cv2

Launch For FAST

Firstly, use translate_pic.m matlab script in Prepare to transfer jpg image into binary file. You can also run confirm.m to confirm if the picture loaded successfully. (There are binary file exist in simulation folder, you donot need to run those to generate binary file for test)

1. 

2. You can check the input pixel in Workspace, the variable name is test.

   

Secondly, run vcs FIFO.v feature.v matrix77point.v feature_wrapper.v score_top.v NMS.v matrix33_mat.v FAST_top.v tb_FAST.v +v2k -R +lint -sverilog -full64 -debug_pp -timescale=1ns/10ps -l fast.

1. Run in ECE server to see the simulation result.

2. After runing, two useful files are generate named x.csv and y.csv. They are feature point 's' coordinate. (x.csv and y.csv already exist in the simulation folder )

Thirdly, compare the simulation results with software result. (Files are in Verification folder)

1. Open Matlab (should have computer version Toolbox) and run matlab script named cmp_FAST.m

   

2. The red point is feature point found by software and the blue point is the feature point found by hardware.

Directory Structure

• Prepare_FAST:

  • test.jpg -> figure for test
  • translate_pic.m -> translate jpg into binary file
  • confirm.m -> make sure the jpg in translate to binary file correctly
  • testsave.bmp -> picture in gray

• Prepare_BRIEF:

  • binary_file -> the converted binary file that used for verilog
  • jpeg -> original images and the croped images
  • bin2jpg.py -> regenerate the figure, from the one got from convert_binary.py
  • convert_binary.py -> write the binary values into txt
  • cropimg.py -> cut to (720, 1280), turn to grayscale
  • getkp.py -> Get the keypoints with Python
  • jpg2intensity.py -> write the intensity into txt file
  • orb.py -> Python code for orb extraction

• RTL_FAST

  • design_FAST_NEW -> The final RTL used
    • FIFO.v
    • NMS.v
    • feature.v
    • feature_wrapper.v
    • matrix33_mat.v
    • matrix77point.v
    • tb_FAST.v
  • design_FAST_OLD -> Old version

• RTL_BRIEF:

  • binary_test.v
  • brief.v
  • biref_ctrl.v
  • dram.v
  • dram_ori.v
  • gaussian_core.v
  • gaussian_ctrl.v
  • gaussian_top.v

• Simulation_FAST

  • RTL code same as design_FAST_NEW
  • prev.bin -> Input image in pixel
  • x.csv -> x coordinate of the feature point
  • y.csv -> y coordinate of the feature point

• Simulation_BRIEF

  • keypoints
  • result -> including all the vcd files generated (expect the top level BRIEF which is too large to upload to git)
  • Makefile
  • README.md
  • binary_test_tb.v
  • dram_ori_tb.v
  • dram_tb.v
  • gaussian_core_tb.v
  • gaussian_ctrl_tb.v
  • gaussian_top_tb.v

• Verification_FAST

  • cmp_FAST.m -> compare the hardware simulation with software
  • prev.bin -> input image in pixel
  • x.csv -> x coordinate of the feature point
  • y.csv -> y coordinate of the feature point

• Verification_BRIEF

  • brief -> visualized brief result
  • reconstructed -> reconstructed images after gaussian blur
  • result -> the txt file containing the data got from Simulation
    • brief
    • gaussian
  • intensity2img.py
  • matcher.py

• python

  python to process data in ORB SLAM3 (you can ignore them)

  • C++ code to learn how to learn ORB SLAM3 (you can ignore them)

actual progress

April:

1. Rotation Matrix
2. Euler Angles and Rotation Matrix
3. Basic Lie Group and Lie Algebra
4. Images in cv
5. ORB sw implement

May

1. Understand the algorithm and design for the top view.
2. Sliding windows
3. Score
4. Finish RTL design

June

1. Debug
2. Synthesis, APR