The access_mapping repository is designed as a ROS (Robotic Operating System) package to be built inside a catkin workspace as setup in the installation section below. The goal of this package is to provide ROS nodes capable of identifying potential barriers or constraints in the environment for the purpose of autonomous robot or handicapped/accessibility navigation and traversability. A long term use of these capabilities would be the potential to generate global maps of the environment (using existing Simultaneous Localization and Mapping (SLAM)-based approaches) that are annotated and labeled with different barriers. These global maps, that have been annotated with the location of barriers such as stairs, doors, and handicap-accessible blue buttons, could then be exploited for front-end use, allowing users to plan an optimized path from point A to point B that accounts for all personally potential barriers, whether the user is a handicapped wheelchair user or a small autonomous rover.
Figure 1: SLAM-annotation block diagram
All work on the access_mapping package is part of ongoing academic research at the Assistive Robotics Lab at Virginia Tech. The idea for the project stemmed from the social engineering principles surrounding handicapped navigation on the Virginia Tech campus and grew out of its applicability to both fields of autonomous navigation and social engineering. All research was funded by the National Science Foundation (NSF) as part of a Research Experience for Undergraduates (REU) program at Virginia Tech in Automotive Engineering.
All installation steps below were performed and tested on a fresh Ubuntu 18.04 installation. Support for other platforms is not available at this time. Instructions below assume a fresh Ubuntu 18.04 installation, and it is advised to gain access to one if not already available. All testing on installation was performed on a Ubuntu 18.04 installation dual-booted with Windows 10. See dual-boot options and instructions.
EaseUS is recommended for disk partition management.
WinDirStat is recommended for hard disk space management.
A hard disk partition of 30GB or more will be sufficient for installation of Ubuntu and all required software for the access_mapping package.
All required software will be installed with associated dependencies in the installation section below. The access_mapping package is being developed as a possible plug-in map-annotator for the existing SLAM package: rtabmap_ros. See their ROS wiki page here.
Procedure to install ROS Melodic version was followed on the ROS wiki.
Tutorials on the ROS wiki setup a catkin workspace with the legacy catkin_make tools, as seen here.
However, for this installation, the still-in-beta catkin build tools will be used to build the workspace.
A python 3 virtual environment will be set up to install all required python 3 packages and modules. Learn more about python virtual environments.
In order to utilize open source computer vision software (OpenCV)'s dnn neural network module for object detection, OpenCV 3.4.7 will be installed inside the python 3 virtual environment. This tutorial on pyimagesearch.com was followed to download and compile OpenCV 3.4.7 from source.
This is not a requirement of the access_mapping package. But all rosbag files used by access_mapping should include (at least) the following ZED topics:
/zed/zed_node/rgb/image_rect_color
/zed/zed_node/rgb/camera_info
/zed/zed_node/depth/depth_registered
/zed/zed_node/depth/camera_info
/zed/zed_node/odom
/tf
/tf_staticA ZED camera with its Software Development Kit (SDK) is required to record all compatible rosbag files. See the ZED website for more information, as well as the Access_SLAMNav_deployment/mapper repository, for more information on recording rosbag files with a ZED camera.
All installation steps below were performed and tested on a fresh Ubuntu 18.04 installation. Development for other platforms are not supported at this time. All software as described above will be installed.
- Open a new terminal window:
Ctrl+Alt+t - Get installation scripts: Navigate to home directory and clone from
Access_SLAMNav_deploymentrepository by typing the following commands.
sudo apt update
sudo apt -y install git
cd ~
git clone https://github.com/ARLab-VT/Access_SLAMNav_deployment.git
cd Access_SLAMNav_deployment.git/slam_annotator- Allow execution of the install script:
chmod +x install_first.sh- Run the installation script:
./install_first.shAllow the script to run to completion. Confirm installation of all required packages as the script runs by pressing enter or typing password as prompted. install_first.sh downloads ROS Melodic and builds OpenCV 3.4.7 from source into a python 3 virtual environment.
- Open a new terminal window:
Ctrl+Alt+t - Setup the catkin workspace by typing the following commands:
rossource
cd ~
mkdir -p catkin_ws_cb/src
cd catkin_ws_cb/src
git clone https://github.com/ros/geometry
git clone https://github.com/ros/geometry2
git clone -b melodic https://github.com/ros-perception/vision_opencv.git
git clone https://github.com/ARLab-VT/access_mapping.git
cd ..
workon cv
pip install rospkg catkin_pkg pyyaml empy
catkin config -DPYTHON_EXECUTABLE=~/virtualenvs/cv/bin/python -DPYTHON_INCLUDE_DIR=~/.virtualenvs/cv/include/python3.6m
catkin config --no-install
catkin build
source devel/setup.bash
echo -e "\n# Source the catkin workspace" >> ~/.bashrc
echo "alias catsource='source ~/catkin_ws_cb/devel/setup.bash'" >> ~/.bashrc
source ~/.bashrcConfirm all installation requests. A catkin workspace was set up and several ROS packages were built for use with python 3. The access_mapping package was also built for python 3.
- Download the YOLOv3 neural network model for use with OpenCV's
dnnneural network module. Navigate to the link below and save theyolo-cocofolder in the source directory of theaccess_mappingpackage. Download yolo-coco as a .zip file.
cd <your_downloads_folder>
mv <your_yolo_file.zip> ~/catkin_ws_cb/src/access_mapping
cd ~/catkin_ws_cb/src/access_mapping
unzip <your_yolo_file.zip>
rm <your_yolo_file.zip> Example:
cd ~/Downloads
mv yolo-coco.zip ~/catkin_ws_cb/src/access_mapping
cd ~/catkin_ws_cb/src/access_mapping
unzip yolo-coco.zip
rm yolo-coco.zip- Ensure proper system setup by sourcing .setup files:
rossource(alias created during install for sourcing ROS workspace, orsource /opt/ros/melodic/setup.bash)catsource(alias created during install for sourcing catkin workspace, orsource ~/catkin_ws_cb/devel/setup.bash) Optional: add the two sourcing commands above to your.bashrcfile, (or other shell setup file) to run every time your terminal opens:
echo -e "\n# Source ROS and catkin workspace" >> ~/.bashrc
echo "rossource && catsource" >> ~/.bashrc- Open a new terminal window and source setup files:
rossource
catsource- Run ROS master:
roscore-
Open another terminal window and source setup files.
-
Run
gate.pynode inaccess_mappingpackage with default values:
cd ~/catkin_ws_cb
roscd access_mapping
cd nodes
rosrun access_mapping gate.py --rosbag <path_to_rosbag_file>Demo rosbag file is available here: link. Move the rosbag file to <path_to_rosbag_file> as mentioned above.
-
Open new terminal window and source setup files.
-
Run
object_detect.pynode inaccess_mappingpackage with default values:
workon cv
rosrun access_mapping object_detect.py --yolo ../yolo-coco
Note: required --yolo argument asks for path to folder containing YOLO configuration files, as downloaded in step 6 of install section.
-
Quitting: Click on terminal window where
gate.pynode is running and press'q'. Alternatively, each node can be shutdown with keyboard-interruptctrl+c. -
Please see project detail in Contribute section for more info on each node and optional command-line flags.
Following the default deployment method described above, the output of the object_detect node is shown below when set to detect people and chairs. A video file of the output is included in the images subdirectory.
Figure 2: RGB image from YOLO object detection
Figure 3: Annotated depth image from ZED camera
The 3D global coordinates of objects can be aggregated in a list and utilized in many ways. The output below shows the projection of these points onto a 2D occupancy grid for data visualization:
[output not currently available]
The goal of this section is to provide documentation on overall usage of the access_mapping package.
The gate.py node, located in the access_mapping/nodes subdirectory, acts in place of the standard rosbag play command line tool, with added control over the speed and timing of publishing messages contained in the rosbag. The node was developed to account for execution overhead involved with object recognition algorithms. The node utilizes the rosbag API to read a rosbag from a file and play it back within given constraints. The goal of the gate.py node is to control the release of ROS messages based on the processing time of all connected 'slam-annotators'. Slam-annotators are all nodes within the access_mapping package that develop and export a list of detected constraints in the environment, as well as their locations. All 'slam-annotators' should be connected to the gate.py node and communicate their readiness to process more messages upon completion of object recognition algorithms. Please see the documentation within the gate.py python script for usage details.
The object_detect.py node, located in the access_mapping/nodes subdirectory, is the only current 'slam-annotator' node. The object_detect annotator performs object recognition and image localization using a pre-trained YOLO neural network. The node can be configured to detect any of the 80 objects in the 'coco' dataset. The object_detect node communicates with the gate.py node to regulate processing time, and can be played back in any of 4 separate playback modes. Multiple instances of the object_detect node can be initialized on separate machines to recognize and publish the locations of various objects in the environment, and still communicate with the same gate node. Note: the object_detect node is python 3 dependent and should be played while the python 3 virtual environment cv, as setup during installation, is activated. Please see the documentation within the object_detect.py python script for usage details.
The following ROS topics are used by the access_mapping package:
| Topic | Message type | Description |
|---|---|---|
| /zed/zed_node/rgb/image_rect_color | sensor_msgs/Image | RGB image stream |
| /zed/zed_node/rgb/camera_info | sensor_msgs/CameraInfo | RGB camera info |
| /zed/zed_node/depth/depth_registered | sensor_msgs/Image | Depth image stream |
| /zed/zed_node/depth/camera_info | sensor_msgs/CameraInfo | Depth camera info |
| /tf | tf2_msgs/TFMessage | Transform information |
| /zed/zed_node/odom | nav_msgs/Odometry | Camera odometry information |
| /tf_static | tf2_msgs/TFMessage | Basic transform information |
| /ready | std_msgs/String | Communication line between gate and slam-annotators |
| /end | std_msgs/String | For processing gate shutdown |
The rqt_graph output below shows the interaction of the gate.py node with two slam-annotators, person_detect.py and chair_detect.py, over the topics described above:
Figure 4: access_mapping ROS node/topic diagram
Work is in progress on creating a launch file, SLAM_annotate.launch, to run the gate.py node along with any number of slam-annotator nodes. For now, please see the Deployment section above for proper setup and deployment of the ROS system. All launch files should be placed in the access_mapping/launch subdirectory.
There are no current new custom message types needed by the access_mapping package. In the future, it might be necessary to create new message types to export all required data out of the slam-annotators. All custom message types should be placed in the access_mapping/msg subdirectory, and the CMakeLists.txt file and package.xml file should be updated as needed.
When the object_detect.py node is run in step-mode:
rosrun access_mapping object_detect.py --yolo ../yolo-coco --playback sthe user should be able to step through each set of messages in the rosbag file, (loaded by the gate node), by pressing the 's' key. The user may exit the program by pressing 'q'. One 'step' through the rosbag should only load one new set of the 6 ROS topics recorded by the ZED camera. However, when the 's' key is held down for sustained periods of time, the rosbag will begin playing all messages nonstop. Functionality should be to only step once per press of the 's' key. The bug is most attributable to the method in which keyboard input is read by the object_detect.py script.
Functionality of the object_detect.py node should be to transform 3D points obtained in the camera's frame into a global frame. Calling of the tf_global function in the object_detect.py script does not currently perform this functionality correctly and results in various error messages. Current work is focused on solving this problem.
Data from the object_detect.py node is used to modify an occupancy grid hard-coded within the node. This occupancy grid is then displayed in rviz. Methods do not currently exist to create a dynamic occupancy grid that grows as points on the map grow. There is also no current implementation for clean storage or export of this occupancy grid for further use outside of the object_detect node. The implementation of some kind of export functionality will be necessary for future work, with an end goal to have occupancy grid data from the slam-annotators to be imported for use in current SLAM-mapping technologies, primarily the ROS package rtabmap_ros.
The access_mapping ROS package has only been tested and developed on a fresh Ubuntu 18.04 installation. Work in the future will be required to develop the package for robustness and transportability to other system setups and installations.
Current functionality only finds objects from an image stream and uses their location to modify a hard-coded occupancy grid, as described in the "Occupancy grid" section above. Desired functionality is to aggregate all object observations, accounting for repeated objects, and output a useful data structure for data visualization and mapping.
As mentioned in the "Occupancy grid" section above, the end goal of the slam-annotator nodes would be to export and display object location information alongside the output of the rtabmap_ros ROS SLAM package. The end-product would be an occupancy grid map that shows all physical barriers recorded during the mapping session that is annotated with objects/barriers detected by the slam-annotator nodes.
As setup currently, the object_detect.py node utilizes a YOLOv3 neural network for all object detection algorithms. In the future, it might be desired or necessary to incorporate support for other popular, versatile, or home-made neural network models. Configuration and usage of the YOLO network would be replaced within the object_detect.py node, or perhaps another separate node might be made for each desired neural network model.
The YOLO network in use currently is pre-trained on a set of 80 class labels, as found in the "coco" dataset. An end goal of this project is to recognize various barriers to handicapped or disabled users, including stairs, handicapped-accessible blue buttons, doors, etc. It will be necessary to find or prepare a dataset of images in a desired class to train a neural network for object recognition on any of these potential barriers.
Several functionalities within the ROS nodes require user input for proper setup and execution, including loading a neural network from a file and reading and playing back a rosbag from a file. Any time user input is required, there are many chances for breaking code from invalid or improperly formatted input. More work can be done in error-handling invalid user input. Another nice feature to add would be a clean, efficient, and easy way for users to know and format information they can specify on the command-line when running the nodes. A configuration file can also be setup that the user fills out with all necessary information, and the system handles the formatting and parsing of the information.
As currently setup, the access_mapping package only supports SLAM-annotation on a prerecorded rosbag with the necessary ROS topics. A useful feature in the future would be to add support for live SLAM-annotation that runs concurrently as the mapping software. This would require many changes to how the gate.py node accesses its information.
Current setup only allows for use with the ZED camera and the associated ZED SDK. Further work is required for integration of other recording setups and increased robustness.
The object_detect.py node is the only available slam-annotator, and relies on image streams from a stereo ZED camera for object detection. Accessibility constraints/barriers that are ripe for detection by the current implementation include:
- Doors and doorways
- Handicapped accessible blue buttons/signs
- Stairs
- Road signs
Other slam-annotators may be developed that account for barriers in the environment not so easily recognized by just an image stream. Such barriers include:
- Different kinds of terrain (grass, cracked roads, dirt, etc.)
- Sidewalk curbs or curb-cuts
- Heavily-inclined slopes
- Wi-fi signal strength
- Access to lighting for use by solar panels
- Loud noise-level zones (construction, machinery, etc.)
Once a working setup is developed using the current implementation, a single map may be generated in a single mapping session that includes all barrier/constraint annotations in the produced occupancy grid. What if someone were to make a separate map during a different mapping session and want to merge the results into the first map? This area could easily turn into many more projects in the future.
Data generated and stored by the current implementation, in whatever data structure that takes on, can be used in the future for front-end users for the purpose of navigation. After global mapping has been performed in a given area, with all barrier/constraint annotations included, any front-end user, whether that be an autonomous robot or a disabled person, will need an interface or method by which their navigation stack takes into account the given barriers in the environment for global path planning.
ROS:
http://wiki.ros.org/Documentation
http://wiki.ros.org/melodic/Installation/Ubuntu
Catkin tools:
https://catkin-tools.readthedocs.io/en/latest/quick_start.html
https://answers.ros.org/question/243192/catkin_make-vs-catkin-build/
Python virtual environment:
https://realpython.com/python-virtual-environments-a-primer/
Python 3:
https://medium.com/@beta_b0t/how-to-setup-ros-with-python-3-44a69ca36674
OpenCV-Python:
https://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_tutorials.html
https://www.pyimagesearch.com/2018/05/28/ubuntu-18-04-how-to-install-opencv/
ZED camera:
https://www.stereolabs.com/zed/
3D coordinates:
http://nicolas.burrus.name/index.php/Research/KinectCalibration