Skip to content

Latest commit

 

History

History
145 lines (98 loc) · 5.02 KB

README.md

File metadata and controls

145 lines (98 loc) · 5.02 KB
Raw

Build Status

Build Status

Introduction

Version Support: ROS2 Dashing, Fast-RTPS 1.8.0

This test allows you to test performance and latency of various communication means like ROS 2, ROS 2 Waitset, FastRTPS, Connext DDS Micro and Eclipse Cyclone DDS.

It can be extended to other communication frameworks easily.

A detailed description can be found here: Design Article

Requirements

Building and running performance test

ROS 2: https://github.com/ros2/ros2/wiki/Installation

Additional dependencies are Java and others declared in the package.xml file

sudo apt-get install default-jre
rosdep install -y --from performance_test --ignore-src

Generating graphical plots

The script which generates the graphical plots needs matplotlib and pandas:

sudo apt-get install python-matplotlib python-pip
pip install pandas

How to build and run

source ros2_install_path/setup.bash
mkdir -p perf_test_ws/src
cd perf_test_ws/src
git clone https://github.com/ApexAI/performance_test.git
cd ..
colcon build --cmake-args -DCMAKE_BUILD_TYPE=Release
source install/setup.bash
ros2 run performance_test perf_test --help

Plot results

To plot the results, you will need to install the perfplot tool from the apex_performance_plotter python module.

pip3 install performance_test/helper_scripts/apex_performance_plotter

This tool will convert performance test log files into PDFs containing graphs of the results.

Note: Some of the dependencies of apex_performance_plotter (specifically pandas, at the time of writing) require python 3.6. It is possible to get apex_performance_plotter working with older dependencies that run with python 3.5, but that is beyond the scope of this document

Example

After building, a simple experiment can be run using the following.

Before you start please create a directory for the output.

mkdir experiment
cd experiment
ros2 run performance_test perf_test -c ROS2 -l log -t Array1k --max_runtime 30

The generated log-files can then be plotted into a PDF file using:

perfplot log_Array1k_19-09-2019_12-13-49

This will generate a file called log_Array1k_19-09-2019_12-13-49.pdf.

Save results to a SQL database

See Add SQL support readme for instructions and implementation details.

Batch run experiments (for advanced users)

Multiple experiments can be run using the following command:

python src/performance_test/performance_test/helper_scripts/run_experiment.py

You need to edit the python script to call the performance test tool with the desired parameters.

Relay mode

Testing latency between multiple machines is difficult as it is hard precisely synchronize clocks between them. To overcome this issue performance test supports relay mode which allows for a round-trip style of communication.

On the main machine: ros2 run performance_test perf_test -c ROS2 -t Array1k --roundtrip_mode Main On the relay machine: ros2 run performance_test perf_test -c ROS2 -t Array1k --roundtrip_mode Relay

Note that on the main machine the round trip latency is reported and will be roughly double the latency compared to the latency reported in non-relay mode.

Memory analysis

You can use OSRF memory tools to find memory allocations in your application. To enable it you need to do the following steps, assuming you already did compile performance test before:

  1. Enter your work space: cd perf_test_ws/src
  2. Clone OSRF memory memory tools: git clone https://github.com/osrf/osrf_testing_tools_cpp.git
  3. Build everything cd .. && ament build --parallel --build-tests --cmake-args -DCMAKE_BUILD_TYPE=Release
  4. You need to preload the memory library to make diagnostics work: export LD_PRELOAD=$(pwd)/install/lib/libmemory_tools_interpose.so
  5. Run with memory check enabled: ros2 run performance_test perf_test -c ROS2 -l log -t Array1k --max_runtime 10 --check_memory

Note that enabling this feature will cause a huge performance impact.

Custom environment data

You can set the APEX_PERFORMANCE_TEST environment variable before running performance test to add custom data to the output CSV file. This information will then also be visible in the files outputted by the plotter script.

Example:

export APEX_PERFORMANCE_TEST="
My Version: 1.0.4
My Image Version: 5.2
My OS Version: Ubuntu 16.04
"
ros2 run performance_test -c ROS2 -t Array1k

Troubleshooting

  1. When running performance test it prints ERROR: You must compile with FastRTPS support to enable FastRTPS as communication mean.

You need to build with --cmake-args -DPERFORMANCE_TEST_FASTRTPS_ENABLED=ON to switch from ROS 2 to FastRTPS.

Known Limitations

  • FastRTPS waitset does not support a timeout which can lead to the receiving not aborting. In that case the performance test must be manually killed.