Communication issues with many Docker containers

[cr01927]

First brought up here - https://answers.ros.org/question/346990/ros2-eloquent-tf2-listeners-occasionally-prevent-receiving-msgs/

Bug report

Required Info:

• Operating System:
  Ubuntu 16.04

• Installation type:
  Using the official ROS2 Docker image found here -
  https://hub.docker.com/layers/ros/library/ros/eloquent-ros-core/images/sha256-7fb316a173581035fc63d0ea4d6e7df50bfd4d793b1c9114c5802fb1d0969c8d?context=explore

• Version or commit hash:

  • Image digest sha256:516e840e543e8be6d58bd4dbf37eff7f5f4da7cd3c4f110a92776f480a9a9287

• DDS implementation:

  • Fast-RTPS
  • Edit: Also occurs with OpenSplice, CycloneDDS

• Client library (if applicable):

  • rclpy, but have seen somewhat similar issues with ros2-web-bridge (so roslibjs)
  • Did not see issue with rclpy publisher, rclcpp listeners

Steps to reproduce issue

Install Docker (v19.03.8) and docker-compose (1.25.4)
Clone this repository - https://github.com/cr01927/ros2-tf2-error-example
Run this command for 20 seconds or so, then kill with ctrl-c

docker-compose up --build --scale listener=20 > log.txt

Parse the output with

less log.txt | egrep -o 'listener_([0-9]+)' | sort | uniq -c

Expected behavior

The output of the above command should show each listener with the same number of log messages, meaning each listener node executed its callback function the same number of times (+1 for an initialization log msg)

Actual behavior

# log msgs | listener name
     16 listener_1
     16 listener_10
     17 listener_11
     17 listener_12
      1 listener_13
     17 listener_14
     15 listener_15
     16 listener_16
     16 listener_17
      2 listener_18
     14 listener_19
     17 listener_2
      2 listener_20
     17 listener_3
      2 listener_4
     17 listener_5
     16 listener_6
     17 listener_7
     14 listener_8
     17 listener_9

Some listeners never receive any msgs, or very few

Additional information

I believe this is related to using a TF listener - I could not get this example to reproduce the error until I added the listener.

ETA: Fixed docker-compose redirect to log file, fixed grep regex pattern, and added OpenSplice to list of DDS implementations where this occurs
ETA: Added CycloneDDS to list of DDS implementations where this occurs

[fujitatomoya]

I am curious on this. Not sure if this is related to ros2/rmw_fastrtps#349, could you try to run with rmw_cyclonedds instead if this still reproduces?

[cr01927]

Just ran on cyclonedds, and the issue still presented. Also, I wrote a cpp listener and tested it on various DDS implementations, and the issue did not present, so it looks like its in rclpy or the Python implementation of TF2.

[fujitatomoya]

@cr01927

got it, thanks for the information.