ros2-foxy: OS-0 sensor hangs while activating #79
Comments
|
Interesting, I'll try to reproduce tomorrow. |
|
I think I have the same issue with a OS1. The driver works well in WSL with Ubuntu Jammy and Humble (as long as the UDP ports are forwarded), but does not in a Ubuntu Jammy Docker container. The sensor gets configured (i.e. switches from operating mode "Standby" to "Normal") but then hangs at: See the full log$ros2 launch ouster_ros sensor.launch.xml sensor_hostname:=192.168.1.60 lidar_port:=7502 imu_port:=7503 [INFO] [launch]: All log files can be found below /home/grisi/.ros/log/2023-03-24-11-11-56-349559-molisens-349 |
|
@Grisly00 thanks for sharing the info. This is mostly an issue an issue with the underlying DDS driver, the asynchronous callback does not work all the time. The only solution probably for us is not to have this dependency to start with. |
|
@crowncastlejsl I was able to re-produce the issue, did you happen to try |
Previous commitTo try the New error messages: [component_container_mt-1] [ERROR] [1680715918.147608892] [ouster.os_container]: Component constructor threw an exception: tf_prefix
[ERROR] [launch_ros.actions.load_composable_nodes]: Failed to load node 'os_cloud' of type 'ouster_ros::OusterCloud' in container '/ouster/os_container': Component constructor threw an exception: tf_prefix
...
[component_container_mt-1] [ERROR] [1680715919.221592574] [ouster.os_sensor]: exception thrown while configuring the sensor, details: sensor_hostnameCurrent
|
|
All the reported issues are related to missing parameters For the current |
|
@crowncastlejsl I pushed a fix #100 which resolves an issue when running the driver on an arm processor. Would you mind giving it a try and see if it resolves the issue that you are seeing? |
|
@crowncastlejsl Just to make it easier for you, here is what your updated parameters should be like: ouster:
os_sensor:
ros__parameters:
sensor_hostname: 10.0.0.32
udp_dest: 10.0.0.60
mtp_dest: ''
mtp_main: false
lidar_mode: 2048x10
timestamp_mode: TIME_FROM_ROS_TIME
udp_profile_lidar: LEGACY
lidar_port: 7501
imu_port: 7502
metadata: ''
os_cloud:
ros__parameters:
tf_prefix: left_os_zero
timestamp_mode: TIME_FROM_ROS_TIME |
|
I merged my proposed fix, let me know if you are still observing the issue after updating to latest |
|
Thanks for your help with this issue. I apologize for not testing things out sooner - I've been away from the hardware for a little while. Unfortunately, the behavior is still the same as the original issue Platform:~$ uname -v
#76~20.04.1-Ubuntu SMP Mon Mar 20 15:54:19 UTC 2023~$ uname -i
x86_64
|
|
No worries, I will re-open the ticket till I have this figured out. Mind sharing your docker-engine version? |
|
Certainly: Docker Version~$ docker version
Client: Docker Engine - Community
Version: 23.0.3
API version: 1.42
Go version: go1.19.7
Git commit: 3e7cbfd
Built: Tue Apr 4 22:06:10 2023
OS/Arch: linux/amd64
Context: default
Server: Docker Engine - Community
Engine:
Version: 23.0.3
API version: 1.42 (minimum version 1.12)
Go version: go1.19.7
Git commit: 59118bf
Built: Tue Apr 4 22:06:10 2023
OS/Arch: linux/amd64
Experimental: true
containerd:
Version: 1.6.20
GitCommit: 2806fc1057397dbaeefbea0e4e17bddfbd388f38
runc:
Version: 1.1.5
GitCommit: v1.1.5-0-gf19387a
docker-init:
Version: 0.19.0
GitCommit: de40ad0 |
|
I am on the same docker version, so it isn't the issue. First I want to check if you had the chance to pull and try the code after we #103 and #106. There were changes that may help resolve the issue that you are facing. 🤞 In case this didn't work, I would kind ask if you could modify the But first try to pull the latest changes and check if it had helped at all. Thank you. |
|
Using the repo URL with the docker-compose -f ./docker/docker-compose.yml up --build
|
|
I did look briefly at your logs and compared them, the only difference I have observed that somehow within your sensor logs it stops updating and progressing after 14 ms, which makes me think that the timer used within the driver has stopped or got stuck. Can I ask how many cores you have on the machine you are running your docker compose on? I do have doubts that you have low CPU/RAM and it is not working nicely with the underlying RMW? I see that you are running with fastrtps, have you had the chance to test with any other RMW implementation? |
|
I don't think my system is the issue - the CPU is a 14-core Regarding the other things to try:
|
The choice of timestamp mode is irrelevant here. You can keep using TIME_FROM_ROS_TIME mode. I was referring to the ROS timer callbacks which the driver uses to receive and process packets. These are for some reasons appear to have stopped updating in your logs, but on my end they continue to update to the point I could see the sensor node gets configured and activated.
Based on your logs you are using FastRTPS, you could try to switch to CycloneDDS since it is free and reasonably easy to setup and use. Assuming that you have CycloneDDS properly installed on your system, you could then replace FastRTPS with CycloneDDS by declaring |
|
I cloned the repo down into a subdirectory ( Start of Dockerfile: Is there a way to otherwise inspect the ROS time to make sure it's incrementing appropriately in the container? |
|
Log statements in your logs (the ones you submitted earlier) stops being emitted after 14 ms. Specifically the message |
|
I reran the last test with the recent updates to the
Updated
|
|
I do get the same message about the executor so it is unlikely the culprit here. But I will re-examine the log files that you attached and try to find out what's going astray. PS. the error message doesn't happen when using FastRTPS |
|
Again, thanks for your help. Can you currently reproduce things on your side? Please let me know if there's anything else you'd like me to try. |
|
I re-generated the logs locally but (not using Docker this time) .. the only differences that I observed the following: my logs state: The other difference which I don't understand is that your logs doesn't display the firmware rev as I would expect, for example my logs look lile this Your logs only show the following: So unless that you explicitly edited these lines out then I am not sure how this could happen since it is a single log statement and should have information about the firmware rev. |
|
Sorry, there was a bug in my conversion script that was eliminating lines that didn't have a timestamp themselves, which would include any multi-line log statements. Just for reference, here is the script. Relative time conversion scriptimport re
from datetime import datetime
from pathlib import Path
regex = re.compile('\[(\d+\.\d+)\]')
def convert_timestamps(text):
start = None
for line in text.splitlines():
if (m := regex.search(line)):
ts = float(m.group(1))
dt = datetime.fromtimestamp(ts)
if start is None:
start = dt
new_line = regex.sub(f'[{dt-start}]', line)
yield new_line
else:
yield line
def convert_file(path: Path):
base = '_'.join(p.stem.split('_')[:2])
with p.with_name(f'{base}_rel_time.log').open('w') as file:
for line in convert_timestamps(p.read_text()):
file.write(f'{line}\n')
p = sorted(Path('logs').glob('os_sensor*.log'))[0]
print(p.name)
convert_file(p)
p = sorted(Path('logs').glob('os_cloud*.log'))[0]
print(p.name)
convert_file(p)Updated log files: The portion you mentioned is now there: [INFO] [0:00:01.805704] [ouster.os_sensor]: ouster client version: 0.8.1+unknown-release
product: OS-0-64-U02, sn: 122226001747, firmware rev: v2.3.1I notice that my firmware revision is behind yours. Does that need to be updated? |
No, the firmware 2.3.1 isn't way too behind and should work with the existing ROS2 driver |
|
Unfortunately the logs didn't expose any perceivable problem, I am unable to replicate the issue locally even though I used your scripts. Did you get to try FastRTPS and see if that works better with your setup by any chance? |
|
@crowncastlejsl I have recently merged a change that replaces the use of ROS timers for the polling data from the sensor with regular threads. It may or may not help with your specific case, there shouldn't be any changes to your build configuration so feel free to pull and try. |
|
Hi @crowncastlejsl, sorry it has been a while since we interacted.. I just wanted to notify you about a recent update made to the driver in which it improved many aspects of its behavior. One of which should address the issue you had with the driver earlier on. The new changes combined the three disjointed components that composed the driver into a single node. This should remove the intera-dependency within the driver on its components. The issue that you were having (although I couldn't produce in my tests) should be elevated by this changes. Please give it a try and let me know if we can close this ticket or you are still facing the issue. Note to use the combined mode, you will need to use |
|
Hi @Samahu, thanks for the update and for your help with this issue! I think this change you mentioned was the one to fix it. Everything seems to be working now. I think we can close. |
Describe the bug
Driver can connect to the sensor and successfully pulls the metadata, but it hangs while the sensor is activating. The topics are visible using RQT, but data never actually gets published.
Setup
docker-compose.yml- container runtime configurationouster_params.yml- sensor configurationros_entrypoint.sh- Docker container entrypointResults
Command
Output
Platform (please complete the following information):**
The text was updated successfully, but these errors were encountered: