Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cannot move GP4 + YRC1000micro with ROS using rostopic pub or MoveIt + RViz configuration #472

Closed
adamlouis opened this issue Mar 12, 2022 · 31 comments
Closed

Cannot move GP4 + YRC1000micro with ROS using rostopic pub or MoveIt + RViz configuration #472

adamlouis opened this issue Mar 12, 2022 · 31 comments
Labels
more-info-needed Waiting on additional information from poster

Comments

@adamlouis
Copy link

adamlouis commented Mar 12, 2022 •
edited
Loading

Hi! I am trying to get a new GP4 robot working with ROS + MoveIt using a YRC1000micro controller.

I can't get to the robot to move ... but I seem close. I would appreciate any help or debugging steps to get our robot moving.

We purchased our robot with the ROS package installed.

summary

  • I am running ros noetic on ubuntu 20.04 (LTS)
  • I can read the robot state & view it in Rviz
  • MoveIt commands from Rviz yield error:
    • Failed to initialize MotoRos motion, trajectory execution ABORTED. If safe, call the 'robot_enable' service to (re-)enable Motoplus motion and retry.
  • rostopic commands fail with the same error
  • I already called rosservice call /robot_enable "{}" and calling it again does not change the behavior
  • I can start servos & run jobs using the ethernet protocol

Why can't I move the robot?

What simple test can I run to verify configuration of remote control of the GP4 over ROS with the YRC1000?

details

1. set up motoman driver

First, I followed the instructions here: https://wiki.ros.org/motoman_driver/Tutorials/Usage

I start roscore in terminal 1

terminal 1

roscore

then, I load the gp4 urdf & joint names

command terminal

roslaunch motoman_gp4_support load_gp4.launch
rosparam set controller_joint_names "[joint_1_s,joint_2_l,joint_3_u,joint_4_r,joint_5_b,joint_6_t]"

then, I run the driver for our controller w/ the robot's pre-set IP

terminal 2

roslaunch motoman_driver robot_interface_streaming_yrc1000.launch robot_ip:=192.168.1.31

2. run a moveit config

next, I run my move it configuration.

I adapted my configuration from this one in motoman_experimental: https://github.com/ros-industrial/motoman_experimental/tree/kinetic-devel/motoman_mpl80_moveit_config

My modifications were:
- replace mpl80 with gp4
- replace motoman_mpl_support with motoman_gp4_support
- replace mpl80 urdf & srdf with gp4 urdf & srdf
- remove components that duplicate the interface streaming above
- remove fake_execution and sim ... because I am trying to test a real robot
- remove db ... don't need it for my test

I forked the motoman_experimental repo and created a branch here with my motoman_gp4_moveit_config configuration: https://github.com/adamlouis/motoman_experimental/tree/motoman_gp4_moveit_config/motoman_gp4_moveit_config

I would be happy to clean that up & find a way to include it in motoman_experimental when I figure out what's wrong.

then, I run it with

terminal 3

roslaunch motoman_gp4_moveit_config demo.launch

at this point, if I move the robot using our teach pendant, the movement is reflected in Rviz.

this is good & tells me that I am able to read state from the yrc1000micro.

3. prepare for execution

now, I am ready to move the robot

command terminal

rosservice call /robot_enable "{}"

I hear relays click and see the servo lights turn on

output in command terminal

success: True
message: "Motoman robot is now enabled and will accept motion commands."

output in terminal 2

Motoman robot is now enabled and will accept motion commands.

This is good - in theory, I can move the robot.

4. Try to move with RViz + MoveIt

I try to move the robot 2 ways & get the same behavior.

First, in Rviz + MoveIt, I set a goal position. I choose a goal position only small distance from the current position. I click "Plan" and "Execute".

On "Execute", I get the following output:

output in terminal 2

Failed to initialize MotoRos motion, trajectory execution ABORTED. If safe, call the 'robot_enable' service to (re-)enable Motoplus motion and retry.

output in terminal 3

Controller is taking too long to execute trajectory (the expected upper bound for the trajectory execution was 2.526993 seconds). Stopping trajectory.

If I run the command to enable the service

command terminal

rosservice call /robot_enable "{}"

& repeat the above steps, I get the same error.

5. Try to move with rostopic

It's possible I wrote a bad moveit config, so I try to run some trivial command to move the robot with ROS.

The easiest thing seemed like: rostopic pub /joint_trajectory_action/goal control_msgs/FollowJointTrajectoryActionGoal . Is there some simpler command to test "write" commands to the yrc1000 over ROS?

This command fails with the same errors as above:

output in terminal 2

Failed to initialize MotoRos motion, trajectory execution ABORTED. If safe, call the 'robot_enable' service to (re-)enable Motoplus motion and retry.

output in terminal 3

Controller is taking too long to execute trajectory (the expected upper bound for the trajectory execution was 2.526993 seconds). Stopping trajectory.

Here's my full command. The positions I provide are the robot's current positions.

rostopic pub /joint_trajectory_action/goal control_msgs/FollowJointTrajectoryActionGoal "header:
  seq: 0
  stamp:
    secs: 0
    nsecs: 0
  frame_id: ''
goal_id:
  stamp:
    secs: 0
    nsecs: 0
  id: ''
goal:
  trajectory:
    header:
      seq: 0
      stamp:
        secs: 0
        nsecs: 0
      frame_id: ''
    joint_names:
    - joint_1_s
    - joint_2_l
    - joint_3_u
    - joint_4_r
    - joint_5_b
    - joint_6_t
    points:
    - positions: [0.10212502628564835, 0.10033370554447174, 0.05954036861658096, 0.007940606214106083, -1.3190429210662842, 4.0864481925964355]
      velocities: [0,0,0,0,0,0]
      accelerations: [0,0,0,0,0,0,0]
      effort: [0,0,0,0,0,0]
      time_from_start: {secs: 0, nsecs: 0}
  path_tolerance:
  - {name: '', position: 0.0, velocity: 0.0, acceleration: 0.0}
  goal_tolerance:
  - {name: '', position: 0.0, velocity: 0.0, acceleration: 0.0}
  goal_time_tolerance: {secs: 0, nsecs: 0}"

Some notes:

  • INIT_ROS is the "Current Job" on the teach pendant
  • the green "Servo" light on the teach pendant is on
  • the key in the teach pendant is switched to remote mode
  • I can turn servos on and start jobs with the ethernet commands SVO and START - which implies to me remote control is ok
  • I have not confirmed that CMD REMOTE SEL is set to "valid" in the pseudo input display per page 22 / 291 of the "YRC1000micro OPTIONS INSTRUCTIONS for Ethernet Function". We have a different teach pendant than is referenced in the documentation & I have not found this setting. Is this related? I suspect I am ok, since I could start jobs over the ethernet protocol
  • I get 3 errors / warnings when starting robot_interface_streaming_yrc1000.launch. The warnings seem fine & the errors seems fine per @gavanderhoorn's reply in Failed to find topic_list parameter #242 & Failed to find topic_list parameter #235 ...
[ERROR] [1647042622.182945880]: Failed to find topic_list parameter
[ WARN] [1647042622.190244041]: Failed to get '~port' parameter: using default (50242)
[ERROR] [1647042622.194533748]: Failed to find topic_list parameter
[ WARN] [1647042622.210121014]: Tried to connect when socket already in connected state

Thanks for reading & thanks in advance for any help! Would appreciate any pointers to help get us up an running.
@marip8
Copy link
Member

marip8 commented Mar 14, 2022 •
edited
Loading

Thanks for the detailed issue explanation; I don't see any obvious issues with your approach or your MoveIt config.

MoveIt has some dynamic reconfigure parameters relating to trajectory execution that might be causing your issue. You might try turning off execution_duration_monitoring or increasing the allowed_execution_duration_scaling. Your planned trajectory should have time stamps associated with each waypoint (from the time parameterization stage of planning) and MoveIt does some monitoring to make sure that the trajectory execution on the controller actually completes within the amount of time the planned trajectory should have taken (with a scale factor applied to it). Disabling the execution_duration_monitoring circumvents this check, and increasing the allowed_execution_duration_scaling increases the scale factor for the trajectory time limit. Making these changes has resolved a similar issue for me recently, but I haven't delved into why this problem exists in the first place since my trajectory seemed to be time parameterized properly

image

@gavanderhoorn
Copy link
Member

gavanderhoorn commented Mar 14, 2022 •
edited
Loading

rostopic pub-ing to an action goal topic is not a good idea. Not just with motoman_driver, but in general.

I'd suggest using code similar to what is linked in #229.

I'm slightly confused as well: AFAIK and IIRC, goals with a JointTrajectory with a single point in them should not be accepted. motoman_driver absolutely needs two points: current state and destination point. The rostopic pub command line shown only includes a destination point.

Additionally, I'd suggest to use motoman_gp4_support which provides convenience .launch files which take care of everything wrt configuration of the driver for this particular robot variant.

Starting the driver then only needs:

roslaunch motoman_gp4_support robot_interface_streaming_gp4.launch robot_ip:=192.168.1.31
 controller:=yrc1000

nothing more is needed.

@adamlouis: please try again with just that robot_interface_streaming_gp4.launch file, nothing else.

And please start it using roslaunch --screen, and copy-paste the full output from the terminal into a comment here, after you've invoked robot_enable again and tried to execute a trajectory again (preferably using MoveIt or the action client script linked in #229, not rostopic).

I'd also suggest updating your ROSCONSOLE_FORMAT env var to '[${severity}] [${time}] [${node}] [${logger}]: ${message}', so it's clear which node printed which line.

Edit: and don't use demo.launch to control a real robot.

The MoveIt configuration you've shared includes a moveit_planning_execution.launch. That would be the proper entrypoint normally. I'm not sure how you've changed it, so it could be it's not in working order

@gavanderhoorn
Copy link
Member

@marip8 wrote:

MoveIt has some dynamic reconfigure parameters relating to trajectory execution that might be causing your issue. [...] Making these changes has resolved a similar issue for me recently, but I haven't delved into why this problem exists in the first place since my trajectory seemed to be time parameterized properly

If that was with motoman_driver, something is wrong, as that should not be necessary.

Unless perhaps the joint limits for the specific robot were not configured correctly and whatever parameterised the trajectory is expecting the robot to be able to move faster than it really can.

@marip8
Copy link
Member

marip8 commented Mar 14, 2022

If that was with motoman_driver, something is wrong, as that should not be necessary.

Agreed; I'm going to work with the robot more this week, so hopefully I can figure out what is going on. The fix I mentioned might at least get the robot moving in case there are other parameterization/joint velocity limit issues that are MoveIt/URDF related.

@adamlouis
Copy link
Author

adamlouis commented Mar 14, 2022 •
edited by gavanderhoorn
Loading

Thank you @gavanderhoorn & @marip8 for you quick replies and suggestions!

Here's what I've done & observed:

1. MoveIt duration params

You might try turning off execution_duration_monitoring or increasing the allowed_execution_duration_scaling

I did this by adding these 2 lines to my MoveIt launch file:

<param name="move_group/trajectory_execution/allowed_execution_duration_scaling" value="4.0" />
<param name="move_group/trajectory_execution/execution_duration_monitoring" value="false" />

When I do this, I do not get a timeout error, but the robot does not move.

2. use motoman_gp4_support launch files

Additionally, I'd suggest to use motoman_gp4_support which provides convenience .launch files which take care of everything wrt configuration of the driver for this particular robot variant.

Starting the driver then only needs:

roslaunch motoman_gp4_support robot_interface_streaming_gp4.launch robot_ip:=192.168.1.31 controller:=yrc1000

nothing more is needed.

done

  1. update console format

I'd also suggest updating your ROSCONSOLE_FORMAT env var to '[${severity}] [${time}] [${node}] [${logger}]: ${message}', so it's clear which node printed which line.

done

4. MoveIt config changes

Edit: and don't use demo.launch to control a real robot.
The MoveIt configuration you've shared includes a moveit_planning_execution.launch. That would be the proper entrypoint normally. I'm not sure how you've changed it, so it could be it's not in working order

Fair point - I will rename this to something appropriate moving forward.

Likewise, because my MoveIt configuration might be bad, I will use your simple_trajectory_action_client.py script for further tests.

5. Run with motoman_gp4_support launch files + your script

And please start it using roslaunch --screen, and copy-paste the full output from the terminal into a comment here....

Below are the outputs of the 2 terminals - the ROS driver terminal and the simple_trajectory_action_client.py terminal.

I updated the joint names in simple_trajectory_action_client.py to match the joint names of my robot.

Likewise, I ran the script several times with small modifications (e.g. move a different joint a different amount) but none worked.

Strangely, the Python script exits with success, but with the ROS log shows the same error line,

[ERROR] [1647298434.963551694] [/motion_streaming_interface] [ros.motoman_driver]: Failed to initialize MotoRos motion, trajectory execution ABORTED. If safe, call the 'robot_enable' service to (re-)enable Motoplus motion and retry.

At the bottom, it shows the line:

[ INFO] [1647298439.967259077] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action.controllerStateCB]: Inside goal constraints - stopped moving- return success for action

Which is strange, due to 1) the prior error and 2) the fact that the robot did not move

terminal 1: roslaunch --screen motoman_gp4_support robot_interface_streaming_gp4.launch robot_ip:=192.168.1.31 controller:=yrc1000

Click to expand 
$ roslaunch --screen motoman_gp4_support robot_interface_streaming_gp4.launch robot_ip:=192.168.1.31 controller:=yrc1000
... logging to /home/adam/.ros/log/6b59d292-a3e9-11ec-af51-5d9e0201ff92/roslaunch-adam-ubuntu-53282.log
Checking log directory for disk usage. This may take a while.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.

started roslaunch server http://adam-ubuntu:43285/

SUMMARY
========

PARAMETERS
 * /controller_joint_names: ['joint_1_s', 'jo...
 * /robot_ip_address: 192.168.1.31
 * /rosdistro: noetic
 * /rosversion: 1.15.14

NODES
  /
    io_relay (motoman_driver/io_relay)
    joint_state (motoman_driver/robot_state)
    joint_trajectory_action (industrial_robot_client/joint_trajectory_action)
    motion_streaming_interface (motoman_driver/motion_streaming_interface)

auto-starting new master
process[master]: started with pid [53321]
ROS_MASTER_URI=http://localhost:11311

setting /run_id to 6b59d292-a3e9-11ec-af51-5d9e0201ff92
process[rosout-1]: started with pid [53361]
started core service [/rosout]
process[joint_state-2]: started with pid [53364]
process[motion_streaming_interface-3]: started with pid [53368]
process[io_relay-4]: started with pid [53370]
process[joint_trajectory_action-5]: started with pid [53371]
[ INFO] [1647298347.184634433] [/joint_state] [ros.simple_message]: Added message handler for message type: 0
[ INFO] [1647298347.187798055] [/joint_state] [ros.motoman_driver]: Robot state connecting to IP address: '192.168.1.31:50241'
[ERROR] [1647298347.188439160] [/joint_state] [ros.motoman_driver]: Failed to find topic_list parameter
[ INFO] [1647298347.190032841] [/joint_state] [ros.industrial_utils]: Adding joint_1_s to list parameter
[ INFO] [1647298347.190067773] [/joint_state] [ros.industrial_utils]: Adding joint_2_l to list parameter
[ INFO] [1647298347.190083084] [/joint_state] [ros.industrial_utils]: Adding joint_3_u to list parameter
[ INFO] [1647298347.190094583] [/joint_state] [ros.industrial_utils]: Adding joint_4_r to list parameter
[ INFO] [1647298347.190105306] [/joint_state] [ros.industrial_utils]: Adding joint_5_b to list parameter
[ INFO] [1647298347.190116246] [/joint_state] [ros.industrial_utils]: Adding joint_6_t to list parameter
[ INFO] [1647298347.190140721] [/joint_state] [ros.industrial_utils]: Found user-specified joint names in 'controller_joint_names': [joint_1_s, joint_2_l, joint_3_u, joint_4_r, joint_5_b, joint_6_t]
[ INFO] [1647298347.190607239] [/joint_state] [ros.simple_message]: Connected to server
[ INFO] [1647298347.190642770] [/joint_state] [ros.simple_message]: Initializing message manager with default comms fault handler
[ INFO] [1647298347.190663735] [/joint_state] [ros.simple_message]: Default communications fault handler successfully initialized
[ INFO] [1647298347.190680806] [/joint_state] [ros.simple_message]: Initializing message manager
[ INFO] [1647298347.190701048] [/joint_state] [ros.simple_message]: Added message handler for message type: 1
[ INFO] [1647298347.193239179] [/joint_state] [ros.simple_message]: Added message handler for message type: 10
[ INFO] [1647298347.193701622] [/joint_state] [ros.simple_message]: Added message handler for message type: 15
[ INFO] [1647298347.193936324] [/motion_streaming_interface] [ros.motoman_driver]: Joint Trajectory Interface connecting to IP address: '192.168.1.31:50240'
[ WARN] [1647298347.194079574] [/io_relay] [ros.motoman_driver.io.init]: Failed to get '~port' parameter: using default (50242)
[ INFO] [1647298347.194338138] [/joint_state] [ros.simple_message]: Entering message manager spin loop
[ERROR] [1647298347.195537643] [/motion_streaming_interface] [ros.motoman_driver]: Failed to find topic_list parameter
[ INFO] [1647298347.196025652] [/io_relay] [ros.simple_message]: Connected to server
[ INFO] [1647298347.196308786] [/motion_streaming_interface] [ros.industrial_utils]: Adding joint_1_s to list parameter
[ INFO] [1647298347.196335098] [/motion_streaming_interface] [ros.industrial_utils]: Adding joint_2_l to list parameter
[ INFO] [1647298347.196351681] [/motion_streaming_interface] [ros.industrial_utils]: Adding joint_3_u to list parameter
[ INFO] [1647298347.196366141] [/motion_streaming_interface] [ros.industrial_utils]: Adding joint_4_r to list parameter
[ INFO] [1647298347.196383065] [/motion_streaming_interface] [ros.industrial_utils]: Adding joint_5_b to list parameter
[ INFO] [1647298347.196397435] [/motion_streaming_interface] [ros.industrial_utils]: Adding joint_6_t to list parameter
[ INFO] [1647298347.196418679] [/motion_streaming_interface] [ros.industrial_utils]: Found user-specified joint names in 'controller_joint_names': [joint_1_s, joint_2_l, joint_3_u, joint_4_r, joint_5_b, joint_6_t]
[ INFO] [1647298347.196436590] [/motion_streaming_interface] [ros.motoman_driver]: MotomanJointTrajectoryStreamer: init
[ INFO] [1647298347.196451851] [/motion_streaming_interface] [ros.motoman_driver]: JointTrajectoryStreamer: init
[ INFO] [1647298347.196746734] [/motion_streaming_interface] [ros.simple_message]: Connected to server
[ WARN] [1647298347.197159913] [/motion_streaming_interface] [ros.motoman_driver]: Unable to read velocity limits from 'robot_description' param.  Velocity validation disabled.
[ INFO] [1647298347.199751812] [/joint_trajectory_action] [ros.industrial_utils]: Adding joint_1_s to list parameter
[ INFO] [1647298347.200548324] [/joint_trajectory_action] [ros.industrial_utils]: Adding joint_2_l to list parameter
[ INFO] [1647298347.200565047] [/joint_trajectory_action] [ros.industrial_utils]: Adding joint_3_u to list parameter
[ INFO] [1647298347.200576109] [/joint_trajectory_action] [ros.industrial_utils]: Adding joint_4_r to list parameter
[ INFO] [1647298347.200586007] [/joint_trajectory_action] [ros.industrial_utils]: Adding joint_5_b to list parameter
[ INFO] [1647298347.200595394] [/joint_trajectory_action] [ros.industrial_utils]: Adding joint_6_t to list parameter
[ INFO] [1647298347.200612311] [/joint_trajectory_action] [ros.industrial_utils]: Found user-specified joint names in 'controller_joint_names': [joint_1_s, joint_2_l, joint_3_u, joint_4_r, joint_5_b, joint_6_t]
[ INFO] [1647298347.200633131] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Filtered joint names to 6 joints
[ INFO] [1647298347.202854303] [/motion_streaming_interface] [ros.motoman_driver]: Unlocking mutex
[ INFO] [1647298347.202914524] [/motion_streaming_interface] [ros.motoman_driver]: Starting Motoman joint trajectory streamer thread
[ INFO] [1647298347.208008060] [/motion_streaming_interface] [ros.motoman_driver]: Connecting to robot motion server
[ WARN] [1647298347.208033019] [/motion_streaming_interface] [ros.simple_message]: Tried to connect when socket already in connected state

[ WARN] [1647298388.531971119] [/motion_streaming_interface] [ros.motoman_driver]: Motoman robot is now enabled and will accept motion commands.

[ INFO] [1647298434.959602612] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Received new goal
[ INFO] [1647298434.959656093] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Publishing trajectory
[ INFO] [1647298434.959967726] [/motion_streaming_interface] [ros.motoman_driver]: Receiving joint trajectory message
[ERROR] [1647298434.963551694] [/motion_streaming_interface] [ros.motoman_driver]: Failed to initialize MotoRos motion, trajectory execution ABORTED. If safe, call the 'robot_enable' service to (re-)enable Motoplus motion and retry.
[ INFO] [1647298434.975282255] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298435.001286794] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298435.028803686] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298435.055601711] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298435.082140510] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298435.108650533] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298435.135041384] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298435.162159451] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298435.188905900] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298435.216550190] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298435.242780508] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298435.268879837] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298435.295539047] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298435.322248321] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298435.349102237] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298435.376684058] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298435.403537282] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298435.430806033] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298435.456939815] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298435.484267806] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298435.511258691] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298435.537291667] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298435.564528661] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298435.591329756] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298435.617849671] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298435.644477661] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298435.671872648] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298435.698306085] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298435.725477130] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298435.752301939] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298435.779548415] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298435.805874489] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298435.833276650] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298435.860251607] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298435.887337118] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298435.913151404] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298435.939842989] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298435.967217206] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298435.994118431] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298436.021437349] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298436.048077848] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298436.074274566] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298436.101025607] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298436.128105722] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298436.154253643] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298436.180386932] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298436.206255243] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298436.232766980] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298436.259924718] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298436.286555589] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298436.313560552] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298436.340344600] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298436.367745929] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298436.393934041] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298436.421619893] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298436.448299819] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298436.475355254] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298436.501351084] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298436.528121664] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298436.554262449] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298436.580262202] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298436.606431521] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298436.632564639] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298436.658484650] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298436.685035169] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298436.712117584] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298436.738450885] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298436.764850187] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298436.791476241] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298436.817691037] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298436.844245325] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298436.870302000] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298436.897187885] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298436.923694338] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298436.950168063] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298436.976419138] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298437.003536242] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298437.031253847] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298437.057672779] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298437.084606732] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298437.109900948] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298437.137142298] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298437.163803615] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298437.190628417] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298437.217124848] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298437.244268124] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298437.271487790] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298437.297850455] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298437.323929271] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298437.350128199] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298437.376166205] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298437.402366173] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298437.429401302] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298437.455745401] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298437.482037620] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298437.509243217] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298437.535677997] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298437.561533255] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298437.587994363] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298437.614257397] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298437.640767418] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298437.667777953] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298437.694034761] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298437.720785926] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298437.748370154] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298437.775601206] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298437.802566869] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298437.828742146] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298437.855879166] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298437.882397609] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298437.908750815] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298437.935197625] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298437.960852864] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298437.989396021] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298438.015669117] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298438.042167842] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298438.069136018] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298438.096235973] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298438.123296425] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298438.149574600] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298438.176174498] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298438.202080652] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298438.228797080] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298438.255543647] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298438.281334397] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298438.308175983] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298438.334267086] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298438.360369185] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298438.387260913] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298438.413880862] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298438.440918704] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298438.468138096] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298438.495743115] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298438.521706496] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298438.549170295] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298438.575519758] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298438.601535793] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298438.628312172] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298438.655258645] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298438.681601756] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298438.708356643] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298438.735632712] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298438.761657324] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298438.788209057] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298438.814598284] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298438.841213200] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298438.868219782] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298438.894256828] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298438.920613671] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298438.946138273] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298438.973239237] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298439.000421539] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298439.030329479] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298439.056596942] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298439.084242021] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298439.111413581] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298439.138029332] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298439.165591482] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298439.192382409] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298439.219608162] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298439.246233770] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298439.273145049] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298439.300270665] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298439.326470285] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298439.353417242] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298439.379720990] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298439.407385266] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298439.433316636] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298439.460897226] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298439.487847045] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298439.515507703] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298439.541349512] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298439.567955305] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298439.594453186] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298439.620919609] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298439.647749614] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298439.675298149] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298439.701256165] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298439.727695760] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298439.754164699] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298439.781093216] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298439.807805481] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298439.834129718] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298439.860679536] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298439.887384559] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298439.914229354] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298439.939980983] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory
[ INFO] [1647298439.967259077] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action.controllerStateCB]: Inside goal constraints - stopped moving-  return success for action

terminal 2: python3 simple_trajectory_action_client.py

[INFO] [1647298434.646652] [/simple_trajectory_action_client] [rosout]: Waiting for driver's action server to become available ..
[INFO] [1647298434.928927] [/simple_trajectory_action_client] [rosout]: Connected to trajectory action server
[INFO] [1647298434.958179] [/simple_trajectory_action_client] [rosout]: Submitting goal ..
[INFO] [1647298434.959258] [/simple_trajectory_action_client] [rosout]: Waiting for completion ..
[INFO] [1647298439.968406] [/simple_trajectory_action_client] [rosout]: Done.

Thanks again for your help!

@gavanderhoorn
Copy link
Member

I'm starting to suspect your YRC1000 is somehow not configured correctly.

Could you do one more test with the action client script, and make a wireshark trace of the network traffic between the YRC1000 and your ROS PC at the same time? Then please attach it here (make sure to compress it).

Additionally:

  1. could you tell us which INFORM job you've installed from here (there are a couple, and it matters)
  2. could you make a CMOS backup, zip it and attach it here?

@gavanderhoorn gavanderhoorn added the more-info-needed Waiting on additional information from poster label Mar 15, 2022
@ted-miller
Copy link
Contributor

Please also send that CMOS to the Partner Support team in the separate other email chain. They can check to ensure there isn't anything configured that would prevent robot motion.

@gavanderhoorn
Copy link
Member

There is a parallel support effort, next to this thread?

@adamlouis: could you please let us know when that has reached a conclusion?

I'll close this one for now, as I don't like potentially duplicating effort.

@ted-miller
Copy link
Contributor

@gavanderhoorn: I asked him to open this issue because I didn't see anything wrong with the description of his procedure. It appears that the robot-side is working ok. I figured there was something wrong in the support package that I'm not aware of.

I just wanted to have our support group check to make sure there isn't some safety-circuit configured improperly or something.

@adamlouis
Copy link
Author

Thanks all! I will work on this later today.

I'm starting to suspect your YRC1000 is somehow not configured correctly.

Could you do one more test with the action client script, and make a wireshark trace of the network traffic between the YRC1000 and your ROS PC at the same time? Then please attach it here (make sure to compress it).

Additionally:

could you tell us which INFORM job you've installed from here (there are a couple, and it matters)
could you make a CMOS backup, zip it and attach it here?

I will do this after I send the zip to Yaskawa/Motoman partner support & hear back

Please also send that CMOS to the Partner Support team in the separate other email chain. They can check to ensure there isn't anything configured that would prevent robot motion.

Will do later today

There is a parallel support effort, next to this thread?

@adamlouis: could you please let us know when that has reached a conclusion?

I'll close this one for now, as I don't like potentially duplicating effort.

As Ted mentioned - I emailed our contacts at Yaskawa. Over that email, I was asked to open the GitHub issue here. Later, I was asked for the same .zip you requested. I will send my zip to partner support & come back here if needed (or to report a resolution).

FYI - I have been able to get our robot up & running by programming against the YRC1000 ethernet interface. I suspect I will be able to do our initial task with just the ethernet interface. In any case, I would like to figure the ROS configuration out to support more sophisticated use cases as they emerge.

Thanks again!

@gavanderhoorn
Copy link
Member

@ted-miller wrote:

@gavanderhoorn: I asked him to open this issue because I didn't see anything wrong with the description of his procedure. It appears that the robot-side is working ok. I figured there was something wrong in the support package that I'm not aware of.

I just wanted to have our support group check to make sure there isn't some safety-circuit configured improperly or something.

This all makes complete sense, but please ask people to mention that in the initial post.

Especially when things like the CMOS settings haven't been checked yet, as that is almost impossible to diagnose from "the ROS side". Knowing that's being looked at would change the approach to diagnose what's going on.

@adamlouis wrote:

FYI - I have been able to get our robot up & running by programming against the YRC1000 ethernet interface. I suspect I will be able to do our initial task with just the ethernet interface. In any case, I would like to figure the ROS configuration out to support more sophisticated use cases as they emerge.

I would suggest to still create the wireshark trace.

It should not take very long, and there is a good chance we'll be able to diagnose what's going on from that -- provided the CMOS backup shows no problems with the configuration of the YRC1000u itself.

@gavanderhoorn
Copy link
Member

@adamlouis: let us know whether you'll still create the Wireshark trace and we'll re-open.

@ted-miller
Copy link
Contributor

The support guys found that the Inform job was incorrect. (Sorry, I should have caught that.)

image

For anyone experiencing similar issue in the future, please use the standard INIT_ROS jobs posted to GitHub. The motion API command will only work if the control-group is being blocked by a WAIT command. This is a safety mechanism to prevent conflicting motion from multiple points of control.

@gavanderhoorn
Copy link
Member

How did that happen?

Was the job not the one we host here?

@adamlouis
Copy link
Author

adamlouis commented Mar 16, 2022 •
edited
Loading

There were 2 issues. The first is resolved. For the second, I have a work around (or perhaps this is how it must be done).

1. error in INIT_ROS code

We purchased our GP4 / YRC1000 / SMart Pendant with ROS pre-configured. There was already a INIT_ROS job present when we received it.

When I sent the back up .zip file of our YRC1000 to the Motoman partner support team, they noticed the error Ted mentioned above.

I edited the job to match what they provided in the email, which matches the code here.

I (or anyone here) did not deliberately edit the INIT_ROS job after we received our shipment. Though, of course it is possible I did so in error / negligence.

2. INIT_ROS does not begin execution and block on a WAIT command except when run from "PLAY" mode

After editing the job to match the code referenced above, calling rosservice call /robot_enable "{}" when in REMOTE mode on the Smart Pendant did not advance the Job to the wait instruction.

Rather, I see "INIT_ROS" as the "Current Job" on the smart pendant home screen, but when I view the "Current Job" page, the job is at the "Start Job" instruction.

If I turn the pendant key to PLAY mode and run the INIT_ROS job manually from the pendant touch screen, the job executes up until the wait instruction.

Then, if I switch the pedant to REMOTE remote mode, run rosserivce call /robot_enable "{}", and move the robot in RViz / MoveIt with the config referenced in my initial post, the robot moves as expected.

Thank you all for you support & quick replies while figuring this out!

@adamlouis
Copy link
Author

I realized point 2 in my post above is true when starting other custom pendant jobs with the ethernet interface, not just when starting INIT_ROS with ROS. I had not observed this behavior when starting custom jobs over ethernet earlier.

It seems I must first go to PLAY mode before starting jobs in REMOTE mode.

In either case, I think I am unblocked & can figure this out.

Thanks all again for your help & for your work on this repo!

@gavanderhoorn
Copy link
Member

You should be able to start everything in REMOTE mode.

Something does not seem to be right still.

@ted-miller
Copy link
Contributor

We purchased our GP4 / YRC1000 / SMart Pendant with ROS pre-configured. There was already a INIT_ROS job present when we received it.

This sounds like the error came from our manufacturing area. I'll look into it.

It seems I must first go to PLAY mode before starting jobs in REMOTE mode.

This is not normal. Could you please use Wireshark to record the network traffic when you attempt robot_enable while in remote mode?

@gavanderhoorn gavanderhoorn reopened this Mar 17, 2022
@adamlouis
Copy link
Author

Thanks for the feedback!

I will try to post the wireshark output by EOD tomorrow.

Likewise, I will let you know if we figure out a more minimal set of steps reproduce the behavior.

@gavanderhoorn
Copy link
Member

Friendly ping @adamlouis

@adamlouis
Copy link
Author

@gavanderhoorn & @ted-miller - I've attached a zip of my wireshark traffic here. There was a fair amount of data constantly passing between my compute & the yrc1000, but the capture represents a 5-10s window during which I ran rosservice call /robot_enable "{}". The YRC is 192.168.1.31 and my computer is 192.168.1.32. Let me know if this is what you were looking for.

I can confirm that my ros terminal printed [ros.motoman_driver]: Motoman robot is now enabled and will accept motion commands. when I issued this command and the robot would not move via rviz/moveit after I did this.

I have consistently reproduced the behavior I described before & have a few more details. My flow is:

  • start withe the physical key on the smart pendant in TEACH mode
  • turn the physical key on the smart pendant to to REMOTE mode
  • start driver with roslaunch ...
  • bug: rosservice rosservice call /robot_enable ... claims to work here, but actually fails and the robot will not move with RViz / MoveIt ... this call is in my wireshark capture ... the INIT_ROS job is stuck at the first "Start Job" line
  • toggle the physical key on the smart pendant to PLAY mode ... you don't need to actually run anything ... this is what fixes things
  • toggle the physical key on the smart pendant to REMOTE mode
  • run rosservice rosservice call /robot_enable "{}" ... the INIT_ROS job proceeds to the "Wait ..." line as expected
  • rviz / moveit works!

The behavior above is true of other custom jobs I wrote and not just INIT_ROS. If I use the START command on the ASCII ethernet protocol of the yrc1000 to start my own pendant jobs, I observe the same behavior that those jobs do not start until I toggle the smart pendant to PLAY mode. Again, I don't need to do anything in PLAY mode - simply toggling there & back is enough to make remote invocation of jobs work as expected.

This seems to indicate the issue is with something on the YRC / pendant side vs. on the ROS driver side given that starting jobs in REMOTE mode fails similarly even when not using any of the code in this repo.

Thanks again! Let me know if you have any other experiments / evidence you'd like me to capture.

rosservice-enable.zip

@gavanderhoorn
Copy link
Member

Seems like MotoROS considers starting INIT_ROS a success:

ROS-Industrial SimpleMessage, Motoman Motion Reply (0x7d2), 76 bytes, little-endian
    Prefix
        Packet Length: 72
    Header
        Message Type: Motoman Motion Reply (0x000007d2)
        Communications Type: Service Reply (3)
        Reply Code: Success (1)
    Body
        Robot ID: -1
        Sequence Number: -1
        Command: Motion_Ctrl (2001)
        Result: Success/True (0)
        Subcode: Unknown (0)
        Data (reserved)
            J0:    0,000000000
            J1:    0,000000000
            J2:    0,000000000
            J3:    0,000000000
            J4:    0,000000000
            J5:    0,000000000
            J6:    0,000000000
            J7:    0,000000000
            J8:    0,000000000
            J9:    0,000000000

does imply there is something not entirely correct on the YRC1000 side.

the capture represents a 5-10s window during which I ran rosservice call /robot_enable "{}".

Could have been informative if you'd included the traffic when you're attempting to execute a trajectory, but I'm sort-of expecting to just see acceptance of traj pts until the buffer is full and then nothing.

@EricMarcil
Copy link
Contributor

EricMarcil commented Mar 22, 2022 •
edited by gavanderhoorn
Loading

@adamlouis Can you check that this installation step was done properly:

Controller Configuration

Verify that the controller is enabled to receive remote commands in Remote mode:

Using standard pendant:

Rotate the pendant key-switch (upper left of pendant) fully counter-clockwise into TEACH mode.

Upgrade your Security Level to MANAGEMENT

Touching [System Info]→[Security].

Default password is all 9's.
Using the pendant, select [In/OUT]→[PSEUDO INPUT SIG]

For DX100, DX200, and FS100, set input #82015 (CMD REMOTE SEL) to ON. The signal number is #87015 on YRC.

Move the cursor to the circle beside the specified input
Press [INTERLOCK]+[SELECT] to turn on the input

Rotate the pendant key-switch (upper left of pendant) fully clockwise into REMOTE mode.

@EricMarcil
Copy link
Contributor

EricMarcil commented Mar 22, 2022 •
edited by gavanderhoorn
Loading

@adamlouis Also check the Operate Enable Settings:

Upgrade your Security Level to MANAGEMENT (as above)
Touching [SETUP]→[OPERATE ENABLE].
Make sure that all the operations are set to PERMIT

@ted-miller
Copy link
Contributor

@adamlouis
The wireshark indicates that the robot does briefly become "ready", but then becomes "not ready" 124 ms after the robot_enable service is called. I think there is still something wrong with the robot job.

Could you please save out a fresh copy of the following files and attach them here. Be sure to zip them; they compress extremely well.

INIT_ROS.JBI
PANELBOX.LOG
ALL.PRM
CMOS.BIN

@adamlouis
Copy link
Author

Could you please save out a fresh copy of the following files...

done!

Using the pendant, select [In/OUT]→[PSEUDO INPUT SIG]
For DX100, DX200, and FS100, set input #82015 (CMD REMOTE SEL) to ON. The signal number is #87015 on YRC.
Move the cursor to the circle beside the specified input
Press [INTERLOCK]+[SELECT] to turn on the input

Touching [SETUP]→[OPERATE ENABLE].
Make sure that all the operations are set to PERMIT

@EricMarcil -- I referenced this in my initial post & the private email thread, but I am using a smart pendant and not a standard pendant. I have not found PSUEDO INPUT SIG, CMD REMOTE SEL, INTERLOCK, or SETUP on the smart pendant and have not seen a value of 87015 (or values w/ this order of magnitude) on any IO screen.

Do you have documentation to accomplish this with the smart pendant rather than the standard pendant?

Are you able to read the necessary data from the system backup rather than the pendant UI?

For my own reference -- are the smart pendant & standard pendant two clients of the same controller core, or do the two pendants require different controller configuration?

Thank you!

YC1000-0322-GitHub.zip

@gavanderhoorn
Copy link
Member

It's (almost certainly) not the cause of the problem, but I notice you're running MotoROS v1.9.0.

That's rather old, as the current version would be v1.9.10.

You might want to upgrade. The binaries can be found here for the YRC1000u.

@ted-miller
Copy link
Contributor

I have not found PSUEDO INPUT SIG, CMD REMOTE SEL, INTERLOCK, or SETUP on the smart pendant and have not seen a value of 87015 (or values w/ this order of magnitude) on any IO screen. Do you have documentation to accomplish this with the smart pendant rather than the standard pendant?

This step is not required for the Smart Pendant. The signal must already be enabled for the SP to work. I have updated the installation instructions to hopefully make this clearer.

For my own reference -- are the smart pendant & standard pendant two clients of the same controller core, or do the two pendants require different controller configuration?

They are two clients of the same controller core.

Are you able to read the necessary data from the system backup rather than the pendant UI?

Yes. I have found the problem. The job is set to "step" mode where it only executes one step at a time. (I didn't know this was a thing, nor can I explain why it behaves differently in PLAY mode.)

Smart Pendant instructions:
(The installation wiki has been updated with instructions for both pendant types.)

  1. Rotate the pendant key-switch (top of pendant) to the middle position for PLAY mode.

  2. Touch [MENU]→[Job List]

  3. Select the INIT_ROS job from the list and touch the [RUN] button in the lower left.

  4. In the ADVANCED RUN pane, change the Mode to One Cycle.

  5. Rotate the pendant key-switch (top of pendant) fully clockwise into REMOTE mode.

@ted-miller
Copy link
Contributor

@adamlouis: I just learned some new information about this issue. The instructions above are not sufficient long-term. It will automatically switch back to STEP mode when you toggle the key switch.

Touch MENU > System Settings > Classic Interface. Start the Classic Interface and press the Connect button.
image

From the main menu, touch Setup > Operate Cond. Cursor down to the Cycle Switch In xxxx Mode fields. Change all of these to CYCLE.
image

The touch the power-symbol button in the top right to return back to the Smart Pendant interface.

@gavanderhoorn
Copy link
Member

@adamlouis: have you had a chance to figure this out? The last comment(s) by @ted-miller should help with the STEP mode problem.

@gavanderhoorn
Copy link
Member

Assuming this has been fixed, I'm going to close the issue.

Please let us know your current status @adamlouis.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
more-info-needed Waiting on additional information from poster
Milestone
No milestone
Development

No branches or pull requests
5 participants
@adamlouis @gavanderhoorn @ted-miller @EricMarcil @marip8