Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

improve behavior if the process server itself is frozen / non-reachable #136

Open
doudou opened this issue Nov 16, 2017 · 1 comment
Open

improve behavior if the process server itself is frozen / non-reachable #136

doudou opened this issue Nov 16, 2017 · 1 comment

Comments

@doudou
Copy link
Member

doudou commented Nov 16, 2017 •
edited
Loading

Syskit currently assumes that the process server does not misbehave. This leads to hard crashes if it dies, or if it's frozen for any reason. Note that the only sane thing to do in this case is to quit, but it should be done in a more orderly manner
@doudou
Copy link
Member Author

doudou commented Nov 16, 2017 

Roby[FATAL]:   putting OroGen.apriltags.Task:0x2b054a8<id:178>(conf: ["default"], orocos_name: apriltags) in quarantine: OroGen.apriltags.Task:0x2b054a8<id:178>(conf: ["default"], orocos_name: apriltags) failed to emit
Roby[FATAL]:   the error is:
Roby[FATAL]:   failed emission of the interrupt event of OroGen.apriltags.Task:0x2b054a8 (Roby::EmissionFailed)
Roby[FATAL]:   = Backtrace
Roby[FATAL]:   = No backtrace
Roby[FATAL]:   failed to stop the '/apriltags' task of type apriltags::Task (Orocos::StateTransitionFailed)
Roby[FATAL]:   = Backtrace
Roby[FATAL]:   = 
Roby[FATAL]:   
Roby[FATAL]:     /home/saipem/rock/dev/tools/orocos.rb/lib/orocos/task_context.rb:94:in `do_stop'
Roby[FATAL]:     /home/saipem/rock/dev/tools/orocos.rb/lib/orocos/task_context.rb:94:in `block in stop'
Roby[FATAL]:     /home/saipem/rock/dev/tools/orocos.rb/lib/orocos/corba.rb:116:in `refine_exceptions'
Roby[FATAL]:     /home/saipem/rock/dev/tools/orocos.rb/lib/orocos/task_context.rb:92:in `stop'
Roby[FATAL]:     /home/saipem/rock/dev/tools/syskit/lib/syskit/task_context.rb:879:in `stop_orocos_task'
Roby[FATAL]:     /home/saipem/rock/dev/tools/syskit/lib/syskit/task_context.rb:908:in `block (2 levels) in <class:TaskContext>'
Roby[FATAL]:     /home/saipem/rock/dev/tools/roby/lib/roby/promise.rb:135:in `block in run_one_pipeline_segment'
Roby[FATAL]:     /home/saipem/rock/dev/tools/roby/lib/roby/droby/event_logging.rb:22:in `log_timepoint_group'
Roby[FATAL]:     /home/saipem/rock/dev/tools/roby/lib/roby/promise.rb:134:in `run_one_pipeline_segment'
Roby[FATAL]:     /home/saipem/rock/dev/tools/roby/lib/roby/promise.rb:96:in `run_pipeline_elements'
Roby[FATAL]:     /home/saipem/rock/dev/tools/roby/lib/roby/promise.rb:80:in `block in run_pipeline'
Roby[FATAL]:     /home/saipem/rock/dev/tools/roby/lib/roby/droby/event_logging.rb:22:in `log_timepoint_group'
Roby[FATAL]:     /home/saipem/rock/dev/tools/roby/lib/roby/promise.rb:78:in `run_pipeline'
Roby[FATAL]:     /home/saipem/.autoproj/gems/ruby/2.3.0/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/safe_task_executor.rb:24:in `block in execute'
Roby[FATAL]:     /home/saipem/.autoproj/gems/ruby/2.3.0/gems/concurrent-ruby-1.0.5/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `block in synchronize'
Roby[FATAL]:     /home/saipem/.autoproj/gems/ruby/2.3.0/gems/concurrent-ruby-1.0.5/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `synchronize'
Roby[FATAL]:     /home/saipem/.autoproj/gems/ruby/2.3.0/gems/concurrent-ruby-1.0.5/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `synchronize'
Roby[FATAL]:     /home/saipem/.autoproj/gems/ruby/2.3.0/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/safe_task_executor.rb:19:in `execute'
Roby[FATAL]:     /home/saipem/.autoproj/gems/ruby/2.3.0/gems/concurrent-ruby-1.0.5/lib/concurrent/promise.rb:531:in `block in realize'
Roby[FATAL]:     /home/saipem/.autoproj/gems/ruby/2.3.0/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:348:in `run_task'
Roby[FATAL]:     /home/saipem/.autoproj/gems/ruby/2.3.0/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:337:in `block (3 levels) in create_worker'
Roby[FATAL]:     /home/saipem/.autoproj/gems/ruby/2.3.0/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in `loop'
Roby[FATAL]:     /home/saipem/.autoproj/gems/ruby/2.3.0/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in `block (2 levels) in create_worker'
Roby[FATAL]:     /home/saipem/.autoproj/gems/ruby/2.3.0/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:in `catch'
Roby[FATAL]:     /home/saipem/.autoproj/gems/ruby/2.3.0/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:in `block in create_worker'
Roby[FATAL]:   = 
Roby[FATAL]:   putting Deployments::TransformerBroadcaster:0x3eebda0<id:78>(log: true, logger_task: , name_mappings: {"transformer_broadcaster"=>"transformer_broadcaster", "transformer_broadcaster_Logger"=>"transformer_broadcaster_Logger"}, on: localhost, process_name: transformer_broadcaster, ready_polling_period: 0.1, spawn_options: {:working_directory=>"/home/saipem/rock/dev/bundles/ucs", :output=>nil, :valgrind=>nil, :gdb=>nil, :cmdline_args=>{}, :wait=>false, :log_level=>nil, :oro_logfile=>"orocos.%m-%p.txt"}) in quarantine: Deployments::TransformerBroadcaster:0x3eebda0<id:78>(log: true, logger_task: , name_mappings: {"transformer_broadcaster"=>"transformer_broadcaster", "transformer_broadcaster_Logger"=>"transformer_broadcaster_Logger"}, on: localhost, process_name: transformer_broadcaster, ready_polling_period: 0.1, spawn_options: {:working_directory=>"/home/saipem/rock/dev/bundles/ucs", :output=>nil, :valgrind=>nil, :gdb=>nil, :cmdline_args=>{}, :wait=>false, :log_level=>nil, :oro_logfile=>"orocos.%m-%p.txt"}) failed to emit
Roby[FATAL]:   the error is:
Roby[FATAL]:   failed emission of the stop event of Deployments::TransformerBroadcaster:0x3eebda0 (Roby::EmissionFailed)
Roby[FATAL]:   = Backtrace
Roby[FATAL]:   = No backtrace
Roby[FATAL]:   reached timeout of 10s in #wait_for_answer (Orocos::RemoteProcesses::Client::TimeoutError)
Roby[FATAL]:   = Backtrace
Roby[FATAL]:   = No backtrace
Roby[WARN]: 2 handled errors
Roby[WARN]: 2 unhandled fatal exceptions, involving 2 tasks that will be forcefully killed
Roby[WARN]: encountered a fatal exception
Roby[WARN]: The following task is being terminated because of an internal error
Roby[WARN]: It has been put under quarantine
Roby[WARN]: Roby::TaskEmergencyTermination: Roby::TaskEmergencyTermination
Roby[WARN]: OroGen.apriltags.Task:0x2b054a8
Roby[WARN]:   owners: 
Roby[WARN]:   arguments: 
Roby[WARN]:     orocos_name: "apriltags",
Roby[WARN]:     conf: ["default"]
Roby[WARN]: = Backtrace
Roby[WARN]: = No backtrace
Roby[WARN]: failed emission of the interrupt event of OroGen.apriltags.Task:0x2b054a8 (Roby::EmissionFailed)
Roby[WARN]: = Backtrace
Roby[WARN]: = No backtrace
Roby[WARN]: failed to stop the '/apriltags' task of type apriltags::Task (Orocos::StateTransitionFailed)
Roby[WARN]: = Backtrace
Roby[WARN]: = 
Roby[WARN]: 
Roby[WARN]:   /home/saipem/rock/dev/tools/orocos.rb/lib/orocos/task_context.rb:94:in `do_stop'
Roby[WARN]:   /home/saipem/rock/dev/tools/orocos.rb/lib/orocos/task_context.rb:94:in `block in stop'
Roby[WARN]:   /home/saipem/rock/dev/tools/orocos.rb/lib/orocos/corba.rb:116:in `refine_exceptions'
Roby[WARN]:   /home/saipem/rock/dev/tools/orocos.rb/lib/orocos/task_context.rb:92:in `stop'
Roby[WARN]:   /home/saipem/rock/dev/tools/syskit/lib/syskit/task_context.rb:879:in `stop_orocos_task'
Roby[WARN]:   /home/saipem/rock/dev/tools/syskit/lib/syskit/task_context.rb:908:in `block (2 levels) in <class:TaskContext>'
Roby[WARN]:   /home/saipem/rock/dev/tools/roby/lib/roby/promise.rb:135:in `block in run_one_pipeline_segment'
Roby[WARN]:   /home/saipem/rock/dev/tools/roby/lib/roby/droby/event_logging.rb:22:in `log_timepoint_group'
Roby[WARN]:   /home/saipem/rock/dev/tools/roby/lib/roby/promise.rb:134:in `run_one_pipeline_segment'
Roby[WARN]:   /home/saipem/rock/dev/tools/roby/lib/roby/promise.rb:96:in `run_pipeline_elements'
Roby[WARN]:   /home/saipem/rock/dev/tools/roby/lib/roby/promise.rb:80:in `block in run_pipeline'
Roby[WARN]:   /home/saipem/rock/dev/tools/roby/lib/roby/droby/event_logging.rb:22:in `log_timepoint_group'
Roby[WARN]:   /home/saipem/rock/dev/tools/roby/lib/roby/promise.rb:78:in `run_pipeline'
Roby[WARN]:   /home/saipem/.autoproj/gems/ruby/2.3.0/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/safe_task_executor.rb:24:in `block in execute'
Roby[WARN]:   /home/saipem/.autoproj/gems/ruby/2.3.0/gems/concurrent-ruby-1.0.5/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `block in synchronize'
Roby[WARN]:   /home/saipem/.autoproj/gems/ruby/2.3.0/gems/concurrent-ruby-1.0.5/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `synchronize'
Roby[WARN]:   /home/saipem/.autoproj/gems/ruby/2.3.0/gems/concurrent-ruby-1.0.5/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `synchronize'
Roby[WARN]:   /home/saipem/.autoproj/gems/ruby/2.3.0/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/safe_task_executor.rb:19:in `execute'
Roby[WARN]:   /home/saipem/.autoproj/gems/ruby/2.3.0/gems/concurrent-ruby-1.0.5/lib/concurrent/promise.rb:531:in `block in realize'
Roby[WARN]:   /home/saipem/.autoproj/gems/ruby/2.3.0/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:348:in `run_task'
Roby[WARN]:   /home/saipem/.autoproj/gems/ruby/2.3.0/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:337:in `block (3 levels) in create_worker'
Roby[WARN]:   /home/saipem/.autoproj/gems/ruby/2.3.0/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in `loop'
Roby[WARN]:   /home/saipem/.autoproj/gems/ruby/2.3.0/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in `block (2 levels) in create_worker'
Roby[WARN]:   /home/saipem/.autoproj/gems/ruby/2.3.0/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:in `catch'
Roby[WARN]:   /home/saipem/.autoproj/gems/ruby/2.3.0/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:in `block in create_worker'
Roby[WARN]: = 
Roby[WARN]: the exception involved
Roby[WARN]:   OroGen.apriltags.Task:0x2b054a8<id:178>(conf: ["default"], orocos_name: apriltags)
Roby[WARN]: encountered a fatal exception
Roby[WARN]: The following task is being terminated because of an internal error
Roby[WARN]: It has been put under quarantine
Roby[WARN]: Roby::TaskEmergencyTermination: Roby::TaskEmergencyTermination
Roby[WARN]: Deployments::TransformerBroadcaster:0x3eebda0
Roby[WARN]:   owners: 
Roby[WARN]:   arguments: 
Roby[WARN]:     on: "localhost",
Roby[WARN]:     process_name: "transformer_broadcaster",
Roby[WARN]:     name_mappings: {"transformer_broadcaster"=>"transformer_broadcaster",
Roby[WARN]:      "transformer_broadcaster_Logger"=>"transformer_broadcaster_Logger"},
Roby[WARN]:     spawn_options: {:working_directory=>"/home/saipem/rock/dev/bundles/ucs",
Roby[WARN]:      :output=>nil,
Roby[WARN]:      :valgrind=>nil,
Roby[WARN]:      :gdb=>nil,
Roby[WARN]:      :cmdline_args=>{},
Roby[WARN]:      :wait=>false,
Roby[WARN]:      :log_level=>nil,
Roby[WARN]:      :oro_logfile=>"orocos.%m-%p.txt"},
Roby[WARN]:     log: true,
Roby[WARN]:     ready_polling_period: 0.1,
Roby[WARN]:     logger_task: nil
Roby[WARN]: = Backtrace
Roby[WARN]: = No backtrace
Roby[WARN]: failed emission of the stop event of Deployments::TransformerBroadcaster:0x3eebda0 (Roby::EmissionFailed)
Roby[WARN]: = Backtrace
Roby[WARN]: = No backtrace
Roby[WARN]: reached timeout of 10s in #wait_for_answer (Orocos::RemoteProcesses::Client::TimeoutError)
Roby[WARN]: = Backtrace
Roby[WARN]: = No backtrace
Roby[WARN]: the exception involved
Roby[WARN]:   Deployments::TransformerBroadcaster:0x3eebda0<id:78>(log: true, logger_task: , name_mappings: {"transformer_broadcaster"=>"transformer_broadcaster", "transformer_broadcaster_Logger"=>"transformer_broadcaster_Logger"}, on: localhost, process_name: transformer_broadcaster, ready_polling_period: 0.1, spawn_options: {:working_directory=>"/home/saipem/rock/dev/bundles/ucs", :output=>nil, :valgrind=>nil, :gdb=>nil, :cmdline_args=>{}, :wait=>false, :log_level=>nil, :oro_logfile=>"orocos.%m-%p.txt"})
Roby[WARN]: OroGen.apriltags.Task:0x2b054a8
Roby[WARN]:   owners: 
Roby[WARN]:   arguments: 
Roby[WARN]:     orocos_name: "apriltags",
Roby[WARN]:     conf: ["default"]
Roby[WARN]: Deployments::TransformerBroadcaster:0x3eebda0
Roby[WARN]:   owners: 
Roby[WARN]:   arguments: 
Roby[WARN]:     on: "localhost",
Roby[WARN]:     process_name: "transformer_broadcaster",
Roby[WARN]:     name_mappings: {"transformer_broadcaster"=>"transformer_broadcaster",
Roby[WARN]:      "transformer_broadcaster_Logger"=>"transformer_broadcaster_Logger"},
Roby[WARN]:     spawn_options: {:working_directory=>"/home/saipem/rock/dev/bundles/ucs",
Roby[WARN]:      :output=>nil,
Roby[WARN]:      :valgrind=>nil,
Roby[WARN]:      :gdb=>nil,
Roby[WARN]:      :cmdline_args=>{},
Roby[WARN]:      :wait=>false,
Roby[WARN]:      :log_level=>nil,
Roby[WARN]:      :oro_logfile=>"orocos.%m-%p.txt"},
Roby[WARN]:     log: true,
Roby[WARN]:     ready_polling_period: 0.1,
Roby[WARN]:     logger_task: nil
rock_pipe_vision[INFO]: stopped OroGen.image_enhancement.Task:0x2bafdb8<id:175>(conf: ["default"], orocos_name: image_enhancement)
Roby[FATAL]: Application error in update_deployment_states
Roby[FATAL]: unexpected message Y from process server (RuntimeError)
Roby[FATAL]: = Backtrace
Roby[FATAL]: = No backtrace
Roby[FATAL]: Execution thread FORCEFULLY quitting because of unhandled exception
Roby[FATAL]: in update_deployment_states: unexpected message Y from process server (RuntimeError)
Roby[FATAL]: = Backtrace
Roby[FATAL]: = No backtrace
Roby[WARN]: the following tasks are still present in the plan:
Roby[WARN]:   Deployments::OrogenDefaultApriltags__Task:0x2b45a80<id:184>(log: true, logger_task: , name_mappings: {"orogen_default_apriltags__Task"=>"apriltags", "orogen_default_apriltags__Task_Logger"=>"apriltags_Logger"}, on: localhost, process_name: apriltags, ready_polling_period: 0.1, spawn_options: {:working_directory=>"/home/saipem/rock/dev/bundles/ucs", :output=>nil, :valgrind=>nil, :gdb=>nil, :cmdline_args=>{}, :wait=>false, :log_level=>nil, :oro_logfile=>"orocos.%m-%p.txt"})
Roby[WARN]:   OroGen.apriltags.Task:0x2b054a8<id:178>(conf: ["default"], orocos_name: apriltags)
Killed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@doudou