Crates using tokio don't compose with each other

[masonk]

This is a followup on tokio-rs/tokio-core#319, an issue which hasn't gone away in the new branch:

Trying to start a runtime from a thread that already has a runtime will result in a panic.

This bug comes up very naturally when using an asynchronous db driver and an asynchronous webserver that are both on tokio. For example, anyone trying to combine the ldap3 + actix-web crates will instantly hit this. But so will anyone using any tokio-based webserver with any tokio-based DB driver.

Here's a minimal repro:
https://github.com/masonk/tokio-crash/tree/master/src

Note that this issue might have complications due to the fact that ldap3 is still on tokio-core, but actix-web isn't. However, if I am not mistaken, the same issue could have happened if both crates were on tokio.

This kind of thing seems like a big problem that will get bigger as the number of crates using tokio increases.

1. How should library authors structure their code so that they play nicely with other crates using tokio, and with their callers' use of tokio?

• E.g., what do you think the author of ldap3 should have done, or should do, here?
• And/or the author of actix?

2. What, if anything, can tokio do to make it easier for library authors to write composable crates?

Stack trace for the repro:

thread 'arbiter:164fcee9-47c5-41ae-99d5-fc4eff9acf04:actor' panicked at 'cannot recursively call into `Core`', libcore/option.rs:1010:5
thread 'arbiter:ce202f9a-329a-4b16-b044-f49ad0315167:actor' panicked at 'cannot recursively call into `Core`', libcore/option.rs:1010:5
thread 'arbiter:13a5003c-8713-4fad-9609-aa6e4d02af3f:actor' panicked at 'cannot recursively call into `Core`', libcore/option.rs:1010:5
thread 'arbiter:7cfaff5d-11b0-4b40-874e-42530e4a5384:actor' panicked at 'cannot recursively call into `Core`', libcore/option.rs:1010:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
stack backtrace:
   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
             at libstd/sys/unix/backtrace/tracing/gcc_s.rs:49
   1: std::sys_common::backtrace::print
             at libstd/sys_common/backtrace.rs:71
             at libstd/sys_common/backtrace.rs:59
   2: std::panicking::default_hook::{{closure}}
             at libstd/panicking.rs:211
   3: std::panicking::default_hook
             at libstd/panicking.rs:227
   4: std::panicking::rust_panic_with_hook
             at libstd/panicking.rs:477
   5: std::panicking::continue_panic_fmt
             at libstd/panicking.rs:391
   6: rust_begin_unwind
             at libstd/panicking.rs:326
   7: core::panicking::panic_fmt
             at libcore/panicking.rs:77
   8: core::option::expect_failed
             at libcore/option.rs:1010
   9: <core::option::Option<T>>::expect
             at libcore/option.rs:322
  10: tokio_core::reactor::Core::run
             at /home/mason/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-core-0.1.17/src/reactor/mod.rs:224
  11: ldap3::conn::LdapConn::with_settings
             at /home/mason/.cargo/registry/src/github.com-1ecc6299db9ec823/ldap3-0.6.0/src/conn.rs:211
  12: ldap3::conn::LdapConn::new
             at /home/mason/.cargo/registry/src/github.com-1ecc6299db9ec823/ldap3-0.6.0/src/conn.rs:203
  13: tokio_crash::main::{{closure}}
             at src/main.rs:8
  14: <actix_web::server::http::HttpServer<H>>::new::{{closure}}
             at /home/mason/.cargo/registry/src/github.com-1ecc6299db9ec823/actix-web-0.7.8/src/server/http.rs:66
  15: <actix_web::server::http::HttpService<H> as actix_web::server::server::Service>::create
             at /home/mason/.cargo/registry/src/github.com-1ecc6299db9ec823/actix-web-0.7.8/src/server/http.rs:446
  16: <alloc::boxed::Box<(dyn actix_web::server::server::Service + 'static)> as actix_web::server::server::Service>::create
             at /home/mason/.cargo/registry/src/github.com-1ecc6299db9ec823/actix-web-0.7.8/src/server/server.rs:38
  17: actix_web::server::server::Server::start_worker::{{closure}}::{{closure}}
             at /home/mason/.cargo/registry/src/github.com-1ecc6299db9ec823/actix-web-0.7.8/src/server/server.rs:265
  18: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &'a mut F>::call_once
             at libcore/ops/function.rs:286
  19: <core::option::Option<T>>::map
             at libcore/option.rs:424
  20: <core::iter::Map<I, F> as core::iter::iterator::Iterator>::next
             at libcore/iter/mod.rs:1394
  21: <alloc::vec::Vec<T> as alloc::vec::SpecExtend<T, I>>::spec_extend
             at liballoc/vec.rs:1903
  22: <alloc::vec::Vec<T> as alloc::vec::SpecExtend<T, I>>::from_iter
             at liballoc/vec.rs:1886
  23: <alloc::vec::Vec<T> as core::iter::traits::FromIterator<T>>::from_iter
             at liballoc/vec.rs:1772
  24: core::iter::iterator::Iterator::collect
             at libcore/iter/iterator.rs:1415
  25: actix_web::server::server::Server::start_worker::{{closure}}
             at /home/mason/.cargo/registry/src/github.com-1ecc6299db9ec823/actix-web-0.7.8/src/server/server.rs:263
  26: actix::arbiter::Arbiter::start_with_builder::{{closure}}
             at /home/mason/.cargo/registry/src/github.com-1ecc6299db9ec823/actix-0.7.4/src/arbiter.rs:229
  27: <F as actix::msgs::FnExec<I, E>>::call_box
             at /home/mason/.cargo/registry/src/github.com-1ecc6299db9ec823/actix-0.7.4/src/msgs.rs:115
  28: <actix::msgs::Execute<I, E>>::exec
             at /home/mason/.cargo/registry/src/github.com-1ecc6299db9ec823/actix-0.7.4/src/msgs.rs:99
  29: <actix::arbiter::Arbiter as actix::handler::Handler<actix::msgs::Execute<I, E>>>::handle
             at /home/mason/.cargo/registry/src/github.com-1ecc6299db9ec823/actix-0.7.4/src/arbiter.rs:264
  30: <actix::address::envelope::SyncEnvelopeProxy<A, M> as actix::address::envelope::EnvelopeProxy>::handle
             at /home/mason/.cargo/registry/src/github.com-1ecc6299db9ec823/actix-0.7.4/src/address/envelope.rs:106
  31: <actix::address::envelope::Envelope<A> as actix::address::envelope::EnvelopeProxy>::handle
             at /home/mason/.cargo/registry/src/github.com-1ecc6299db9ec823/actix-0.7.4/src/address/envelope.rs:67
  32: <actix::mailbox::Mailbox<A>>::poll
             at /home/mason/.cargo/registry/src/github.com-1ecc6299db9ec823/actix-0.7.4/src/mailbox.rs:87
  33: <actix::contextimpl::ContextFut<A, C> as futures::future::Future>::poll
             at /home/mason/.cargo/registry/src/github.com-1ecc6299db9ec823/actix-0.7.4/src/contextimpl.rs:323
  34: <alloc::boxed::Box<F> as futures::future::Future>::poll
             at /home/mason/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.24/src/future/mod.rs:113
  35: <alloc::boxed::Box<F> as futures::future::Future>::poll
             at /home/mason/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.24/src/future/mod.rs:113
  36: <futures::task_impl::Spawn<T>>::poll_future_notify::{{closure}}
             at /home/mason/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.24/src/task_impl/mod.rs:314
  37: <futures::task_impl::Spawn<T>>::enter::{{closure}}
             at /home/mason/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.24/src/task_impl/mod.rs:388
  38: futures::task_impl::std::set
             at /home/mason/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.24/src/task_impl/std/mod.rs:78
  39: <futures::task_impl::Spawn<T>>::enter
             at /home/mason/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.24/src/task_impl/mod.rs:388
  40: <futures::task_impl::Spawn<T>>::poll_future_notify
             at /home/mason/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.24/src/task_impl/mod.rs:314
  41: <tokio_current_thread::scheduler::Scheduled<'a, U>>::tick
             at /home/mason/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-current-thread-0.1.1/src/scheduler.rs:353
  42: <tokio_current_thread::scheduler::Scheduler<U>>::tick::{{closure}}
             at /home/mason/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-current-thread-0.1.1/src/scheduler.rs:332
  43: <tokio_current_thread::Borrow<'a, U>>::enter::{{closure}}::{{closure}}
             at /home/mason/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-current-thread-0.1.1/src/lib.rs:747
  44: tokio_current_thread::CurrentRunner::set_spawn
             at /home/mason/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-current-thread-0.1.1/src/lib.rs:783
  45: <tokio_current_thread::Borrow<'a, U>>::enter::{{closure}}
             at /home/mason/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-current-thread-0.1.1/src/lib.rs:746
  46: <std::thread::local::LocalKey<T>>::try_with
             at libstd/thread/local.rs:294
  47: <std::thread::local::LocalKey<T>>::with
             at libstd/thread/local.rs:248
  48: <tokio_current_thread::Borrow<'a, U>>::enter
             at /home/mason/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-current-thread-0.1.1/src/lib.rs:745
  49: <tokio_current_thread::scheduler::Scheduler<U>>::tick
             at /home/mason/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-current-thread-0.1.1/src/scheduler.rs:332
  50: <tokio_current_thread::Entered<'a, P>>::tick
             at /home/mason/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-current-thread-0.1.1/src/lib.rs:591
  51: <tokio_current_thread::Entered<'a, P>>::block_on
             at /home/mason/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-current-thread-0.1.1/src/lib.rs:475
  52: tokio::runtime::current_thread::runtime::Runtime::block_on::{{closure}}
             at /home/mason/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.1.8/src/runtime/current_thread/runtime.rs:155
  53: tokio::runtime::current_thread::runtime::Runtime::enter::{{closure}}::{{closure}}::{{closure}}::{{closure}}
             at /home/mason/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.1.8/src/runtime/current_thread/runtime.rs:196
  54: tokio_executor::global::with_default::{{closure}}
             at /home/mason/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-executor-0.1.4/src/global.rs:173
  55: <std::thread::local::LocalKey<T>>::try_with
             at libstd/thread/local.rs:294
  56: <std::thread::local::LocalKey<T>>::with
             at libstd/thread/local.rs:248
  57: tokio_executor::global::with_default
             at /home/mason/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-executor-0.1.4/src/global.rs:143
  58: tokio::runtime::current_thread::runtime::Runtime::enter::{{closure}}::{{closure}}::{{closure}}
             at /home/mason/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.1.8/src/runtime/current_thread/runtime.rs:194
  59: tokio_timer::timer::handle::with_default::{{closure}}
             at /home/mason/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-timer-0.2.6/src/timer/handle.rs:94
  60: <std::thread::local::LocalKey<T>>::try_with
             at libstd/thread/local.rs:294
  61: <std::thread::local::LocalKey<T>>::with
             at libstd/thread/local.rs:248
  62: tokio_timer::timer::handle::with_default
             at /home/mason/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-timer-0.2.6/src/timer/handle.rs:81
  63: tokio::runtime::current_thread::runtime::Runtime::enter::{{closure}}::{{closure}}
             at /home/mason/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.1.8/src/runtime/current_thread/runtime.rs:187
  64: tokio_timer::clock::clock::with_default::{{closure}}
             at /home/mason/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-timer-0.2.6/src/clock/clock.rs:136
  65: <std::thread::local::LocalKey<T>>::try_with
             at libstd/thread/local.rs:294
  66: <std::thread::local::LocalKey<T>>::with
             at libstd/thread/local.rs:248
  67: tokio_timer::clock::clock::with_default
             at /home/mason/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-timer-0.2.6/src/clock/clock.rs:119
  68: tokio::runtime::current_thread::runtime::Runtime::enter::{{closure}}
             at /home/mason/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.1.8/src/runtime/current_thread/runtime.rs:186
  69: tokio_reactor::with_default::{{closure}}
             at /home/mason/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-reactor-0.1.5/src/lib.rs:232
  70: <std::thread::local::LocalKey<T>>::try_with
             at libstd/thread/local.rs:294
  71: <std::thread::local::LocalKey<T>>::with
             at libstd/thread/local.rs:248
  72: tokio_reactor::with_default
             at /home/mason/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-reactor-0.1.5/src/lib.rs:215
  73: tokio::runtime::current_thread::runtime::Runtime::enter
             at /home/mason/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.1.8/src/runtime/current_thread/runtime.rs:185
  74: tokio::runtime::current_thread::runtime::Runtime::block_on
             at /home/mason/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.1.8/src/runtime/current_thread/runtime.rs:153
  75: actix::arbiter::Arbiter::new_with_builder::{{closure}}
             at /home/mason/.cargo/registry/src/github.com-1ecc6299db9ec823/actix-0.7.4/src/arbiter.rs:114
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

(the other 3 threads have similar traces)

[masonk]

One more question:

If as a user, I encounter this collision, can I hack around it? If so, how?

[carllerche]

It's hard for me to provide much commentary about the details of the two libs. However, if a library spawns its own runtime (or tokio_core::Core) it is providing a blocking interface. That would be the only reason to do so. Just like any library w/ a blocking interface, you cannot use it from a non-blocking context. This is why the panic is there.

If libraries wish to provide a non-blocking API, they should not create a runtime directly. Instead, just use the misc tokio resources (TcpStream, Timer, ...) and document that the library needs to be used from a tokio runtime.

Looking at hyper would be a good example.

I hope, in the future, to better document how to author a Tokio compatible library.

[masonk]

Thanks, that's already helpful.

I am considering trying to patch ldap3 so it works with actix-web.

ldap3 has an async API that accepts a tokio_core::Core as input: https://github.com/inejge/ldap3/blob/master/src/conn.rs#L405.

Does this mean I'll have to migrate ldap3 to tokio before they can interop?

[carllerche]

@masonk it is probable... tokio-core has been deprecated for 6~9 months now. It would probably be best for them to update.

[masonk]

I'm still new to this framework, but would it possible for a Core constructor to accept an other-than-default runtime?

Then I could construct a Core using actix-web's runtime and pass it down to ldap3.

[hawkw]

It looks like there's at least one issue against ldap3 regarding migrating to tokio: inejge/ldap3#29

[DoumanAsh]

@masonk you must note that since actix owns runtime, it might not be easy to share it with ldap3 itself.
While porting from tokio-core to tokio::runtime::current_thread should be pretty straightforward, the need for owned runtime would still be a problem

[masonk]

Actix spawns a 'System' for every worker thread. Each system is powered by a tokio current_thread::Runtime under the hood.

I'd thought that if ldap3 code called tokio::executor::spawn on a thread with an active executor, or from inside of a future that was under execution, tokio would simply find the thread_local static executor that had been previously created by actix (https://docs.rs/tokio-current-thread/0.1.2/src/tokio_current_thread/lib.rs.html#198) and use it.

If that's not the case, how should two libraries that don't know about each other in advance work together to run their tasks on the same executor?

[DoumanAsh]

@masonk if it only uses spawn then it should be ok, I'm only saying that the problem would be if it would need blocking stuff like block_on

[masonk]

The asynchronous API uses spawn, the synchronous API runs its own tokio_core::Core. I think that's the usual story for all APIs built on tokio: only synchronous APIs manage their own core and do the blocking top-level calls.

I think the requirement for tokio is that it should be easy for two different crates that have no prior knowledge of each other to spawn tasks on the same executor. When they can do that they're composing.

After reading a lot of tokio code the past few weeks, it seems like 'new' tokio uses the DefaultExecutor to get rid of explicit executor passing, and this is good enough for most cases. As long as crates are all using the DefaultExecutor then they're composable and their tasks can notified by the same reactor core and driven to completion by the same executor.

[carllerche]

Is there further action required to close this issue?

[masonk]

I think just close this one.

…

On Mon, Oct 15, 2018 at 10:12 PM Carl Lerche ***@***.***> wrote: Is there further action required to close this issue? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#659 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAbdzjzP29ztW_KkEVi5lx8M__Cyl3V-ks5ulWrNgaJpZM4W4vtL> .