Deadlock with futures mpsc and rayon_wait

[kaksmet]

I'm spawning some tasks using rayon, where each task is partially dependent on the previous task. futures::sync::mpsc together with rayon::spawn_future seemed like a good solution to manage this dependency but I'm having problems with deadlocks.

Running the code provided below with two threads in the rayon threadpool works as expected

$ RAYON_NUM_THREADS=2 RUSTFLAGS='--cfg rayon_unstable' cargo run
4 blocked
3 blocked
2 blocked
1 blocked
0 sent
1 unblocked
1 sent
2 unblocked
2 sent
3 unblocked
3 sent
4 unblocked
4 sent

but with three threads it deadlocks quite reliably

$ RAYON_NUM_THREADS=3 RUSTFLAGS='--cfg rayon_unstable' cargo run
1 blocked
4 blocked
2 blocked
3 blocked
0 sent

extern crate futures;
extern crate rayon;

use futures::{Sink, Stream};
use futures::sync::mpsc as futures_mpsc;
use std::{thread, time};

fn main() {
    let size = 5;

    rayon::scope(|scope| {
        let mut prev_receiver = None;

        for i in 0..size {
            let (sender, receiver) = futures_mpsc::channel(1);

            scope.spawn(move |_| {
                do_work(i, prev_receiver.take(), sender);
            });

            prev_receiver = Some(receiver);
        }
    });
}

fn do_work(
    number: usize,
    mut receiver: Option<futures_mpsc::Receiver<()>>,
    mut sender: futures_mpsc::Sender<()>,
) {
    if let Some(receiver) = receiver.take() {
        let receiver_future = receiver.into_future();
        println!("{} blocked", number);
        let _ = rayon::spawn_future(receiver_future).rayon_wait();
        println!("{} unblocked", number);
    }

    thread::sleep(time::Duration::from_millis(10));

    let _ = sender.start_send(());
    println!("{} sent", number);
}

[cuviper]

Have you tried this lately? It works fine for me on the current master rayon-futures, at least.

[kaksmet]

Still deadlocks on master for me with a 4-core machine running macOS 10.12.6

Here's the version I ran, slightly modified to compile with rayon-futures from git:

Cargo.toml

[package]
name = "korv-test"
version = "0.1.0"
authors = ["Ulf Nilsson <kaksmet@gmail.com>"]

[dependencies]
rayon = { git = "https://github.com/rayon-rs/rayon" }
rayon-futures = { git = "https://github.com/rayon-rs/rayon" }
futures = "0.1.16"

main.rs

extern crate futures;
extern crate rayon;
extern crate rayon_futures;

use futures::{Sink, Stream};
use futures::sync::mpsc as futures_mpsc;
use rayon_futures::ScopeFutureExt;
use std::{thread, time};

fn main() {
    let size = 5;

    rayon::scope(|scope| {
        let mut prev_receiver = None;

        for i in 0..size {
            let (sender, receiver) = futures_mpsc::channel(1);

            scope.spawn(move |_| {
                do_work(i, prev_receiver.take(), sender);
            });

            prev_receiver = Some(receiver);
        }
    });
}

fn do_work(
    number: usize,
    mut receiver: Option<futures_mpsc::Receiver<()>>,
    mut sender: futures_mpsc::Sender<()>,
) {
    if let Some(receiver) = receiver.take() {
        let receiver_future = receiver.into_future();
        println!("{} blocked", number);
        //let _ = rayon::spawn_future(receiver_future).rayon_wait();
        let _ = rayon::ThreadPool::global().spawn_future(receiver_future).rayon_wait();
        println!("{} unblocked", number);
    }

    thread::sleep(time::Duration::from_millis(10));

    let _ = sender.start_send(());
    println!("{} sent", number);
}

Here's the output from running with RAYON_LOG enabled

$ RAYON_NUM_THREADS=3 RAYON_LOG=1 RUSTFLAGS='--cfg rayon_unstable' cargo run
    Updating git repository `https://github.com/rayon-rs/rayon`
    Updating registry `https://github.com/rust-lang/crates.io-index`
   Compiling rayon-core v1.2.2 (https://github.com/rayon-rs/rayon#2aa274c6)
   Compiling futures v0.1.16
   Compiling lazy_static v0.2.9
   Compiling scopeguard v0.3.3
   Compiling either v1.3.0
   Compiling libc v0.2.32
   Compiling coco v0.1.1
   Compiling rand v0.3.17
   Compiling num_cpus v1.7.0
   Compiling rayon v0.8.2 (https://github.com/rayon-rs/rayon#2aa274c6)
   Compiling rayon-futures v0.1.0 (https://github.com/rayon-rs/rayon#2aa274c6)
   Compiling korv-test v0.1.0 (file:///Users/ulf/korv-test)
    Finished dev [unoptimized + debuginfo] target(s) in 5.98 secs
     Running `target/debug/korv-test`
InjectJobs { count: 1 }
WaitUntil { worker: 1 }
WaitUntil { worker: 2 }
WaitUntil { worker: 0 }
UninjectedWork { worker: 1 }
DidNotFindWork { worker: 0, yields: 0 }
DidNotFindWork { worker: 2, yields: 0 }
FoundWork { worker: 1, yields: 0 }
DidNotFindWork { worker: 0, yields: 1 }
DidNotFindWork { worker: 2, yields: 1 }
DidNotFindWork { worker: 2, yields: 2 }
DidNotFindWork { worker: 0, yields: 2 }
DidNotFindWork { worker: 2, yields: 3 }
DidNotFindWork { worker: 0, yields: 3 }
StoleWork { worker: 0, victim: 1 }
StoleWork { worker: 2, victim: 1 }
FoundWork { worker: 0, yields: 4 }
JobCompletedOk { owner_thread: 1 }
FoundWork { worker: 2, yields: 4 }
WaitUntil { worker: 1 }
1 blocked
FoundWork { worker: 1, yields: 0 }
4 blocked
WaitUntil { worker: 1 }
WaitUntil { worker: 2 }
FoundWork { worker: 1, yields: 0 }
FoundWork { worker: 2, yields: 0 }
FoundWork { worker: 1, yields: 0 }
3 blocked
StoleWork { worker: 2, victim: 1 }
WaitUntil { worker: 1 }
FoundWork { worker: 2, yields: 0 }
FoundWork { worker: 1, yields: 0 }
2 blocked
DidNotFindWork { worker: 1, yields: 0 }
DidNotFindWork { worker: 1, yields: 1 }
WaitUntil { worker: 2 }
FoundWork { worker: 2, yields: 0 }
DidNotFindWork { worker: 1, yields: 2 }
DidNotFindWork { worker: 2, yields: 0 }
DidNotFindWork { worker: 1, yields: 3 }
DidNotFindWork { worker: 2, yields: 1 }
DidNotFindWork { worker: 1, yields: 4 }
DidNotFindWork { worker: 2, yields: 2 }
DidNotFindWork { worker: 1, yields: 5 }
DidNotFindWork { worker: 2, yields: 3 }
DidNotFindWork { worker: 1, yields: 6 }
DidNotFindWork { worker: 1, yields: 7 }
DidNotFindWork { worker: 2, yields: 4 }
DidNotFindWork { worker: 1, yields: 8 }
DidNotFindWork { worker: 2, yields: 5 }
DidNotFindWork { worker: 1, yields: 9 }
DidNotFindWork { worker: 2, yields: 6 }
DidNotFindWork { worker: 2, yields: 7 }
DidNotFindWork { worker: 2, yields: 8 }
DidNotFindWork { worker: 1, yields: 10 }
DidNotFindWork { worker: 2, yields: 9 }
DidNotFindWork { worker: 2, yields: 10 }
DidNotFindWork { worker: 1, yields: 11 }
DidNotFindWork { worker: 2, yields: 11 }
DidNotFindWork { worker: 1, yields: 12 }
DidNotFindWork { worker: 2, yields: 12 }
DidNotFindWork { worker: 2, yields: 13 }
DidNotFindWork { worker: 1, yields: 13 }
DidNotFindWork { worker: 2, yields: 14 }
DidNotFindWork { worker: 1, yields: 14 }
DidNotFindWork { worker: 2, yields: 15 }
DidNotFindWork { worker: 1, yields: 15 }
DidNotFindWork { worker: 1, yields: 16 }
DidNotFindWork { worker: 2, yields: 16 }
DidNotFindWork { worker: 1, yields: 17 }
DidNotFindWork { worker: 2, yields: 17 }
DidNotFindWork { worker: 1, yields: 18 }
DidNotFindWork { worker: 1, yields: 19 }
DidNotFindWork { worker: 2, yields: 18 }
DidNotFindWork { worker: 1, yields: 20 }
DidNotFindWork { worker: 2, yields: 19 }
DidNotFindWork { worker: 1, yields: 21 }
DidNotFindWork { worker: 1, yields: 22 }
DidNotFindWork { worker: 1, yields: 23 }
DidNotFindWork { worker: 2, yields: 20 }
DidNotFindWork { worker: 1, yields: 24 }
DidNotFindWork { worker: 1, yields: 25 }
DidNotFindWork { worker: 2, yields: 21 }
DidNotFindWork { worker: 1, yields: 26 }
DidNotFindWork { worker: 2, yields: 22 }
DidNotFindWork { worker: 1, yields: 27 }
DidNotFindWork { worker: 1, yields: 28 }
DidNotFindWork { worker: 2, yields: 23 }
DidNotFindWork { worker: 1, yields: 29 }
DidNotFindWork { worker: 2, yields: 24 }
DidNotFindWork { worker: 1, yields: 30 }
DidNotFindWork { worker: 2, yields: 25 }
DidNotFindWork { worker: 1, yields: 31 }
DidNotFindWork { worker: 2, yields: 26 }
DidNotFindWork { worker: 1, yields: 32 }
DidNotFindWork { worker: 2, yields: 27 }
GetSleepy { worker: 1, state: 0 }
DidNotFindWork { worker: 2, yields: 28 }
GotSleepy { worker: 1, old_state: 0, new_state: 4 }
DidNotFindWork { worker: 1, yields: 33 }
DidNotFindWork { worker: 1, yields: 34 }
DidNotFindWork { worker: 2, yields: 29 }
DidNotFindWork { worker: 1, yields: 35 }
DidNotFindWork { worker: 1, yields: 36 }
DidNotFindWork { worker: 2, yields: 30 }
DidNotFindWork { worker: 1, yields: 37 }
DidNotFindWork { worker: 2, yields: 31 }
DidNotFindWork { worker: 1, yields: 38 }
DidNotFindWork { worker: 2, yields: 32 }
GetSleepy { worker: 2, state: 4 }
DidNotFindWork { worker: 1, yields: 39 }
DidNotFindWork { worker: 1, yields: 40 }
DidNotFindWork { worker: 2, yields: 32 }
GetSleepy { worker: 2, state: 4 }
DidNotFindWork { worker: 1, yields: 41 }
DidNotFindWork { worker: 2, yields: 32 }
DidNotFindWork { worker: 1, yields: 42 }
GetSleepy { worker: 2, state: 4 }
DidNotFindWork { worker: 2, yields: 32 }
GetSleepy { worker: 2, state: 4 }
DidNotFindWork { worker: 2, yields: 32 }
GetSleepy { worker: 2, state: 4 }
DidNotFindWork { worker: 1, yields: 43 }
DidNotFindWork { worker: 2, yields: 32 }
GetSleepy { worker: 2, state: 4 }
DidNotFindWork { worker: 1, yields: 44 }
DidNotFindWork { worker: 2, yields: 32 }
DidNotFindWork { worker: 1, yields: 45 }
GetSleepy { worker: 2, state: 4 }
DidNotFindWork { worker: 2, yields: 32 }
GetSleepy { worker: 2, state: 4 }
DidNotFindWork { worker: 2, yields: 32 }
DidNotFindWork { worker: 1, yields: 46 }
GetSleepy { worker: 2, state: 4 }
DidNotFindWork { worker: 2, yields: 32 }
DidNotFindWork { worker: 1, yields: 47 }
GetSleepy { worker: 2, state: 4 }
DidNotFindWork { worker: 2, yields: 32 }
GetSleepy { worker: 2, state: 4 }
DidNotFindWork { worker: 1, yields: 48 }
DidNotFindWork { worker: 2, yields: 32 }
DidNotFindWork { worker: 1, yields: 49 }
GetSleepy { worker: 2, state: 4 }
DidNotFindWork { worker: 1, yields: 50 }
DidNotFindWork { worker: 2, yields: 32 }
DidNotFindWork { worker: 1, yields: 51 }
GetSleepy { worker: 2, state: 4 }
DidNotFindWork { worker: 1, yields: 52 }
DidNotFindWork { worker: 2, yields: 32 }
DidNotFindWork { worker: 1, yields: 53 }
GetSleepy { worker: 2, state: 4 }
DidNotFindWork { worker: 1, yields: 54 }
DidNotFindWork { worker: 2, yields: 32 }
GetSleepy { worker: 2, state: 4 }
DidNotFindWork { worker: 1, yields: 55 }
DidNotFindWork { worker: 2, yields: 32 }
GetSleepy { worker: 2, state: 4 }
DidNotFindWork { worker: 2, yields: 32 }
DidNotFindWork { worker: 1, yields: 56 }
GetSleepy { worker: 2, state: 4 }
DidNotFindWork { worker: 2, yields: 32 }
GetSleepy { worker: 2, state: 4 }
DidNotFindWork { worker: 2, yields: 32 }
GetSleepy { worker: 2, state: 4 }
DidNotFindWork { worker: 2, yields: 32 }
DidNotFindWork { worker: 1, yields: 57 }
GetSleepy { worker: 2, state: 4 }
DidNotFindWork { worker: 2, yields: 32 }
GetSleepy { worker: 2, state: 4 }
DidNotFindWork { worker: 1, yields: 58 }
DidNotFindWork { worker: 2, yields: 32 }
DidNotFindWork { worker: 1, yields: 59 }
GetSleepy { worker: 2, state: 4 }
DidNotFindWork { worker: 2, yields: 32 }
DidNotFindWork { worker: 1, yields: 60 }
GetSleepy { worker: 2, state: 4 }
DidNotFindWork { worker: 2, yields: 32 }
GetSleepy { worker: 2, state: 4 }
DidNotFindWork { worker: 1, yields: 61 }
DidNotFindWork { worker: 2, yields: 32 }
GetSleepy { worker: 2, state: 4 }
DidNotFindWork { worker: 2, yields: 32 }
DidNotFindWork { worker: 1, yields: 62 }
GetSleepy { worker: 2, state: 4 }
DidNotFindWork { worker: 2, yields: 32 }
GetSleepy { worker: 2, state: 4 }
DidNotFindWork { worker: 1, yields: 63 }
DidNotFindWork { worker: 2, yields: 32 }
DidNotFindWork { worker: 1, yields: 64 }
GetSleepy { worker: 2, state: 4 }
FellAsleep { worker: 1 }
DidNotFindWork { worker: 2, yields: 32 }
GetSleepy { worker: 2, state: 1 }
GotSleepy { worker: 2, old_state: 1, new_state: 7 }
DidNotFindWork { worker: 2, yields: 33 }
DidNotFindWork { worker: 2, yields: 34 }
DidNotFindWork { worker: 2, yields: 35 }
DidNotFindWork { worker: 2, yields: 36 }
DidNotFindWork { worker: 2, yields: 37 }
DidNotFindWork { worker: 2, yields: 38 }
DidNotFindWork { worker: 2, yields: 39 }
DidNotFindWork { worker: 2, yields: 40 }
DidNotFindWork { worker: 2, yields: 41 }
DidNotFindWork { worker: 2, yields: 42 }
DidNotFindWork { worker: 2, yields: 43 }
DidNotFindWork { worker: 2, yields: 44 }
DidNotFindWork { worker: 2, yields: 45 }
DidNotFindWork { worker: 2, yields: 46 }
DidNotFindWork { worker: 2, yields: 47 }
DidNotFindWork { worker: 2, yields: 48 }
DidNotFindWork { worker: 2, yields: 49 }
DidNotFindWork { worker: 2, yields: 50 }
DidNotFindWork { worker: 2, yields: 51 }
DidNotFindWork { worker: 2, yields: 52 }
DidNotFindWork { worker: 2, yields: 53 }
DidNotFindWork { worker: 2, yields: 54 }
DidNotFindWork { worker: 2, yields: 55 }
DidNotFindWork { worker: 2, yields: 56 }
DidNotFindWork { worker: 2, yields: 57 }
DidNotFindWork { worker: 2, yields: 58 }
DidNotFindWork { worker: 2, yields: 59 }
DidNotFindWork { worker: 2, yields: 60 }
DidNotFindWork { worker: 2, yields: 61 }
DidNotFindWork { worker: 2, yields: 62 }
DidNotFindWork { worker: 2, yields: 63 }
DidNotFindWork { worker: 2, yields: 64 }
FellAsleep { worker: 2 }
Tickle { worker: 0, old_state: 1 }
0 sent
GotAwoken { worker: 1 }
GotAwoken { worker: 2 }
JobCompletedOk { owner_thread: 1 }
StoleWork { worker: 1, victim: 0 }
DidNotFindWork { worker: 2, yields: 0 }
FoundWork { worker: 1, yields: 0 }
DidNotFindWork { worker: 0, yields: 0 }
DidNotFindWork { worker: 2, yields: 1 }
DidNotFindWork { worker: 1, yields: 0 }
DidNotFindWork { worker: 0, yields: 1 }
DidNotFindWork { worker: 2, yields: 2 }
DidNotFindWork { worker: 1, yields: 1 }
DidNotFindWork { worker: 0, yields: 2 }
DidNotFindWork { worker: 2, yields: 3 }
DidNotFindWork { worker: 1, yields: 2 }
DidNotFindWork { worker: 0, yields: 3 }
DidNotFindWork { worker: 2, yields: 4 }
DidNotFindWork { worker: 1, yields: 3 }
DidNotFindWork { worker: 0, yields: 4 }
DidNotFindWork { worker: 2, yields: 5 }
DidNotFindWork { worker: 1, yields: 4 }
DidNotFindWork { worker: 0, yields: 5 }
DidNotFindWork { worker: 2, yields: 6 }
DidNotFindWork { worker: 1, yields: 5 }
DidNotFindWork { worker: 0, yields: 6 }
DidNotFindWork { worker: 2, yields: 7 }
DidNotFindWork { worker: 1, yields: 6 }
DidNotFindWork { worker: 0, yields: 7 }
DidNotFindWork { worker: 2, yields: 8 }
DidNotFindWork { worker: 1, yields: 7 }
DidNotFindWork { worker: 0, yields: 8 }
DidNotFindWork { worker: 2, yields: 9 }
DidNotFindWork { worker: 1, yields: 8 }
DidNotFindWork { worker: 0, yields: 9 }
DidNotFindWork { worker: 2, yields: 10 }
DidNotFindWork { worker: 1, yields: 9 }
DidNotFindWork { worker: 0, yields: 10 }
DidNotFindWork { worker: 2, yields: 11 }
DidNotFindWork { worker: 0, yields: 11 }
DidNotFindWork { worker: 1, yields: 10 }
DidNotFindWork { worker: 2, yields: 12 }
DidNotFindWork { worker: 0, yields: 12 }
DidNotFindWork { worker: 2, yields: 13 }
DidNotFindWork { worker: 1, yields: 11 }
DidNotFindWork { worker: 0, yields: 13 }
DidNotFindWork { worker: 2, yields: 14 }
DidNotFindWork { worker: 1, yields: 12 }
DidNotFindWork { worker: 0, yields: 14 }
DidNotFindWork { worker: 2, yields: 15 }
DidNotFindWork { worker: 1, yields: 13 }
DidNotFindWork { worker: 0, yields: 15 }
DidNotFindWork { worker: 2, yields: 16 }
DidNotFindWork { worker: 0, yields: 16 }
DidNotFindWork { worker: 1, yields: 14 }
DidNotFindWork { worker: 2, yields: 17 }
DidNotFindWork { worker: 0, yields: 17 }
DidNotFindWork { worker: 1, yields: 15 }
DidNotFindWork { worker: 2, yields: 18 }
DidNotFindWork { worker: 1, yields: 16 }
DidNotFindWork { worker: 0, yields: 18 }
DidNotFindWork { worker: 2, yields: 19 }
DidNotFindWork { worker: 1, yields: 17 }
DidNotFindWork { worker: 0, yields: 19 }
DidNotFindWork { worker: 1, yields: 18 }
DidNotFindWork { worker: 2, yields: 20 }
DidNotFindWork { worker: 0, yields: 20 }
DidNotFindWork { worker: 1, yields: 19 }
DidNotFindWork { worker: 2, yields: 21 }
DidNotFindWork { worker: 0, yields: 21 }
DidNotFindWork { worker: 1, yields: 20 }
DidNotFindWork { worker: 2, yields: 22 }
DidNotFindWork { worker: 1, yields: 21 }
DidNotFindWork { worker: 0, yields: 22 }
DidNotFindWork { worker: 2, yields: 23 }
DidNotFindWork { worker: 1, yields: 22 }
DidNotFindWork { worker: 0, yields: 23 }
DidNotFindWork { worker: 2, yields: 24 }
DidNotFindWork { worker: 0, yields: 24 }
DidNotFindWork { worker: 1, yields: 23 }
DidNotFindWork { worker: 2, yields: 25 }
DidNotFindWork { worker: 0, yields: 25 }
DidNotFindWork { worker: 1, yields: 24 }
DidNotFindWork { worker: 2, yields: 26 }
DidNotFindWork { worker: 0, yields: 26 }
DidNotFindWork { worker: 1, yields: 25 }
DidNotFindWork { worker: 2, yields: 27 }
DidNotFindWork { worker: 0, yields: 27 }
DidNotFindWork { worker: 1, yields: 26 }
DidNotFindWork { worker: 2, yields: 28 }
DidNotFindWork { worker: 0, yields: 28 }
DidNotFindWork { worker: 1, yields: 27 }
DidNotFindWork { worker: 0, yields: 29 }
DidNotFindWork { worker: 2, yields: 29 }
DidNotFindWork { worker: 1, yields: 28 }
DidNotFindWork { worker: 2, yields: 30 }
DidNotFindWork { worker: 0, yields: 30 }
DidNotFindWork { worker: 1, yields: 29 }
DidNotFindWork { worker: 2, yields: 31 }
DidNotFindWork { worker: 1, yields: 30 }
DidNotFindWork { worker: 0, yields: 31 }
DidNotFindWork { worker: 2, yields: 32 }
DidNotFindWork { worker: 1, yields: 31 }
GetSleepy { worker: 2, state: 0 }
DidNotFindWork { worker: 0, yields: 32 }
GotSleepy { worker: 2, old_state: 0, new_state: 6 }
DidNotFindWork { worker: 1, yields: 32 }
GetSleepy { worker: 0, state: 6 }
DidNotFindWork { worker: 2, yields: 33 }
DidNotFindWork { worker: 0, yields: 32 }
GetSleepy { worker: 1, state: 6 }
GetSleepy { worker: 0, state: 6 }
DidNotFindWork { worker: 2, yields: 34 }
DidNotFindWork { worker: 1, yields: 32 }
DidNotFindWork { worker: 0, yields: 32 }
DidNotFindWork { worker: 2, yields: 35 }
GetSleepy { worker: 1, state: 6 }
GetSleepy { worker: 0, state: 6 }
DidNotFindWork { worker: 1, yields: 32 }
DidNotFindWork { worker: 2, yields: 36 }
DidNotFindWork { worker: 0, yields: 32 }
GetSleepy { worker: 1, state: 6 }
GetSleepy { worker: 0, state: 6 }
DidNotFindWork { worker: 2, yields: 37 }
DidNotFindWork { worker: 1, yields: 32 }
DidNotFindWork { worker: 0, yields: 32 }
GetSleepy { worker: 1, state: 6 }
DidNotFindWork { worker: 1, yields: 32 }
GetSleepy { worker: 1, state: 6 }
DidNotFindWork { worker: 2, yields: 38 }
GetSleepy { worker: 0, state: 6 }
DidNotFindWork { worker: 1, yields: 32 }
DidNotFindWork { worker: 2, yields: 39 }
DidNotFindWork { worker: 0, yields: 32 }
GetSleepy { worker: 1, state: 6 }
GetSleepy { worker: 0, state: 6 }
DidNotFindWork { worker: 1, yields: 32 }
DidNotFindWork { worker: 2, yields: 40 }
DidNotFindWork { worker: 0, yields: 32 }
GetSleepy { worker: 1, state: 6 }
DidNotFindWork { worker: 2, yields: 41 }
GetSleepy { worker: 0, state: 6 }
DidNotFindWork { worker: 1, yields: 32 }
DidNotFindWork { worker: 2, yields: 42 }
DidNotFindWork { worker: 0, yields: 32 }
GetSleepy { worker: 0, state: 6 }
GetSleepy { worker: 1, state: 6 }
DidNotFindWork { worker: 2, yields: 43 }
DidNotFindWork { worker: 0, yields: 32 }
DidNotFindWork { worker: 1, yields: 32 }
DidNotFindWork { worker: 2, yields: 44 }
GetSleepy { worker: 0, state: 6 }
GetSleepy { worker: 1, state: 6 }
DidNotFindWork { worker: 2, yields: 45 }
DidNotFindWork { worker: 0, yields: 32 }
DidNotFindWork { worker: 1, yields: 32 }
DidNotFindWork { worker: 2, yields: 46 }
GetSleepy { worker: 0, state: 6 }
DidNotFindWork { worker: 0, yields: 32 }
GetSleepy { worker: 1, state: 6 }
DidNotFindWork { worker: 1, yields: 32 }
GetSleepy { worker: 0, state: 6 }
GetSleepy { worker: 1, state: 6 }
DidNotFindWork { worker: 2, yields: 47 }
DidNotFindWork { worker: 0, yields: 32 }
DidNotFindWork { worker: 1, yields: 32 }
GetSleepy { worker: 0, state: 6 }
DidNotFindWork { worker: 2, yields: 48 }
GetSleepy { worker: 1, state: 6 }
DidNotFindWork { worker: 0, yields: 32 }
DidNotFindWork { worker: 2, yields: 49 }
DidNotFindWork { worker: 1, yields: 32 }
GetSleepy { worker: 0, state: 6 }
DidNotFindWork { worker: 2, yields: 50 }
DidNotFindWork { worker: 0, yields: 32 }
GetSleepy { worker: 1, state: 6 }
DidNotFindWork { worker: 2, yields: 51 }
DidNotFindWork { worker: 1, yields: 32 }
GetSleepy { worker: 0, state: 6 }
DidNotFindWork { worker: 2, yields: 52 }
GetSleepy { worker: 1, state: 6 }
DidNotFindWork { worker: 0, yields: 32 }
DidNotFindWork { worker: 1, yields: 32 }
DidNotFindWork { worker: 2, yields: 53 }
GetSleepy { worker: 1, state: 6 }
GetSleepy { worker: 0, state: 6 }
DidNotFindWork { worker: 2, yields: 54 }
DidNotFindWork { worker: 1, yields: 32 }
DidNotFindWork { worker: 0, yields: 32 }
DidNotFindWork { worker: 2, yields: 55 }
GetSleepy { worker: 0, state: 6 }
GetSleepy { worker: 1, state: 6 }
DidNotFindWork { worker: 2, yields: 56 }
DidNotFindWork { worker: 0, yields: 32 }
DidNotFindWork { worker: 1, yields: 32 }
GetSleepy { worker: 0, state: 6 }
DidNotFindWork { worker: 2, yields: 57 }
DidNotFindWork { worker: 0, yields: 32 }
GetSleepy { worker: 1, state: 6 }
DidNotFindWork { worker: 2, yields: 58 }
DidNotFindWork { worker: 1, yields: 32 }
GetSleepy { worker: 0, state: 6 }
DidNotFindWork { worker: 2, yields: 59 }
GetSleepy { worker: 1, state: 6 }
DidNotFindWork { worker: 0, yields: 32 }
DidNotFindWork { worker: 1, yields: 32 }
DidNotFindWork { worker: 2, yields: 60 }
GetSleepy { worker: 0, state: 6 }
GetSleepy { worker: 1, state: 6 }
DidNotFindWork { worker: 0, yields: 32 }
DidNotFindWork { worker: 2, yields: 61 }
DidNotFindWork { worker: 1, yields: 32 }
GetSleepy { worker: 0, state: 6 }
DidNotFindWork { worker: 2, yields: 62 }
GetSleepy { worker: 1, state: 6 }
DidNotFindWork { worker: 0, yields: 32 }
DidNotFindWork { worker: 1, yields: 32 }
DidNotFindWork { worker: 2, yields: 63 }
GetSleepy { worker: 0, state: 6 }
GetSleepy { worker: 1, state: 6 }
DidNotFindWork { worker: 2, yields: 64 }
DidNotFindWork { worker: 0, yields: 32 }
DidNotFindWork { worker: 1, yields: 32 }
FellAsleep { worker: 2 }
GetSleepy { worker: 0, state: 1 }
GetSleepy { worker: 1, state: 1 }
GotSleepy { worker: 0, old_state: 1, new_state: 3 }
GetSleepy { worker: 1, state: 3 }
DidNotFindWork { worker: 0, yields: 33 }
DidNotFindWork { worker: 1, yields: 32 }
GetSleepy { worker: 1, state: 3 }
DidNotFindWork { worker: 1, yields: 32 }
DidNotFindWork { worker: 0, yields: 34 }
GetSleepy { worker: 1, state: 3 }
DidNotFindWork { worker: 1, yields: 32 }
DidNotFindWork { worker: 0, yields: 35 }
GetSleepy { worker: 1, state: 3 }
DidNotFindWork { worker: 1, yields: 32 }
GetSleepy { worker: 1, state: 3 }
DidNotFindWork { worker: 1, yields: 32 }
GetSleepy { worker: 1, state: 3 }
DidNotFindWork { worker: 0, yields: 36 }
DidNotFindWork { worker: 1, yields: 32 }
GetSleepy { worker: 1, state: 3 }
DidNotFindWork { worker: 0, yields: 37 }
DidNotFindWork { worker: 1, yields: 32 }
DidNotFindWork { worker: 0, yields: 38 }
GetSleepy { worker: 1, state: 3 }
DidNotFindWork { worker: 0, yields: 39 }
DidNotFindWork { worker: 0, yields: 40 }
DidNotFindWork { worker: 1, yields: 32 }
GetSleepy { worker: 1, state: 3 }
DidNotFindWork { worker: 0, yields: 41 }
DidNotFindWork { worker: 1, yields: 32 }
DidNotFindWork { worker: 0, yields: 42 }
GetSleepy { worker: 1, state: 3 }
DidNotFindWork { worker: 1, yields: 32 }
GetSleepy { worker: 1, state: 3 }
DidNotFindWork { worker: 0, yields: 43 }
DidNotFindWork { worker: 1, yields: 32 }
DidNotFindWork { worker: 0, yields: 44 }
GetSleepy { worker: 1, state: 3 }
DidNotFindWork { worker: 0, yields: 45 }
DidNotFindWork { worker: 1, yields: 32 }
GetSleepy { worker: 1, state: 3 }
DidNotFindWork { worker: 0, yields: 46 }
DidNotFindWork { worker: 1, yields: 32 }
DidNotFindWork { worker: 0, yields: 47 }
GetSleepy { worker: 1, state: 3 }
DidNotFindWork { worker: 0, yields: 48 }
DidNotFindWork { worker: 1, yields: 32 }
DidNotFindWork { worker: 0, yields: 49 }
DidNotFindWork { worker: 0, yields: 50 }
GetSleepy { worker: 1, state: 3 }
DidNotFindWork { worker: 1, yields: 32 }
GetSleepy { worker: 1, state: 3 }
DidNotFindWork { worker: 1, yields: 32 }
GetSleepy { worker: 1, state: 3 }
DidNotFindWork { worker: 0, yields: 51 }
DidNotFindWork { worker: 1, yields: 32 }
GetSleepy { worker: 1, state: 3 }
DidNotFindWork { worker: 0, yields: 52 }
DidNotFindWork { worker: 1, yields: 32 }
GetSleepy { worker: 1, state: 3 }
DidNotFindWork { worker: 0, yields: 53 }
DidNotFindWork { worker: 1, yields: 32 }
GetSleepy { worker: 1, state: 3 }
DidNotFindWork { worker: 0, yields: 54 }
DidNotFindWork { worker: 1, yields: 32 }
GetSleepy { worker: 1, state: 3 }
DidNotFindWork { worker: 1, yields: 32 }
GetSleepy { worker: 1, state: 3 }
DidNotFindWork { worker: 0, yields: 55 }
DidNotFindWork { worker: 1, yields: 32 }
DidNotFindWork { worker: 0, yields: 56 }
GetSleepy { worker: 1, state: 3 }
DidNotFindWork { worker: 1, yields: 32 }
GetSleepy { worker: 1, state: 3 }
DidNotFindWork { worker: 0, yields: 57 }
DidNotFindWork { worker: 1, yields: 32 }
DidNotFindWork { worker: 0, yields: 58 }
GetSleepy { worker: 1, state: 3 }
DidNotFindWork { worker: 0, yields: 59 }
DidNotFindWork { worker: 1, yields: 32 }
GetSleepy { worker: 1, state: 3 }
DidNotFindWork { worker: 0, yields: 60 }
DidNotFindWork { worker: 1, yields: 32 }
DidNotFindWork { worker: 0, yields: 61 }
GetSleepy { worker: 1, state: 3 }
DidNotFindWork { worker: 0, yields: 62 }
DidNotFindWork { worker: 1, yields: 32 }
DidNotFindWork { worker: 0, yields: 63 }
GetSleepy { worker: 1, state: 3 }
DidNotFindWork { worker: 1, yields: 32 }
GetSleepy { worker: 1, state: 3 }
DidNotFindWork { worker: 1, yields: 32 }
DidNotFindWork { worker: 0, yields: 64 }
GetSleepy { worker: 1, state: 3 }
FellAsleep { worker: 0 }
DidNotFindWork { worker: 1, yields: 32 }
GetSleepy { worker: 1, state: 1 }
GotSleepy { worker: 1, old_state: 1, new_state: 5 }
DidNotFindWork { worker: 1, yields: 33 }
DidNotFindWork { worker: 1, yields: 34 }
DidNotFindWork { worker: 1, yields: 35 }
DidNotFindWork { worker: 1, yields: 36 }
DidNotFindWork { worker: 1, yields: 37 }
DidNotFindWork { worker: 1, yields: 38 }
DidNotFindWork { worker: 1, yields: 39 }
DidNotFindWork { worker: 1, yields: 40 }
DidNotFindWork { worker: 1, yields: 41 }
DidNotFindWork { worker: 1, yields: 42 }
DidNotFindWork { worker: 1, yields: 43 }
DidNotFindWork { worker: 1, yields: 44 }
DidNotFindWork { worker: 1, yields: 45 }
DidNotFindWork { worker: 1, yields: 46 }
DidNotFindWork { worker: 1, yields: 47 }
DidNotFindWork { worker: 1, yields: 48 }
DidNotFindWork { worker: 1, yields: 49 }
DidNotFindWork { worker: 1, yields: 50 }
DidNotFindWork { worker: 1, yields: 51 }
DidNotFindWork { worker: 1, yields: 52 }
DidNotFindWork { worker: 1, yields: 53 }
DidNotFindWork { worker: 1, yields: 54 }
DidNotFindWork { worker: 1, yields: 55 }
DidNotFindWork { worker: 1, yields: 56 }
DidNotFindWork { worker: 1, yields: 57 }
DidNotFindWork { worker: 1, yields: 58 }
DidNotFindWork { worker: 1, yields: 59 }
DidNotFindWork { worker: 1, yields: 60 }
DidNotFindWork { worker: 1, yields: 61 }
DidNotFindWork { worker: 1, yields: 62 }
DidNotFindWork { worker: 1, yields: 63 }
DidNotFindWork { worker: 1, yields: 64 }
FellAsleep { worker: 1 }
^C

[cuviper]

Hmm, OK, this reproduces for me on Linux, but not every time. 1 or 2 threads are consistently fine, 3-7 threads will hang most times (but not always!), then 8 threads seems OK again.

[cuviper]

FWIW, instead of throwing away the inner scope handle in scope.spawn(move |_| { ..., you could pass that scope to do_work to call scope.spawn_future. But it doesn't change the deadlock behavior for me.

[cuviper]

This seems to be a dependency inversion. e.g. with future B depending on future A via mpsc. Suppose a given thread executes do_work() -> A.rayon_wait() and it's not ready yet. This job is the one that will send() to B. While A is blocked, it steals some other job, which then executes do_work() -> B.rayon_wait(). Now we're blocked probing for B, no longer checking A. Then something completes A, but we don't notice, and we never get to make progress.

[cuviper]

Reduced example:

extern crate futures;
extern crate rayon;
extern crate rayon_futures;

use futures::{Sink, Stream};
use futures::sync::mpsc as futures_mpsc;
use rayon_futures::ScopeFutureExt;

fn main() {
    let config = rayon::Configuration::new().num_threads(1);
    let pool = rayon::ThreadPool::new(config).unwrap();
    pool.scope(|scope| {
        let (mut sender1, receiver1) = futures_mpsc::channel(1);
        let (mut sender2, receiver2) = futures_mpsc::channel(1);

        scope.spawn(move |_| {
            sender1.start_send(()).unwrap();
        });

        scope.spawn(move |scope| {
            scope.spawn_future(receiver2.into_future()).rayon_wait().unwrap();
        });

        scope.spawn(move |scope| {
            scope.spawn_future(receiver1.into_future()).rayon_wait().unwrap();
            sender2.start_send(()).unwrap();
        });
    });
}

With a single thread, the spawns will run LIFO:

• wait on channel 1, look for more work
  • wait on channel 2, look for more work
    • send to channel 1
  • still waiting for channel 2...

With more threads, the order can be arbitrarily mixed, since local threads steal LIFO and remote threads steal FIFO. So fixing this isn't just a matter of spawning in a better order.

[nikomatsakis]

Hmm, interesting. Yes, I think the idea of rayon_wait was sort of flawed. We should probably just remove it. My original intention with the futures work was certainly not to encourage people to write using rayon_wait -- that was supposed to be a "fallback".

I originally thought we would solve these sorts of things by growing additional threads, but I suppose that this example shows that even that isn't always a solution. (I can't tell you how many times I have rediscovered this exact deadlock pattern -- that is, one caused by the implicit edge of using the same stack -- in my life.)

[cuviper]

FWIW, Future::wait was deprecated in rust-lang/futures-rs#634. I guess they now have Blocking::wait, but I don't know if we need the same -- or even if we can, given the bug here.

[cuviper]

This code was removed in #716.