This repository has been archived by the owner on Jun 1, 2021. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 99
Investigation of build failures
Volker Stampa edited this page Oct 11, 2016
·
5 revisions
https://travis-ci.org/RBMHTechnology/eventuate/builds/163089992#L1806
[info] EventsourcedActorThroughputSpecLeveldb:
[info] An EventsourcedActor
[info] when configured with stateSync = true
438.589676352428 events/sec
[info] - should have some acceptable write throughput (batching layer has no effect)
[info] An EventsourcedActor
[info] when configured with stateSync = false
8670.345395954255 events/sec
[info] - should have some acceptable write throughput (batching layer has some effect)
[info] Several EventsourcedActors
[info] when configured with stateSync = true
[ERROR] [09/20/2016 09:42:44.355] [test-akka.actor.default-dispatcher-7] [akka://test/user/$k] replay failed, stopping self
akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://test/user/$e#-1631700046]] after [10000 ms]. Sender[Actor[akka://test/user/$k#1008840159]] sent message of type "com.rbmhtechnology.eventuate.EventsourcingProtocol$Replay".
at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604)
at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:122)
at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:599)
at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:597)
at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:322)
at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:274)
at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:278)
at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:230)
at java.lang.Thread.run(Thread.java:745)
[ERROR] [09/20/2016 09:42:44.357] [test-akka.actor.default-dispatcher-25] [akka://test/user/$o] replay failed, stopping self
akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://test/user/$e#-1631700046]] after [10000 ms]. Sender[Actor[akka://test/user/$o#742367396]] sent message of type "com.rbmhtechnology.eventuate.EventsourcingProtocol$Replay".
at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604)
at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:122)
at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:599)
at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:597)
at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:322)
at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:274)
at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:278)
at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:230)
at java.lang.Thread.run(Thread.java:745)
[info] - should have some reasonable overall write throughput (batching layer has some effects) *** FAILED ***
[info] java.lang.AssertionError: assertion failed: timeout (1 minute) during expectMsg while waiting for 10000
[info] at scala.Predef$.assert(Predef.scala:165)
[info] at akka.testkit.TestKitBase$class.expectMsg_internal(TestKit.scala:349)
[info] at akka.testkit.TestKitBase$class.expectMsg(TestKit.scala:335)
[info] at akka.testkit.TestKit.expectMsg(TestKit.scala:734)
[info] at com.rbmhtechnology.eventuate.EventsourcedActorThroughputSpec$$anonfun$3$$anonfun$apply$mcV$sp$5$$anonfun$apply$mcV$sp$6.apply$mcV$sp(EventsourcedActorThroughputSpec.scala:145)
[info] at com.rbmhtechnology.eventuate.EventsourcedActorThroughputSpec$$anonfun$3$$anonfun$apply$mcV$sp$5$$anonfun$apply$mcV$sp$6.apply(EventsourcedActorThroughputSpec.scala:131)
[info] at com.rbmhtechnology.eventuate.EventsourcedActorThroughputSpec$$anonfun$3$$anonfun$apply$mcV$sp$5$$anonfun$apply$mcV$sp$6.apply(EventsourcedActorThroughputSpec.scala:131)
[info] at org.scalatest.Transformer$$anonfun$apply$1.apply(Transformer.scala:22)
[info] at org.scalatest.Transformer$$anonfun$apply$1.apply(Transformer.scala:22)
[info] at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
[info] ...
[0m[[0minfo[0m] [0m[32mReplicationCycleSpecLeveldb:[0m[0m
[0m[[0minfo[0m] [0m[32mEvent log replication[0m[0m
[ERROR] [09/19/2016 07:59:19.123] [location-akka.actor.default-dispatcher-31] [akka.tcp://[email protected]:55845/user/$b/$a/$a] replication failure detection limit reached (3000 milliseconds), publishing Unavailable for C_1/L1 (last exception being reported)
[ERROR] [09/19/2016 07:59:19.123] [location-akka.actor.default-dispatcher-51] [akka.tcp://[email protected]:55845/user/$a/$a/$a] replication failure detection limit reached (3000 milliseconds), publishing Unavailable for B_1/L1 (last exception being reported)
com.rbmhtechnology.eventuate.ReplicationProtocol$ReplicationReadTimeoutException: Replication read timed out after 1000 milliseconds
at com.rbmhtechnology.eventuate.Replicator$$anonfun$com$rbmhtechnology$eventuate$Replicator$$read$1.applyOrElse(ReplicationEndpoint.scala:661)
at com.rbmhtechnology.eventuate.Replicator$$anonfun$com$rbmhtechnology$eventuate$Replicator$$read$1.applyOrElse(ReplicationEndpoint.scala:660)
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
at scala.util.Failure$$anonfun$recover$1.apply(Try.scala:216)
at scala.util.Try$.apply(Try.scala:192)
at scala.util.Failure.recover(Try.scala:216)
at scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324)
at scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:405)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[ERROR] [09/19/2016 07:59:19.126] [location-akka.actor.default-dispatcher-18] [akka.tcp://[email protected]:44041/user/$b/$a/$a] replication failure detection limit reached (3000 milliseconds), publishing Unavailable for B_1/L1 (last exception being reported)
com.rbmhtechnology.eventuate.ReplicationProtocol$ReplicationReadTimeoutException: Replication read timed out after 1000 milliseconds
at com.rbmhtechnology.eventuate.Replicator$$anonfun$com$rbmhtechnology$eventuate$Replicator$$read$1.applyOrElse(ReplicationEndpoint.scala:661)
at com.rbmhtechnology.eventuate.Replicator$$anonfun$com$rbmhtechnology$eventuate$Replicator$$read$1.applyOrElse(ReplicationEndpoint.scala:660)
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
at scala.util.Failure$$anonfun$recover$1.apply(Try.scala:216)
at scala.util.Try$.apply(Try.scala:192)
at scala.util.Failure.recover(Try.scala:216)
at scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324)
at scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:405)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[ERROR] [09/19/2016 07:59:19.126] [location-akka.actor.default-dispatcher-5] [akka.tcp://[email protected]:44041/user/$a/$a/$a] replication failure detection limit reached (3000 milliseconds), publishing Unavailable for A_1/L1 (last exception being reported)
com.rbmhtechnology.eventuate.ReplicationProtocol$ReplicationReadTimeoutException: Replication read timed out after 1000 milliseconds
at com.rbmhtechnology.eventuate.Replicator$$anonfun$com$rbmhtechnology$eventuate$Replicator$$read$1.applyOrElse(ReplicationEndpoint.scala:661)
at com.rbmhtechnology.eventuate.Replicator$$anonfun$com$rbmhtechnology$eventuate$Replicator$$read$1.applyOrElse(ReplicationEndpoint.scala:660)
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
at scala.util.Failure$$anonfun$recover$1.apply(Try.scala:216)
at scala.util.Try$.apply(Try.scala:192)
at scala.util.Failure.recover(Try.scala:216)
at scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324)
at scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:405)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[ERROR] [09/19/2016 07:59:19.136] [location-akka.actor.default-dispatcher-27] [akka.tcp://[email protected]:38829/user/$a/$a/$a] replication failure detection limit reached (3000 milliseconds), publishing Unavailable for A_1/L1 (last exception being reported)
[ERROR] [09/19/2016 07:59:19.146] [location-akka.actor.default-dispatcher-18] [akka.tcp://[email protected]:38829/user/$b/$a/$a] replication failure detection limit reached (3000 milliseconds), publishing Unavailable for C_1/L1 (last exception being reported)
[ERROR] [09/19/2016 07:59:22.142] [location-akka.actor.default-dispatcher-52] [akka.tcp://[email protected]:55845/user/$a/$a/$a] replication failure detection limit reached (3000 milliseconds), publishing Unavailable for B_1/L1 (last exception being reported)
[ERROR] [09/19/2016 07:59:22.142] [location-akka.actor.default-dispatcher-33] [akka.tcp://[email protected]:55845/user/$b/$a/$a] replication failure detection limit reached (3000 milliseconds), publishing Unavailable for C_1/L1 (last exception being reported)
[ERROR] [09/19/2016 07:59:22.146] [location-akka.actor.default-dispatcher-16] [akka.tcp://[email protected]:44041/user/$b/$a/$a] replication failure detection limit reached (3000 milliseconds), publishing Unavailable for B_1/L1 (last exception being reported)
com.rbmhtechnology.eventuate.ReplicationProtocol$ReplicationReadTimeoutException: Replication read timed out after 1000 milliseconds
at com.rbmhtechnology.eventuate.Replicator$$anonfun$com$rbmhtechnology$eventuate$Replicator$$read$1.applyOrElse(ReplicationEndpoint.scala:661)
at com.rbmhtechnology.eventuate.Replicator$$anonfun$com$rbmhtechnology$eventuate$Replicator$$read$1.applyOrElse(ReplicationEndpoint.scala:660)
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
at scala.util.Failure$$anonfun$recover$1.apply(Try.scala:216)
at scala.util.Try$.apply(Try.scala:192)
at scala.util.Failure.recover(Try.scala:216)
at scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324)
at scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:405)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[ERROR] [09/19/2016 07:59:22.146] [location-akka.actor.default-dispatcher-36] [akka.tcp://[email protected]:44041/user/$a/$a/$a] replication failure detection limit reached (3000 milliseconds), publishing Unavailable for A_1/L1 (last exception being reported)
com.rbmhtechnology.eventuate.ReplicationProtocol$ReplicationReadTimeoutException: Replication read timed out after 1000 milliseconds
at com.rbmhtechnology.eventuate.Replicator$$anonfun$com$rbmhtechnology$eventuate$Replicator$$read$1.applyOrElse(ReplicationEndpoint.scala:661)
at com.rbmhtechnology.eventuate.Replicator$$anonfun$com$rbmhtechnology$eventuate$Replicator$$read$1.applyOrElse(ReplicationEndpoint.scala:660)
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
at scala.util.Failure$$anonfun$recover$1.apply(Try.scala:216)
at scala.util.Try$.apply(Try.scala:192)
at scala.util.Failure.recover(Try.scala:216)
at scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324)
at scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:405)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[ERROR] [09/19/2016 07:59:22.157] [location-akka.actor.default-dispatcher-27] [akka.tcp://[email protected]:38829/user/$a/$a/$a] replication failure detection limit reached (3000 milliseconds), publishing Unavailable for A_1/L1 (last exception being reported)
[ERROR] [09/19/2016 07:59:22.166] [location-akka.actor.default-dispatcher-47] [akka.tcp://[email protected]:38829/user/$b/$a/$a] replication failure detection limit reached (3000 milliseconds), publishing Unavailable for C_1/L1 (last exception being reported)
[ERROR] [09/19/2016 07:59:25.366] [location-akka.actor.default-dispatcher-16] [akka.tcp://[email protected]:44041/user/$a/$a/$a] replication failure detection limit reached (3000 milliseconds), publishing Unavailable for A_1/L1 (last exception being reported)
com.rbmhtechnology.eventuate.ReplicationProtocol$ReplicationReadTimeoutException: Replication read timed out after 1000 milliseconds
at com.rbmhtechnology.eventuate.Replicator$$anonfun$com$rbmhtechnology$eventuate$Replicator$$read$1.applyOrElse(ReplicationEndpoint.scala:661)
at com.rbmhtechnology.eventuate.Replicator$$anonfun$com$rbmhtechnology$eventuate$Replicator$$read$1.applyOrElse(ReplicationEndpoint.scala:660)
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
at scala.util.Failure$$anonfun$recover$1.apply(Try.scala:216)
at scala.util.Try$.apply(Try.scala:192)
at scala.util.Failure.recover(Try.scala:216)
at scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324)
at scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:405)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[ERROR] [09/19/2016 07:59:25.373] [location-akka.actor.default-dispatcher-6] [akka.tcp://[email protected]:55845/user/$a/$a/$a] replication failure detection limit reached (3000 milliseconds), publishing Unavailable for B_1/L1 (last exception being reported)
com.rbmhtechnology.eventuate.ReplicationProtocol$ReplicationReadTimeoutException: Replication read timed out after 1000 milliseconds
at com.rbmhtechnology.eventuate.Replicator$$anonfun$com$rbmhtechnology$eventuate$Replicator$$read$1.applyOrElse(ReplicationEndpoint.scala:661)
at com.rbmhtechnology.eventuate.Replicator$$anonfun$com$rbmhtechnology$eventuate$Replicator$$read$1.applyOrElse(ReplicationEndpoint.scala:660)
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
at scala.util.Failure$$anonfun$recover$1.apply(Try.scala:216)
at scala.util.Try$.apply(Try.scala:192)
at scala.util.Failure.recover(Try.scala:216)
at scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324)
at scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:405)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[ERROR] [09/19/2016 07:59:25.376] [location-akka.actor.default-dispatcher-5] [akka.tcp://[email protected]:44041/user/$b/$a/$a] replication failure detection limit reached (3000 milliseconds), publishing Unavailable for B_1/L1 (last exception being reported)
com.rbmhtechnology.eventuate.ReplicationProtocol$ReplicationReadTimeoutException: Replication read timed out after 1000 milliseconds
at com.rbmhtechnology.eventuate.Replicator$$anonfun$com$rbmhtechnology$eventuate$Replicator$$read$1.applyOrElse(ReplicationEndpoint.scala:661)
at com.rbmhtechnology.eventuate.Replicator$$anonfun$com$rbmhtechnology$eventuate$Replicator$$read$1.applyOrElse(ReplicationEndpoint.scala:660)
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
at scala.util.Failure$$anonfun$recover$1.apply(Try.scala:216)
at scala.util.Try$.apply(Try.scala:192)
at scala.util.Failure.recover(Try.scala:216)
at scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324)
at scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:405)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[ERROR] [09/19/2016 07:59:25.383] [location-akka.actor.default-dispatcher-50] [akka.tcp://[email protected]:55845/user/$b/$a/$a] replication failure detection limit reached (3000 milliseconds), publishing Unavailable for C_1/L1 (last exception being reported)
[ERROR] [09/19/2016 07:59:25.386] [location-akka.actor.default-dispatcher-50] [akka.tcp://[email protected]:38829/user/$a/$a/$a] replication failure detection limit reached (3000 milliseconds), publishing Unavailable for A_1/L1 (last exception being reported)
[ERROR] [09/19/2016 07:59:25.387] [location-akka.actor.default-dispatcher-18] [akka.tcp://[email protected]:38829/user/$b/$a/$a] replication failure detection limit reached (3000 milliseconds), publishing Unavailable for C_1/L1 (last exception being reported)
[ERROR] [09/19/2016 07:59:28.386] [location-akka.actor.default-dispatcher-36] [akka.tcp://[email protected]:44041/user/$a/$a/$a] replication failure detection limit reached (3000 milliseconds), publishing Unavailable for A_1/L1 (last exception being reported)
com.rbmhtechnology.eventuate.ReplicationProtocol$ReplicationReadTimeoutException: Replication read timed out after 1000 milliseconds
at com.rbmhtechnology.eventuate.Replicator$$anonfun$com$rbmhtechnology$eventuate$Replicator$$read$1.applyOrElse(ReplicationEndpoint.scala:661)
at com.rbmhtechnology.eventuate.Replicator$$anonfun$com$rbmhtechnology$eventuate$Replicator$$read$1.applyOrElse(ReplicationEndpoint.scala:660)
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
at scala.util.Failure$$anonfun$recover$1.apply(Try.scala:216)
at scala.util.Try$.apply(Try.scala:192)
at scala.util.Failure.recover(Try.scala:216)
at scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324)
at scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:405)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[ERROR] [09/19/2016 07:59:28.393] [location-akka.actor.default-dispatcher-33] [akka.tcp://[email protected]:55845/user/$a/$a/$a] replication failure detection limit reached (3000 milliseconds), publishing Unavailable for B_1/L1 (last exception being reported)
[ERROR] [09/19/2016 07:59:28.395] [location-akka.actor.default-dispatcher-30] [akka.tcp://[email protected]:44041/user/$b/$a/$a] replication failure detection limit reached (3000 milliseconds), publishing Unavailable for B_1/L1 (last exception being reported)
com.rbmhtechnology.eventuate.ReplicationProtocol$ReplicationReadTimeoutException: Replication read timed out after 1000 milliseconds
at com.rbmhtechnology.eventuate.Replicator$$anonfun$com$rbmhtechnology$eventuate$Replicator$$read$1.applyOrElse(ReplicationEndpoint.scala:661)
at com.rbmhtechnology.eventuate.Replicator$$anonfun$com$rbmhtechnology$eventuate$Replicator$$read$1.applyOrElse(ReplicationEndpoint.scala:660)
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
at scala.util.Failure$$anonfun$recover$1.apply(Try.scala:216)
at scala.util.Try$.apply(Try.scala:192)
at scala.util.Failure.recover(Try.scala:216)
at scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324)
at scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:405)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[ERROR] [09/19/2016 07:59:28.403] [location-akka.actor.default-dispatcher-50] [akka.tcp://[email protected]:55845/user/$b/$a/$a] replication failure detection limit reached (3000 milliseconds), publishing Unavailable for C_1/L1 (last exception being reported)
[ERROR] [09/19/2016 07:59:28.406] [location-akka.actor.default-dispatcher-17] [akka.tcp://[email protected]:38829/user/$b/$a/$a] replication failure detection limit reached (3000 milliseconds), publishing Unavailable for C_1/L1 (last exception being reported)
[ERROR] [09/19/2016 07:59:28.406] [location-akka.actor.default-dispatcher-51] [akka.tcp://[email protected]:38829/user/$a/$a/$a] replication failure detection limit reached (3000 milliseconds), publishing Unavailable for A_1/L1 (last exception being reported)
[ERROR] [09/19/2016 07:59:31.406] [location-akka.actor.default-dispatcher-30] [akka.tcp://[email protected]:44041/user/$a/$a/$a] replication failure detection limit reached (3000 milliseconds), publishing Unavailable for A_1/L1 (last exception being reported)
com.rbmhtechnology.eventuate.ReplicationProtocol$ReplicationReadTimeoutException: Replication read timed out after 1000 milliseconds
at com.rbmhtechnology.eventuate.Replicator$$anonfun$com$rbmhtechnology$eventuate$Replicator$$read$1.applyOrElse(ReplicationEndpoint.scala:661)
at com.rbmhtechnology.eventuate.Replicator$$anonfun$com$rbmhtechnology$eventuate$Replicator$$read$1.applyOrElse(ReplicationEndpoint.scala:660)
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
at scala.util.Failure$$anonfun$recover$1.apply(Try.scala:216)
at scala.util.Try$.apply(Try.scala:192)
at scala.util.Failure.recover(Try.scala:216)
at scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324)
at scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:405)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[ERROR] [09/19/2016 07:59:31.413] [location-akka.actor.default-dispatcher-3] [akka.tcp://[email protected]:55845/user/$a/$a/$a] replication failure detection limit reached (3000 milliseconds), publishing Unavailable for B_1/L1 (last exception being reported)
com.rbmhtechnology.eventuate.ReplicationProtocol$ReplicationReadTimeoutException: Replication read timed out after 1000 milliseconds
at com.rbmhtechnology.eventuate.Replicator$$anonfun$com$rbmhtechnology$eventuate$Replicator$$read$1.applyOrElse(ReplicationEndpoint.scala:661)
at com.rbmhtechnology.eventuate.Replicator$$anonfun$com$rbmhtechnology$eventuate$Replicator$$read$1.applyOrElse(ReplicationEndpoint.scala:660)
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
at scala.util.Failure$$anonfun$recover$1.apply(Try.scala:216)
at scala.util.Try$.apply(Try.scala:192)
at scala.util.Failure.recover(Try.scala:216)
at scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324)
at scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:405)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[ERROR] [09/19/2016 07:59:31.416] [location-akka.actor.default-dispatcher-18] [akka.tcp://[email protected]:44041/user/$b/$a/$a] replication failure detection limit reached (3000 milliseconds), publishing Unavailable for B_1/L1 (last exception being reported)
com.rbmhtechnology.eventuate.ReplicationProtocol$ReplicationReadTimeoutException: Replication read timed out after 1000 milliseconds
at com.rbmhtechnology.eventuate.Replicator$$anonfun$com$rbmhtechnology$eventuate$Replicator$$read$1.applyOrElse(ReplicationEndpoint.scala:661)
at com.rbmhtechnology.eventuate.Replicator$$anonfun$com$rbmhtechnology$eventuate$Replicator$$read$1.applyOrElse(ReplicationEndpoint.scala:660)
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
at scala.util.Failure$$anonfun$recover$1.apply(Try.scala:216)
at scala.util.Try$.apply(Try.scala:192)
at scala.util.Failure.recover(Try.scala:216)
at scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324)
at scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:405)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[ERROR] [09/19/2016 07:59:31.426] [location-akka.actor.default-dispatcher-18] [akka.tcp://[email protected]:38829/user/$a/$a/$a] replication failure detection limit reached (3000 milliseconds), publishing Unavailable for A_1/L1 (last exception being reported)
[ERROR] [09/19/2016 07:59:32.323] [location-akka.actor.default-dispatcher-5] [akka.tcp://[email protected]:55845/user/$b/$a/$a] replication failure detection limit reached (3000 milliseconds), publishing Unavailable for C_1/L1 (last exception being reported)
[ERROR] [09/19/2016 07:59:32.326] [location-akka.actor.default-dispatcher-51] [akka.tcp://[email protected]:38829/user/$b/$a/$a] replication failure detection limit reached (3000 milliseconds), publishing Unavailable for C_1/L1 (last exception being reported)
[ERROR] [09/19/2016 07:59:34.425] [location-akka.actor.default-dispatcher-18] [akka.tcp://[email protected]:44041/user/$a/$a/$a] replication failure detection limit reached (3000 milliseconds), publishing Unavailable for A_1/L1 (last exception being reported)
com.rbmhtechnology.eventuate.ReplicationProtocol$ReplicationReadTimeoutException: Replication read timed out after 1000 milliseconds
at com.rbmhtechnology.eventuate.Replicator$$anonfun$com$rbmhtechnology$eventuate$Replicator$$read$1.applyOrElse(ReplicationEndpoint.scala:661)
at com.rbmhtechnology.eventuate.Replicator$$anonfun$com$rbmhtechnology$eventuate$Replicator$$read$1.applyOrElse(ReplicationEndpoint.scala:660)
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
at scala.util.Failure$$anonfun$recover$1.apply(Try.scala:216)
at scala.util.Try$.apply(Try.scala:192)
at scala.util.Failure.recover(Try.scala:216)
at scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324)
at scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:405)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[ERROR] [09/19/2016 07:59:34.433] [location-akka.actor.default-dispatcher-50] [akka.tcp://[email protected]:55845/user/$a/$a/$a] replication failure detection limit reached (3000 milliseconds), publishing Unavailable for B_1/L1 (last exception being reported)
[ERROR] [09/19/2016 07:59:34.435] [location-akka.actor.default-dispatcher-16] [akka.tcp://[email protected]:44041/user/$b/$a/$a] replication failure detection limit reached (3000 milliseconds), publishing Unavailable for B_1/L1 (last exception being reported)
com.rbmhtechnology.eventuate.ReplicationProtocol$ReplicationReadTimeoutException: Replication read timed out after 1000 milliseconds
at com.rbmhtechnology.eventuate.Replicator$$anonfun$com$rbmhtechnology$eventuate$Replicator$$read$1.applyOrElse(ReplicationEndpoint.scala:661)
at com.rbmhtechnology.eventuate.Replicator$$anonfun$com$rbmhtechnology$eventuate$Replicator$$read$1.applyOrElse(ReplicationEndpoint.scala:660)
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
at scala.util.Failure$$anonfun$recover$1.apply(Try.scala:216)
at scala.util.Try$.apply(Try.scala:192)
at scala.util.Failure.recover(Try.scala:216)
at scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324)
at scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:405)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[ERROR] [09/19/2016 07:59:34.446] [location-akka.actor.default-dispatcher-50] [akka.tcp://[email protected]:38829/user/$a/$a/$a] replication failure detection limit reached (3000 milliseconds), publishing Unavailable for A_1/L1 (last exception being reported)
[ERROR] [09/19/2016 07:59:35.343] [location-akka.actor.default-dispatcher-3] [akka.tcp://[email protected]:55845/user/$b/$a/$a] replication failure detection limit reached (3000 milliseconds), publishing Unavailable for C_1/L1 (last exception being reported)
[ERROR] [09/19/2016 07:59:35.346] [location-akka.actor.default-dispatcher-20] [akka.tcp://[email protected]:38829/user/$b/$a/$a] replication failure detection limit reached (3000 milliseconds), publishing Unavailable for C_1/L1 (last exception being reported)
[0m[[0minfo[0m] [0m[31m- must support bi-directional cyclic replication networks *** FAILED ***[0m[0m
[0m[[0minfo[0m] [0m[31m java.lang.AssertionError: assertion failed: timeout (20 seconds) while expecting 297 messages (got 219)[0m[0m
[0m[[0minfo[0m] [0m[31m at scala.Predef$.assert(Predef.scala:165)[0m[0m
[0m[[0minfo[0m] [0m[31m at akka.testkit.TestKitBase$$anonfun$receiveN_internal$1.apply(TestKit.scala:645)[0m[0m
[0m[[0minfo[0m] [0m[31m at akka.testkit.TestKitBase$$anonfun$receiveN_internal$1.apply(TestKit.scala:642)[0m[0m
[0m[[0minfo[0m] [0m[31m at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)[0m[0m
[0m[[0minfo[0m] [0m[31m at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)[0m[0m
[0m[[0minfo[0m] [0m[31m at scala.collection.immutable.Range.foreach(Range.scala:166)[0m[0m
[0m[[0minfo[0m] [0m[31m at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)[0m[0m
[0m[[0minfo[0m] [0m[31m at scala.collection.AbstractTraversable.map(Traversable.scala:104)[0m[0m
[0m[[0minfo[0m] [0m[31m at akka.testkit.TestKitBase$class.receiveN_internal(TestKit.scala:642)[0m[0m
[0m[[0minfo[0m] [0m[31m at akka.testkit.TestKitBase$class.expectMsgAllOf_internal(TestKit.scala:509)[0m[0m
[0m[[0minfo[0m] [0m[31m ...[0m[0m
[ERROR] [09/19/2016 07:59:35.694] [location-akka.remote.default-remote-dispatcher-7] [akka.tcp://[email protected]:38829/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Flocation%40127.0.0.1%3A44041-0/endpointWriter] AssociationError [akka.tcp://[email protected]:38829] -> [akka.tcp://[email protected]:44041]: Error [Shut down address: akka.tcp://[email protected]:44041] [
akka.remote.ShutDownAssociation: Shut down address: akka.tcp://[email protected]:44041
Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down.
]
[ERROR] [09/19/2016 07:59:35.694] [location-akka.remote.default-remote-dispatcher-7] [akka.tcp://[email protected]:38829/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Flocation%40127.0.0.1%3A44041-3] AssociationError [akka.tcp://[email protected]:38829] <- [akka.tcp://[email protected]:44041]: Error [Shut down address: akka.tcp://[email protected]:44041] [
akka.remote.ShutDownAssociation: Shut down address: akka.tcp://[email protected]:44041
Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down.
]
[ERROR] [09/19/2016 07:59:35.693] [location-akka.remote.default-remote-dispatcher-9] [akka.tcp://[email protected]:55845/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Flocation%40127.0.0.1%3A44041-3] AssociationError [akka.tcp://[email protected]:55845] <- [akka.tcp://[email protected]:44041]: Error [Shut down address: akka.tcp://[email protected]:44041] [
akka.remote.ShutDownAssociation: Shut down address: akka.tcp://[email protected]:44041
Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down.
]
[ERROR] [09/19/2016 07:59:35.693] [location-akka.remote.default-remote-dispatcher-9] [akka.tcp://[email protected]:55845/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Flocation%40127.0.0.1%3A44041-1/endpointWriter] AssociationError [akka.tcp://[email protected]:55845] -> [akka.tcp://[email protected]:44041]: Error [Shut down address: akka.tcp://[email protected]:44041] [
akka.remote.ShutDownAssociation: Shut down address: akka.tcp://[email protected]:44041
Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down.
]
[ERROR] [09/19/2016 07:59:37.453] [location-akka.actor.default-dispatcher-50] [akka.tcp://[email protected]:55845/user/$a/$a/$a] replication failure detection limit reached (3000 milliseconds), publishing Unavailable for B_1/L1 (last exception being reported)
[ERROR] [09/19/2016 07:59:38.363] [location-akka.actor.default-dispatcher-3] [akka.tcp://[email protected]:55845/user/$b/$a/$a] replication failure detection limit reached (3000 milliseconds), publishing Unavailable for C_1/L1 (last exception being reported)
[ERROR] [09/19/2016 07:59:40.473] [location-akka.actor.default-dispatcher-5] [akka.tcp://[email protected]:55845/user/$a/$a/$a] replication failure detection limit reached (3000 milliseconds), publishing Unavailable for B_1/L1 (last exception being reported)
[ERROR] [09/19/2016 07:59:41.383] [location-akka.actor.default-dispatcher-50] [akka.tcp://[email protected]:55845/user/$b/$a/$a] replication failure detection limit reached (3000 milliseconds), publishing Unavailable for C_1/L1 (last exception being reported)
[ERROR] [09/19/2016 07:59:43.493] [location-akka.actor.default-dispatcher-3] [akka.tcp://[email protected]:55845/user/$a/$a/$a] replication failure detection limit reached (3000 milliseconds), publishing Unavailable for B_1/L1 (last exception being reported)
[ERROR] [09/19/2016 07:59:44.403] [location-akka.actor.default-dispatcher-5] [akka.tcp://[email protected]:55845/user/$b/$a/$a] replication failure detection limit reached (3000 milliseconds), publishing Unavailable for C_1/L1 (last exception being reported)
[ERROR] [09/19/2016 07:59:46.512] [location-akka.actor.default-dispatcher-3] [akka.tcp://[email protected]:55845/user/$a/$a/$a] replication failure detection limit reached (3000 milliseconds), publishing Unavailable for B_1/L1 (last exception being reported)
[ERROR] [09/19/2016 07:59:47.423] [location-akka.actor.default-dispatcher-5] [akka.tcp://[email protected]:55845/user/$b/$a/$a] replication failure detection limit reached (3000 milliseconds), publishing Unavailable for C_1/L1 (last exception being reported)
[ERROR] [09/19/2016 07:59:49.533] [location-akka.actor.default-dispatcher-50] [akka.tcp://[email protected]:55845/user/$a/$a/$a] replication failure detection limit reached (3000 milliseconds), publishing Unavailable for B_1/L1 (last exception being reported)
[ERROR] [09/19/2016 07:59:50.433] [location-akka.actor.default-dispatcher-3] [akka.tcp://[email protected]:55845/user/$b/$a/$a] replication failure detection limit reached (3000 milliseconds), publishing Unavailable for C_1/L1 (last exception being reported)
[ERROR] [09/19/2016 07:59:52.553] [location-akka.actor.default-dispatcher-50] [akka.tcp://[email protected]:55845/user/$a/$a/$a] replication failure detection limit reached (3000 milliseconds), publishing Unavailable for B_1/L1 (last exception being reported)
[ERROR] [09/19/2016 07:59:53.453] [location-akka.actor.default-dispatcher-3] [akka.tcp://[email protected]:55845/user/$b/$a/$a] replication failure detection limit reached (3000 milliseconds), publishing Unavailable for C_1/L1 (last exception being reported)
[0m[[0minfo[0m] [0m[31mException encountered when attempting to run a suite with class name: com.rbmhtechnology.eventuate.ReplicationCycleSpecLeveldb *** ABORTED ***[0m[0m
[0m[[0minfo[0m] [0m[31m java.util.concurrent.TimeoutException: Futures timed out after [10 seconds][0m[0m
[0m[[0minfo[0m] [0m[31m at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)[0m[0m
[0m[[0minfo[0m] [0m[31m at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)[0m[0m
[0m[[0minfo[0m] [0m[31m at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:190)[0m[0m
[0m[[0minfo[0m] [0m[31m at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)[0m[0m
[0m[[0minfo[0m] [0m[31m at scala.concurrent.Await$.result(package.scala:190)[0m[0m
[0m[[0minfo[0m] [0m[31m at com.rbmhtechnology.eventuate.MultiLocationSpec$$anonfun$afterEach$1.apply(LocationSpec.scala:142)[0m[0m
[0m[[0minfo[0m] [0m[31m at com.rbmhtechnology.eventuate.MultiLocationSpec$$anonfun$afterEach$1.apply(LocationSpec.scala:142)[0m[0m
[0m[[0minfo[0m] [0m[31m at scala.collection.immutable.List.foreach(List.scala:381)[0m[0m
[0m[[0minfo[0m] [0m[31m at com.rbmhtechnology.eventuate.MultiLocationSpec$class.afterEach(LocationSpec.scala:142)[0m[0m
[0m[[0minfo[0m] [0m[31m at com.rbmhtechnology.eventuate.ReplicationCycleSpecLeveldb.afterEach(LocationSpecsLeveldb.scala:52)[0m[0m
[0m[[0minfo[0m] [0m[31m ...[0m[0m