Indefinite errors occur if the AWS Logs/Flow Logs inputs are quickly started/stoped

[danotorrey]

The AWS Logs/Flow Logs inputs do not properly handle rapid start/stop attempts. If the input is started, then stopped and started again (possibly with an accidental double-click), and instance of the AWS Kinesis worker will enter into a persistent error state and log the following message every few seconds forever. The only way to stop this currently is to completely restart the Graylog server.

This is not a common scenario (and I don't think this is a high-priority issue), but I wanted to capture it as an issue, so that perhaps it could be addressed some day.

java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@278fd2ee rejected from java.util.concurrent.ThreadPoolExecutor@23b05240[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) ~[?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) ~[?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) ~[?:1.8.0_181]
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134) ~[?:1.8.0_181]
at com.amazonaws.services.kinesis.leases.impl.LeaseRenewer.renewLeases(LeaseRenewer.java:105) ~[amazon-kinesis-client-1.9.0.jar:?]
at com.amazonaws.services.kinesis.leases.impl.LeaseCoordinator.runRenewer(LeaseCoordinator.java:258) ~[amazon-kinesis-client-1.9.0.jar:?]
at com.amazonaws.services.kinesis.leases.impl.LeaseCoordinator$RenewerRunnable.run(LeaseCoordinator.java:182) [amazon-kinesis-client-1.9.0.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_181]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_181]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_181]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]

[danotorrey]

I talked to @bernd about this. This is a challenge to fix, because the Kinesis worker maintains it's own worker thread pool. Shutdown requires termination of those and release of other resources. We cannot simply terminate the parent worker.

Perhaps pausing the worker startup until previous instances have shut down is an option.