Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automatic thread dump when JMS listener has "no thread available" #832

Open
donaldoakes opened this issue Apr 3, 2020 · 0 comments
Open
Milestone

Comments

@donaldoakes
Copy link
Collaborator

donaldoakes commented Apr 3, 2020

When a JMS listener indicates "no thread available" as in the stack trace below, periodically trigger a thread dump to understand what's tying up workflow threads. The first time the scenario occurs, we should trigger the thread dump. Thereafter, to avoid polluting stdout we should keep a counter, and when the counter reaches a certain config-driven point, trigger another dump and reset the counter. Or maybe we just trigger the first time, and then again when the backup queue is full.

java.lang.Exception: Scheduler has no thread available for Unscheduled event: InternalEvent.262690909startproc203942469 message:
<EventMessage xmlns="http://mdw.centurylink.com/bpm"><EventType>START</EventType>...
	at com.centurylink.mdw.services.event.EventServicesImpl.processInternalEvents(EventServicesImpl.java:700) [mdw-services-6.1.31.jar!/:na]
	at com.centurylink.mdw.timer.startup.UnscheduledEventMonitor.processUnscheduledEvents(UnscheduledEventMonitor.java:74) [mdw-services-6.1.31.jar!/:na]
	at com.centurylink.mdw.timer.startup.UnscheduledEventMonitor.run(UnscheduledEventMonitor.java:60) [mdw-services-6.1.31.jar!/:na]
	at java.util.TimerThread.mainLoop(Timer.java:555) [na:1.8.0_222]
	at java.util.TimerThread.run(Timer.java:505) [na:1.8.0_222]
@donaldoakes donaldoakes added this to the 6.1.34 milestone Apr 3, 2020
@donaldoakes donaldoakes self-assigned this Apr 3, 2020
@donaldoakes donaldoakes modified the milestones: 6.1.34, 6.1.35 May 1, 2020
@donaldoakes donaldoakes modified the milestones: 6.1.35, 6.1.36 May 14, 2020
@donaldoakes donaldoakes modified the milestones: 6.1.36, 6.x Jun 2, 2020
@donaldoakes donaldoakes removed their assignment Jun 2, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant