spark-sql-streaming-StreamingQueryListenerBus.adoc

StreamingQueryListenerBus — Notification Bus for Streaming Events

StreamingQueryListenerBus is an event bus (i.e. ListenerBus) to dispatch streaming events to StreamingQueryListener streaming event listeners.

StreamingQueryListenerBus is created when StreamingQueryManager is created (as the internal listenerBus).

Figure 1. StreamingQueryListenerBus is Created Once In SparkSession

StreamingQueryListenerBus is also a SparkListener and registers itself with LiveListenerBus to intercept a QueryStartedEvent.

Table 1. StreamingQueryListenerBus’s Internal Registries and Counters

Name	Description
activeQueryRunIds	Collection of active streaming queries by their runIds. runId is added when StreamingQueryListenerBus posts a QueryStartedEvent event to LiveListenerBus runId is removed when StreamingQueryListenerBus postToAll a QueryTerminatedEvent event Used mainly when StreamingQueryListenerBus dispatches an event to listeners (for queries started in the same SparkSession).

Posting StreamingQueryListener Events to LiveListenerBus — post Method

post(event: StreamingQueryListener.Event): Unit

post simply posts the input event straight to LiveListenerBus except QueryStartedEvent events.

For QueryStartedEvent events, post adds the query’s runId to activeQueryRunIds registry first before posting the event to LiveListenerBus followed by postToAll.

Note	post is used exclusively when StreamingQueryManager posts StreamingQueryListener event.

onOtherEvent Method

Caution

FIXME

doPostEvent Method

Caution

FIXME

postToAll Method

Caution

FIXME

Creating StreamingQueryListenerBus Instance

StreamingQueryListenerBus takes the following when created:

• LiveListenerBus

StreamingQueryListenerBus registers itself with LiveListenerBus.

StreamingQueryListenerBus initializes the internal registries and counters.