spark-sql-streaming-StreamingQueryListener.adoc

StreamingQueryListener — Intercepting Streaming Events

StreamingQueryListener is the contract for listeners that want to be notified about the life cycle events of streaming queries, i.e. start, progress and termination of a query.

package org.apache.spark.sql.streaming

abstract class StreamingQueryListener {
  def onQueryStarted(event: QueryStartedEvent): Unit
  def onQueryProgress(event: QueryProgressEvent): Unit
  def onQueryTerminated(event: QueryTerminatedEvent): Unit
}

Table 1. StreamingQueryListener’s Life Cycle Events and Callbacks

Event	Callback	When Posted
QueryStartedEvent id runId name	onQueryStarted	Right after StreamExecution has started running streaming batches.
QueryProgressEvent StreamingQueryProgress	onQueryProgress	ProgressReporter reports query progress (which is when StreamExecution runs batches and a trigger has finished).
QueryTerminatedEvent id runId Optional exception if terminated due to an error	onQueryTerminated	Right before StreamExecution finishes running streaming batches (due to a stop or an exception).

You can register a StreamingQueryListener using StreamingQueryManager.addListener method.

val queryListener: StreamingQueryListener = ...
spark.streams.addListener(queryListener)

You can remove a StreamingQueryListener using StreamingQueryManager.removeListener method.

val queryListener: StreamingQueryListener = ...
spark.streams.removeListener(queryListener)

Figure 1. StreamingQueryListener Notified about Query’s Start (onQueryStarted)

Note	onQueryStarted is used internally to unblock the starting thread of StreamExecution.

Figure 2. StreamingQueryListener Notified about Query’s Progress (onQueryProgress)

Figure 3. StreamingQueryListener Notified about Query’s Termination (onQueryTerminated)

Note	You can also register a streaming event listener using the general SparkListener interface. Details on SparkListener interface can be found in the Mastering Apache Spark 2 gitbook.