Skip to content

Latest commit

 

History

History
70 lines (52 loc) · 1.88 KB

spark-sql-streaming-StreamingQueryProgress.adoc

File metadata and controls

70 lines (52 loc) · 1.88 KB

StreamingQueryProgress

StreamingQueryProgress holds information about the progress of a streaming query.

StreamingQueryProgress is created exclusively when StreamExecution finishes a trigger.

Note

Use lastProgress property of a StreamingQuery to access the most recent StreamingQueryProgress update.

val sq: StreamingQuery = ...
sq.lastProgress
Note

Use recentProgress property of a StreamingQuery to access the most recent StreamingQueryProgress updates.

val sq: StreamingQuery = ...
sq.recentProgress
Note

Use StreamingQueryListener to get notified about StreamingQueryProgress updates while a streaming query is executed.

Table 1. StreamingQueryProgress’s Properties
Name Description

id

Unique identifier of a streaming query

runId

Unique identifier of the current execution of a streaming query

name

Optional query name

timestamp

Time when the trigger has started (in ISO8601 format).

batchId

Unique id of the current batch

durationMs

Durations of the internal phases (in milliseconds)

eventTime

Statistics of event time seen in this batch

stateOperators

Information about stateful operators in the query that store state.

sources

Statistics about the data read from every streaming source in a streaming query

sink

Information about progress made for a sink