spark-sql-streaming-GroupStateTimeout.adoc

GroupStateTimeout

GroupStateTimeout represents the possible timeouts that you can use for the state-aware Dataset operations:

• mapGroupsWithState

• flatMapGroupsWithState

GroupStateTimeout is part of org.apache.spark.sql.streaming package.

import org.apache.spark.sql.streaming.GroupStateTimeout

Table 1. Types of GroupStateTimeouts (in alphabetical order)

GroupStateTimeout	Description
ProcessingTimeTimeout	Timeout based on the processing time. Note FlatMapGroupsWithStateExec requires that batchTimestampMs is specified when ProcessingTimeTimeout is used. batchTimestampMs is defined when IncrementalExecution is created (and so is state). IncrementalExecution is given OffsetSeqMetadata when StreamExecution runs a streaming batch. Caution FIXME Describe OffsetSeqMetadata and StreamExecution.offsetSeqMetadata
EventTimeTimeout	Timeout based on the event time Used when…​FIXME
NoTimeout	No timeout Used when…​FIXME