spark-sql-streaming-StateStoreWriter.adoc

StateStoreWriter — Recording Metrics For Writing to StateStore

StateStoreWriter is a contract for physical operators (i.e. SparkPlan) to record metrics when writing to a StateStore.

Table 1. StateStoreWriter’s SQLMetrics

Name	Description
numOutputRows	Number of output rows
numTotalStateRows	number of total state rows
numUpdatedStateRows	number of updated state rows
allUpdatesTimeMs	total time to update rows
allRemovalsTimeMs	total time to remove rows
commitTimeMs	time to commit changes
stateMemory	memory used by state (store)

Setting StateStore-Specific Metrics for Physical Operator — setStoreMetrics Method

setStoreMetrics(store: StateStore): Unit

setStoreMetrics requests store for metrics to use them to record the following metrics of a physical operator:

• numTotalStateRows as StateStore.numKeys

• stateMemory as StateStore.memoryUsedBytes

setStoreMetrics records the implementation-specific metrics.

Note	setStoreMetrics is used when: FlatMapGroupsWithStateExec is executed StateStoreSaveExec is executed StreamingDeduplicateExec is executed