spark-sql-streaming-StreamSourceProvider.adoc

StreamSourceProvider — Streaming Data Source Provider

StreamSourceProvider is the contract for objects that can create a streaming data source for a format (e.g. text file) or system (e.g. Apache Kafka) by their short names.

StreamSourceProvider is used when DataSource is requested for the name and schema of a streaming source or just creates one.

Table 1. Streaming Source Providers (in alphabetical order)

Name	Description
KafkaSourceProvider	Creates KafkaSourceProvider for kafka format.
TextSocketSourceProvider	Creates TextSocketSources for socket format.

StreamSourceProvider Contract

trait StreamSourceProvider {
  def sourceSchema(
      sqlContext: SQLContext,
      schema: Option[StructType],
      providerName: String,
      parameters: Map[String, String]): (String, StructType)
  def createSource(
      sqlContext: SQLContext,
      metadataPath: String,
      schema: Option[StructType],
      providerName: String,
      parameters: Map[String, String]): Source
}

Note	StreamSourceProvider is an experimental contract.

Table 2. StreamSourceProvider Contract (in alphabetical order)

Method	Description
createSource	Creates a streaming source for a format or system (to continually read data). Note metadataPath is the value of the optional user-specified checkpointLocation option or resolved by StreamingQueryManager. Used exclusively when Spark SQL’s DataSource is requested for a Source for a StreamSourceProvider (which is when StreamingRelation is requested for a logical plan). Tip Read DataSource — Pluggable Data Sources.
sourceSchema	Defines the name and the schema of a streaming source