spark-sql-streaming-DataSource.adoc

DataSource — Pluggable Data Source

DataSource is…​FIXME

DataSource is created when…​FIXME

Tip	Read DataSource — Pluggable Data Sources (for Spark SQL’s batch queries).

Table 1. DataSource’s Internal Registries and Counters

Name	Description
sourceInfo	SourceInfo with the name, the schema, and optional partitioning columns of a source. Used when: DataSource creates a FileStreamSource (that requires the schema and the optional partitioning columns) StreamingRelation is created (for a DataSource)

Creating DataSource Instance

DataSource takes the following when created:

• FIXME

DataSource initializes the internal registries and counters.

createSource Method

Caution

FIXME

Describing Name and Schema of Streaming Source — sourceSchema Internal Method

sourceSchema(): SourceInfo

sourceSchema…​FIXME

Note	sourceSchema is used exclusively when DataSource is requested SourceInfo.