spark-sql-streaming-KafkaSourceRDD.adoc

KafkaSourceRDD

KafkaSourceRDD is an RDD of Kafka’s ConsumerRecords (with keys and values being collections of bytes, i.e. Array[Byte]).

KafkaSourceRDD is created when:

• KafkaRelation buildScan

• KafkaSource getBatch

getPreferredLocations Method

Caution

FIXME

compute Method

Caution

FIXME

getPartitions Method

Caution

FIXME

persist Method

Caution

FIXME

Creating KafkaSourceRDD Instance

KafkaSourceRDD takes the following when created:

• SparkContext

• Collection of key-value settings for executors reading records from Kafka topics

• Collection of KafkaSourceRDDOffsetRange offsets

• Timeout (in milliseconds) to poll data from Kafka

  Used when KafkaSourceRDD is requested for records (for given offsets) and in turn requests CachedKafkaConsumer to poll for Kafka’s ConsumerRecords.

• Flag to…​FIXME

• Flag to…​FIXME

KafkaSourceRDD initializes the internal registries and counters.