Skip to content

Latest commit

 

History

History

rss-feed-into-kafka

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
README.adoc
README.adoc
 
 
docker-compose.yml
docker-compose.yml
 
 

README.adoc

Streaming RSS into Kafka

Introduction

Using the Kafka Connect RSS connector you can stream the contents of an RSS feed into a Kafka topic.

This demo includes several sources:

  • U.S. Securities and Exchange Commission feed

  • BBC News feed

  • HackerNews new posts

Running

docker-compose up -d

KSQL statements

  • You can use Control Center (http://localhost:9021) or launch KSQL CLI:

    docker-compose exec ksql-cli bash -c 'echo -e "\n\n⏳ Waiting for KSQL to be available before launching CLI\n"; while [ $(curl -s -o /dev/null -w %{http_code} http://ksql-server:8088/) -eq 000 ] ; do echo -e $(date) "KSQL Server HTTP state: " $(curl -s -o /dev/null -w %{http_code} http://ksql-server:8088/) " (waiting for 200)" ; sleep 5 ; done; ksql http://ksql-server:8088'

  • Create streams

    CREATE STREAM BBC_NEWS_UK WITH (KAFKA_TOPIC='bbc-news-uk',VALUE_FORMAT='AVRO');
CREATE STREAM MUTUAL_FUND_FILINGS WITH (KAFKA_TOPIC='xbrl-rr',VALUE_FORMAT='AVRO');
CREATE STREAM HACKERNEWS WITH (KAFKA_TOPIC='hackernews',VALUE_FORMAT='AVRO');

  • Query

    SELECT FEED->TITLE,
       DATE,
       TITLE,
       CONTENT,
       LINK
  FROM BBC_NEWS_UK
 LIMIT 5;
    BBC News - UK | 2019-05-17T14:47:10Z | Labour-Tory Brexit talks end without deal | Jeremy Corbyn says discussions "have gone as far as they can", while Theresa May calls them "difficult".
BBC News - UK | 2019-05-17T15:33:15Z | London Bridge inquest: Nurse killed as she helped other victim | Kirsty Boden said "I have to go help", but was stabbed as she knelt over a dying victim, an inquest hears.
BBC News - UK | 2019-05-17T15:18:53Z | Foreign Office warns against Iran travel for British-Iranians | British-Iranian nationals are advised against going to Iran because of an "intolerable risk" of mistreatment.