A skeleton example of how to use Kafka Streams with Event-Driven microservices.
This is an application that generates a simple football statistics like match scores, team and player rankings. It should be a little more comprehensible than a typical code examples based on orders, payments etc.
You can think of this application as a supplement to the basic code examples of Kafka Streams. Kafka Streams is used here extensively for many purposes like data processing, building materialized views, persisting the domain model and exposing the output of services. Furthermore, there is no traditional database, local state stores instead (PostgreSQL plays a role of legacy system only).
This codebase is trying to apply the idea of stateful streams processing with Kafka and Kafka Streams. These blog posts are a great introduction to that concept:
- The Data Dichotomy: Rethinking the Way We Treat Data and Services
- Build Services on a Backbone of Events
- Using Apache Kafka as a Scalable, Event-Driven Backbone for Service Architectures
- Event sourcing using Kafka
- kafka,
- zookeeper,
- postgres - in the role of an external data source,
- football-match - REST requests transformation into events, input validation with the domain model,
- connect - Kafka Connect with Debezium PostgreSQL connector, writes events for insert or update operations on PLAYERS table to a single Kafka topic,
- football-player is receiving notifications from connect service and creating only a single PlayerStartedCareer event using Processor API (see the code),
- football-view-basic - builds the statistics from the events using Kafka Streams DSL, it's the main place where streams are created and processed (see below),
- football-view-top - builds the Top 10 Players ranking based on the statistics generated by football-view-basic and not on the events,
- football-ui - saves the statistics into materialized views (local state stores), exposes them via REST API and publishes using WebSocket.
Additional modules:
- football-common - contains some code that is shared among microservices (obviously it's not a good practice for production), especially:
- events are used by football-match, football-player and football_view,
- statistics - football-view-basic, football-view-top, football-ui,
- football-tests - demo application and integration tests.
Each event represents a state change that occurred to the system.
| Event | Fields |
|---|---|
| MatchScheduled | match id, season, home club, away club, matchDate |
| MatchStarted | match id, home club, away club |
| GoalScored | goal id, match id, minute, scorer, scored for |
| CardReceived | card id, match id, minute, receiver, type (yellow/red) |
| MatchFinished | match id |
| PlayerStartedCareer | palyer id, name |
These events are the source for stream processing. For example, in order to determine the result of a match, you must join MatchStarted and GoalScored streams and then count the goals (see the code).
Topics are organized under logical groups as follows:
- fb-event.* - containing events generated by football-match and football-player,
- fb-view.* - statistics produced by football-view-basic and football-view-top,
- fb-connect.* - events from the PostgreSQL connector.
The prefixes above aim to separate the application topics from the internal Kafka Streams and Debezium topics.
There are only two REST endpoits and both of them are accessed from the outside of the system:
- Query interface in football-ui,
- Command interface in football-match.
Requirements: Java 10, Docker with at least 6 GB memory limit.
-
Add kafka bootstrap address to your
/etc/hosts:127.0.0.1 kafka 127.0.0.1 postgres 127.0.0.1 connect 127.0.0.1 football-match 127.0.0.1 football-player 127.0.0.1 football-view-basic 127.0.0.1 football-view-top 127.0.0.1 football-ui -
Build microservices and Docker containers:
cd football mvn install -
Build and run the demo application:
cd football-tests mvn install -DskipTests java -jar target/football-tests-0.0.1-SNAPSHOT-jar-with-dependencies.jarIt will start Docker containers and wait until all microservices are ready.
If the default startup timeout (180 sec) will be insufficient, try to increase it:
java -jar target/football-tests-0.0.1-SNAPSHOT-jar-with-dependencies.jar 400 -
Wait a minute and launch http://football-ui:18080/. In a few seconds you should see some statistics updated in real time:
The input is historical data from English Football League Championship. The historical events (goal scored, card received etc.) are played back every few seconds.