Faust-Docker-Compose

An example to show how to include a faust project as a service using docker compose, with Kafka, Zookeeper and Schema Registry

Notice that everything runs using docker-compose, including the faust example application. For local development is preferable to run the kafka cluster separate from the faust app.

If you want to generate a faust project from scratch, please use the cookiecutter-faust

Read more about Faust here: https://github.com/robinhood/faust

Project

The project skeleton is defined as a medium/large project according to faust layout

The setup.py has the entrypoint to resolve the entrypoint problem

Applications

• Page Views: This application corresponds to Tutorial: Count page views
• Leader Election: This application corresponds to Tutorial: Leader Election
• Users: This is a custom application to demostrate how to integrate Faust with Avro Schema.

Faust Project Dockerfile

The Dockerfile is based on python:3.7-slim. The most important here is that the entrypoint will wait for kafka too be ready and after that execute the script run.sh

Docker compose

docker-compose.yaml includes zookeepeer, kafka and schema-registry based on confluent-inc. For more information you can go to confluentinc and see the docker compose example here

Useful ENVIRONMENT variables that you may change:

Variable	description	example
WORKER	Entrypoint in setup.py	example
WORKER_PORT	Worker port	6066
KAFKA_BOOSTRAP_SERVER	Kafka servers	kafka://kafka:9092
KAFKA_BOOSTRAP_SERVER_NAME	Kafka server name	kafka
KAFKA_BOOSTRAP_SERVER_PORT	Kafka server port	9092
SCHEMA_REGISTRY_SERVER	Schema registry server name	schema-registry
SCHEMA_REGISTRY_SERVER_PORT	Schema registry server port	8081
SCHEMA_REGISTRY_URL	Schema Registry Server url	http://schema-registry:8081

Commands

• Start application: make run-dev. This command start both the Page Views and Leader Election applications
• Stop and remove containers: make clean
• List topics: make list-topics
• Send events to page_view topic/agent: make send-page-view-event payload='{"id": "foo", "user": "bar"}'

Avro Schemas, Custom Codecs and Serializers

Because we want to be sure that the message that we encode are valid we use Avro Schemas. Avro is used to define the data schema for a record's value. This schema describes the fields allowed in the value, along with their data types.

For our demostration in the Users application we are using the following schema:

{
    "type": "record",
    "namespace": "com.example",
    "name": "AvroUsers",
    "fields": [
        {"name": "first_name", "type": "string"},
        {"name": "last_name", "type": "string"}
    ]
}

In order to use avro schemas with Faust we need to define a custom codec, a custom serializer and be able to talk with the schema-registry. You can find the custom codec called avro_users registered using the codec registation approach described by faust. The AvroSerializer is in charge to encode and decode messages using the schema registry client.

Now the final step is to integrate the faust model with the AvroSerializer.

# users.models

class UserModel(faust.Record, serializer='avro_users'):
    first_name: str
    last_name: str

Now our application is able to send and receive message using arvo schemas!!!! :-)

Tests

Run tests with tox. Make sure that you have installed it.

tox

Achievements

• Application examples
• Integration with Schma Registry
• Schema Registry Client
• Custom codecs
• Custom Serializers
• Avro Schemas
• Make Schema Registry Client and Serializers a python package