Data Ops

Project Description

• This prject is for:
  • Data Pipeline Service with Apache Kafka, Postgres, and MongoDB.
  • API service, and a
  • Next.js Application.

Prerequisites

• Before diving into the project, ensure you have the following prerequisites:
  1. Docker — Docker Compose
  2. Apache Kafka (Stream Processing)
  3. PostgreSQL
  5. Superset(Report/Dashboard)
  6. Kafka UI ( Kafka Monitoring)
  7. Grafana (System Monitoring)

Setup

• The project has the following services running as Docker containers

  1. mongo
  2. postgres
  3. pgadmin
     • dpage/pgadmin4 is a web based administration tool for the PostgreSQL database.
  4. kafka
  5. kafka-ui
  6. zookeeper
  7. schema-registry
  8. sale-order-api
  9. customer-api
  10. nextjs.app

1. PostgreSQL Docker Container

• Check my github.com/nyangweso-rodgers - Running PostgreSQL Docker Container, GitHub repo on how to configure and run postgresql docker container using docker-compose.

  services:

Connect to a Postgres Docker Container

• To connect to a PostgreSQL instance running within a Docker container, you can use the docker exec command combined with the psql command:
• Example:

    #accessing postgres docker container
    docker exec -it postgres psql -U admin -d test_db

• Remarks:
  • Check my GitHub Repo for a list of psql commands

2. MongoDB Docker Container

• Check github.com/nyangweso-rodgers - Run MongoDB Docker Container repo on how to run a mongo docker container using docker-compose.
• Check github.com/nyangweso-rodgers - mongoDB replica set repo, to successfully set up a MongoDB replica set with Docker Compose. This ensures that you have a highly available and resilient MongoDB deployment.

  services:

3. Zookeeper

services:

4. Kafka

services:

5. Schema Registry

services:

6. Debezium

services:

Register Debezium Connector

• To register the above connector, run the below curl commands:

    curl -X POST --location "http://localhost:8083/connectors" -H "Content-Type: application/json" -H "Accept: application/json" -d @register-customer-postgresdb-connector.json

• and

    curl -X POST --location "http://localhost:8083/connectors" -H "Content-Type: application/json" -H "Accept: application/json" -d @register-delegates-survey-postgresdb-connector.json

7. Kafka UI

services:

8. Debezium UI

services:

Build Dashboard

• We can build the dashboards using the following tools:
  1. Metabase
  2. Superset
  3. Redash
  4. Tableau
  5. Power BI

9. Metabase Docker Container

services:

Access Metabase

• Once the Docker Compose is up and running, you can access Metabase at http://localhost:3000 in your web browser.

Connect Metabase to PostgreSQL

• When you first open Metabase, it will ask you to setup a connection to your database. Here are the settings you need to use:
  1. Database type: PostgreSQL
  2. Host: postgres
  3. Port: 5432
  4. Database name:
  5. Username:
  6. Password:
• Now, you should be able to explore your PostgreSQL data using Metabase!

Resources and Further Reading

1. github.com/nyangweso-rodgers - Docker-Commands

2. github.com/nyangweso-rodgers - Setting Express.js Development Environment

3. github.com/nyangweso-rodgers - Docker Compose File