Skip to content

High-Throughput Event Ingestion Engine via Apache Kafka and gRPC #30

Description

@Fatimasanusi

Description:
Polling blockchain RPC nodes individually for real-time liquidity changes introduces high latency and RPC throttling. We need to implement an event-driven ingestion pipeline that streams live block transactions over gRPC into a Kafka broker cluster for async parsing.

Acceptance Criteria:

  • Build a gRPC subscriber client connecting to blockchain indexers (e.g., Yellowstone Geyser for Solana or EVM WebSocket streams).
  • Produce normalized block events into a partitioned Apache Kafka topic.
  • Implement a multi-threaded Rust consumer group that processes transactions asynchronously without blocking the REST API runtime.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions