Skip to content

Latest commit

 

History

History
673 lines (511 loc) · 19.8 KB

README.md

File metadata and controls

673 lines (511 loc) · 19.8 KB

Firehose

A type-safe event processing framework for Go. Build composable event pipelines with conditional execution, middleware support, and multiple naming conventions.

Choose the convention that fits your team — SQL, BDD (Given-When-Then), Kafka Streams, or MapReduce — all express the same pipeline.

Problem

Applications process events from various sources (HTTP requests, message queues, timers, system events, user input) and react with side effects. Without a structured approach, event handling becomes scattered across the codebase, difficult to test, hard to modify, and impossible to compose or reuse.

Solution

Firehose provides a declarative framework for event processing pipelines. Four rule types express the same pipeline — source → input condition → action → output condition → destination:

Stage SQLRule ScenarioRule (BDD) StreamRule (Kafka Streams) MapReduceRule
Source From When Source Source
Input condition Where Given Filter Filter
Action Select Then Map Map
Reduce — — — Reduce
Output condition Having GivenOutput FilterOutput FilterOutput
Destination Into To Sink Sink

SQLRule, ScenarioRule, and StreamRule share the same engine — they can be mixed freely in the same pipeline and use identical components for sources, conditions, actions, and destinations. MapReduceRule adds a Reduce stage for stateful accumulation across events.

rule := &fh.SQLRule[OrderEvent, Email]{
    ID:     "order_confirmation",
    Select: CreateEmail{},
    Into:   emailService,
    From:   orderSource,
    Where:  condition.Cond[OrderEvent](`amount > 100`),
    Having: condition.Cond[Email](`to != ""`),
}

Extend functionality with Middlewares that wrap callbacks, actions, and destinations.

Core Concepts

Events

Events are any Go type. No interface requirements. Optionally implement boolexpr.Symbols to expose attributes for condition evaluation:

type OrderPlaced struct {
    OrderID  string
    Amount   float64
    UserTier string
}

func (o OrderPlaced) Get(key string) (any, error) {
    switch key {
    case "amount":
        return o.Amount, nil
    case "tier":
        return o.UserTier, nil
    default:
        return nil, fmt.Errorf("unknown symbol: %s", key)
    }
}

Type-Safe Pipelines

Rules enforce type safety between pipeline stages. Input event type flows from source through transformation to destination:

// HTTP request events → Order events
SQLRule[HTTPRequest, OrderPlaced]

// Order events → Email notifications
SQLRule[OrderPlaced, EmailSent]

// Timer events → Timer events (identity transformation)
SQLRule[TimerTick, TimerTick]

The compiler ensures transformations match: Action[I, O] must accept the source's event type and produce the destination's event type.

Event Source Fanout

Register multiple rules with the same source instance. The framework detects this and starts the source only once, fanning events out to all rules that share it:

kafkaSource := &KafkaConsumer{Topic: "orders"}

// Both rules share kafkaSource - it starts once, events fan out
head, _ = Add(ctx, head, &SQLRule[OrderEvent, Email]{From: kafkaSource, ...})
head, _ = Add(ctx, head, &SQLRule[OrderEvent, Metrics]{From: kafkaSource, ...})

Different source instances (even of the same type) start independently.

Event Processing Middleware

Middlewares intercept and wrap three points in the pipeline: callbacks (event reception), actions (transformation), and destinations (output). Apply cross-cutting concerns like logging, metrics, retry logic, or rate limiting:

type LoggingMiddleware[I, O any] struct{}

type loggingAction[I, O any] struct {
    ruleID string
    next   Action[I, O]
}

func (a loggingAction[I, O]) Process(ctx context.Context, event I, syms boolexpr.Symbols) (O, error) {
    log.Printf("Processing event in rule %s", a.ruleID)
    out, err := a.next.Process(ctx, event, syms)
    log.Printf("Rule %s completed (err=%v)", a.ruleID, err)

    return out, err
}

func (m LoggingMiddleware[I, O]) WrapAction(
    _ context.Context,
    rule Rule,
    action Action[I, O],
) (Action[I, O], error) {
    return loggingAction[I, O]{ruleID: rule.GetID(), next: action}, nil
}

Middlewares compose in registration order (first middleware wraps later ones).

Error Handling

Rules never panic — they return typed errors that you classify with errors.As:

  • ConditionError — failure while evaluating Where/Given/Filter
  • ActionError — failure inside Action.Process
  • DestinationError — failure inside Destination.Send
  • ReduceError — failure inside Reducer.Reduce (MapReduceRule)

Conditions that evaluate to false are normal control flow, signalled by ErrInputNoMatch and ErrOutputNoMatch.

Each error is wrapped in a RuleError carrying the rule ID:

var ruleErr fh.RuleError
if errors.As(err, &ruleErr) {
    fmt.Printf("rule %s failed: %v", ruleErr.Rule, ruleErr.Err)
}

Features

  • ✅ Type-safe event pipelines - Generic types ensure compile-time correctness
  • ✅ Any Go type as event - No interface requirements, works with existing types
  • ✅ Declarative conditions - Boolean expressions via boolexpr library
  • ✅ Four naming conventions - SQL, BDD, Kafka Streams, and MapReduce
  • ✅ Unified middleware - Single interface for callbacks, transformations, destinations
  • ✅ Source fanout optimization - Shared sources start once, distribute to all rules
  • ✅ Context propagation - Full context.Context support for cancellation and values
  • ✅ Typed error classification - Structured error handling via errors.As
  • ✅ Struct validation - Declarative validation with go-playground/validator

Built-in Components

Firehose ships with reusable building blocks under subpackages.

Conditions (condition)

  • condition.Func[I](func(ctx, event, syms) (bool, error) { ... }) — function adapter for conditions
  • condition.Cond[I]("...") — expression-based filtering using boolexpr
  • &condition.RateLimit[I]{Limit: ..., Burst: ...} — throttle event processing
  • &condition.Once[I]{Duration: ..., Cache: ...} — deduplicate events by EventID
  • condition.Conditions[I]{...} — evaluate multiple conditions in sequence

Actions (actions)

  • actions.Func[I, O](func(ctx, event, syms) (O, error) { ... }) — function adapter for actions
  • &actions.Cache[I, O]{Action: ..., Cache: ..., TTL: ...} — memoize action output per event
  • actions.Chain[I, M, O]{...} — compose two actions (I -> M -> O)
  • actions.Chain3[I, A, B, O]{...} — compose three actions
  • actions.Chain4[I, A, B, C, O]{...} — compose four actions
  • actions.Chain5[I, A, B, C, D, O]{...} — compose five actions
  • &actions.RoundRobin[I, O]{Actions: ...} — dispatch events across actions in sequence
  • &actions.Random[I, O]{Actions: ...} — dispatch events to a random action

Destinations (destinations)

  • destinations.Func[T](func(ctx, event) error { ... }) — function adapter for destinations
  • &destinations.Accumulator[T]{} — accumulate output events in memory (Items() []T)
  • destinations.Fanout[T]{Destinations: ...} — send to all destinations
  • &destinations.RoundRobin[T]{Destinations: ...} — send in round-robin order
  • &destinations.Random[T]{Destinations: ...} — send to a random destination
  • destinations.FromChan[T]{Into: ...} — consume chan T, forward each item to Into
  • destinations.FromSlice[T]{Into: ...} — consume []T, forward each item to Into
  • destinations.ToChan[T]{Into: ...} — wrap T as one-item chan T, forward to Into
  • destinations.ToSlice[T]{Into: ...} — wrap T as one-item []T, forward to Into

Sources (sources)

  • sources.Func[T](func(ctx, cb) (<-chan struct{}, error) { ... }) — function adapter for sources
  • &sources.Manual[T]{} — manual source; call Emit(ctx, event) to trigger events

Cache Storage (cache)

  • cache.NewMemory[O](defaultTTL, cleanupInterval) — in-memory cache backend

Middlewares (middlewares)

  • &middlewares.Panic[I, O]{} — panic recovery for callback/action/destination
  • &middlewares.Slog[I, O]{} — structured event/logging via log/slog
  • &middlewares.Parallel[I, O]{Runner: runner.Basic{}} — run same-source rules in parallel

Environment-Specific Rules

Rules can be enabled only in selected environments:

rule := &fh.SQLRule[Event, Out]{
    ID:           "billing/charge",
    Environments: []string{"production", "staging"},
    Select:       action,
    Into:         destination,
    From:         source,
}

Add includes such a rule only when ENV matches one of Environments. If Environments is empty, the rule is always active.

Error Classification

Each rule execution returns an error to the source's error handler:

type ErrorHandler func(err error)

Common rule errors:

  • ErrInputNoMatch — input condition evaluated to false (normal control flow)
  • ErrOutputNoMatch — output condition evaluated to false (normal control flow)
  • ConditionError — failure while evaluating Where or Having
  • ActionError — failure inside Action.Process
  • DestinationError — failure inside Destination.Send
  • ReduceError — failure inside Reducer.Reduce

Use errors.Is / errors.As to classify errors in monitoring code.

Building Event Sources, Transformations, and Destinations

The framework defines three core interfaces you implement for custom event processing.

Event Sources

Sources produce events and send them to a callback function:

type Source[T any] interface {
    Start(ctx context.Context, cb Callback[T]) (done <-chan struct{}, err error)
}

Example - HTTP event source:

type HTTPSource struct {
    Addr string
}

func (s HTTPSource) Start(ctx context.Context, cb fh.Callback[HTTPRequest]) (<-chan struct{}, error) {
    server := &http.Server{Addr: s.Addr}

    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        event := HTTPRequest{Method: r.Method, Path: r.URL.Path}

        cb(r.Context(), event, func(err error) {
            if err != nil {
                log.Printf("rule failed: %v", err)
            }
        })
    })

    go server.ListenAndServe()
    return ctx.Done(), nil
}

Event Transformations

Actions transform input events to output events:

type Action[I, O any] interface {
    Process(ctx context.Context, event I, syms boolexpr.Symbols) (O, error)
}

Example - Extract user data:

type ExtractUser struct{}

func (a ExtractUser) Process(
    ctx context.Context,
    req HTTPRequest,
    syms boolexpr.Symbols,
) (User, error) {
    userID := req.Headers.Get("X-User-ID")
    return fetchUser(userID), nil
}

Event Destinations

Destinations consume events and produce side effects:

type Destination[T any] interface {
    Send(ctx context.Context, event T) error
}

Example - Database writer:

type DBWriter struct {
    DB *sql.DB
}

func (d DBWriter) Send(ctx context.Context, user User) error {
    _, err := d.DB.ExecContext(ctx, "INSERT INTO users ...", user.ID, user.Name)
    return err
}

Quick Start

Process timer events during business hours:

package main

import (
    "context"
    "errors"
    "fmt"
    "log"
    "os"
    "os/signal"
    "time"

    "github.com/emad-elsaid/boolexpr"
    fh "github.com/emad-elsaid/firehose"
    "github.com/emad-elsaid/firehose/condition"
)

// 1. Define your event type
type Tick struct {
    Time time.Time
}

// 2. Make it conditionally evaluable (optional)
func (t Tick) Get(key string) (any, error) {
    if key == "hour" {
        return t.Time.Hour(), nil
    }
    return nil, fmt.Errorf("unknown symbol: %s", key)
}

// 3. Implement an event source
type Timer struct {
    Interval time.Duration
}

func (t Timer) Start(ctx context.Context, cb fh.Callback[Tick]) (<-chan struct{}, error) {
    go func() {
        ticker := time.NewTicker(t.Interval)
        defer ticker.Stop()
        for {
            select {
            case <-ctx.Done():
                return
            case now := <-ticker.C:
                cb(ctx, Tick{Time: now}, func(err error) {
                    if err != nil {
                        log.Printf("rule failed: %v", err)
                    }
                })
            }
        }
    }()
    return ctx.Done(), nil
}

// 4. Implement a transformation
type FormatTime struct{}

func (FormatTime) Process(ctx context.Context, t Tick, _ boolexpr.Symbols) (string, error) {
    return t.Time.Format("15:04:05"), nil
}

// 5. Implement a destination
type Printer struct{}

func (Printer) Send(ctx context.Context, msg string) error {
    println(msg)
    return nil
}

func main() {
    ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt)
    defer stop()

    // 6. Define a rule with a condition
    rule := &fh.SQLRule[Tick, string]{
        ID:     "print_business_hours",
        Select: FormatTime{},
        Into:   Printer{},
        Where:  condition.Cond[Tick]("hour >= 9 and hour < 17"),
        From:   Timer{Interval: 1 * time.Second},
    }

    // 7. Register and start
    head, _ := fh.Add(ctx, nil, rule)

    errHandler := func(err error) {
        if err != nil && !errors.Is(err, context.Canceled) {
            log.Printf("engine error: %v", err)
        }
    }

    doneChannels := fh.Start(ctx, head, errHandler)
    for _, ch := range doneChannels {
        <-ch
    }
}

Conditional Event Processing

Use condition.Cond to filter events based on their attributes. Events must implement boolexpr.Symbols interface:

type OrderEvent struct {
    Amount   float64
    Country  string
    Premium  bool
}

func (o OrderEvent) Get(key string) (any, error) {
    switch key {
    case "amount":
        return o.Amount, nil
    case "country":
        return o.Country, nil
    case "premium":
        return o.Premium, nil
    default:
        return nil, fmt.Errorf("unknown symbol: %s", key)
    }
}

// Only process high-value orders
Where: condition.Cond[OrderEvent]("amount > 1000")

// Geographic filtering
Where: condition.Cond[OrderEvent](`country = "US" or country = "CA"`)

// Complex conditions
Where: condition.Cond[OrderEvent]("premium = true and amount > 500")

Operators: =, ==, !=, <, <=, >, >=, contains, excludes, starts_with, ends_with

Logic: and, or, (...)

Types: Numbers, strings, booleans, slices

See boolexpr documentation for complete syntax.

API Reference

Core Types

// SQLRule defines a complete event processing pipeline
type SQLRule[I, O any] struct {
    ID           string             // Unique identifier
    Environments []string           // Active only when ENV matches; empty = all environments
    Select       Action[I, O]       // Event transformation
    Into         Destination[O]     // Output handler
    From         Source[I]          // Event source
    Where        Condition[I]       // Optional input condition
    Having       Condition[O]       // Optional output condition
    Middlewares  []Middleware[I, O] // Pipeline interceptors
}

// ScenarioRule is the BDD (Given-When-Then) variant
type ScenarioRule[I, O any] struct {
    ID           string
    Environments []string
    When         Source[I]        // Event source
    Given        Condition[I]     // Optional input condition
    Then         Action[I, O]     // Event transformation
    GivenOutput  Condition[O]     // Optional output condition
    To           Destination[O]   // Output handler
    Middlewares  []Middleware[I, O]
}

// StreamRule is the Kafka Streams variant
type StreamRule[I, O any] struct {
    ID           string
    Environments []string
    Source       Source[I]
    Filter       Condition[I]
    Map          Action[I, O]
    FilterOutput Condition[O]
    Sink         Destination[O]
    Middlewares  []Middleware[I, O]
}

// MapReduceRule adds a stateful Reduce stage (I -> M -> Out)
type MapReduceRule[I, M, Out any] struct {
    ID           string
    Environments []string
    Source       Source[I]
    Filter       Condition[I]
    Map          Action[I, M]
    Reduce       Reducer[M, Out]
    FilterOutput Condition[Out]
    Sink         Destination[Out]
    Middlewares  []Middleware[I, M]
}

// Source produces events
type Source[T any] interface {
    Start(ctx context.Context, cb Callback[T]) (done <-chan struct{}, err error)
}

// Callback receives source events and an error sink function
type Callback[I any] func(context.Context, I, ErrorHandler)

// ErrorHandler receives errors from rule execution
type ErrorHandler func(err error)

// Action transforms events
type Action[I, O any] interface {
    Process(ctx context.Context, event I, syms boolexpr.Symbols) (O, error)
}

// Destination consumes events
type Destination[T any] interface {
    Send(ctx context.Context, event T) error
}

// Condition filters events based on conditions
type Condition[I any] interface {
    Evaluate(ctx context.Context, event I, syms boolexpr.Symbols) (bool, error)
}

// Reducer combines an intermediary value with an accumulator
type Reducer[M, Out any] interface {
    Reduce(ctx context.Context, value M, accumulator Out) (Out, error)
}

// Middleware intercepts pipeline components
type Middleware[I, O any] interface {
    WrapCallback(ctx context.Context, rule Rule, callback Callback[I]) (Callback[I], error)
    WrapAction(ctx context.Context, rule Rule, action Action[I, O]) (Action[I, O], error)
    WrapDestination(ctx context.Context, rule Rule, destination Destination[O]) (Destination[O], error)
}

// RuleError wraps an error with the rule ID that produced it
type RuleError struct {
    Rule string
    Err  error
}

Core Functions

// Add registers one or more rules and returns the updated head
func Add(ctx context.Context, head Rule, rules ...Rule) (Rule, error)

// Start activates all registered event sources and returns done channels
func Start(ctx context.Context, head Rule, errFunc ErrorHandler) []<-chan struct{}

Event Symbol Interface

Events optionally implement this interface for conditional processing:

type Symbols interface {
    Get(key string) (any, error)
}

For convenience, you can embed boolexpr.SymbolsMap which implements this interface:

type MyEvent struct {
    boolexpr.SymbolsMap
}

event := MyEvent{
    SymbolsMap: boolexpr.SymbolsMap{
        "count": 42,
        "name":  "example",
    },
}

Design Principles

  • Event-first architecture - Everything revolves around event types and their flow
  • Minimal core concepts - Five interfaces: Source, Condition, Action, Destination, Middleware
  • Complete type safety - Generics ensure correctness from source to destination
  • Separation of concerns - Components define logic, rules define composition
  • Declarative over imperative - Describe event flows, not execution details
  • Reusability by default - Share sources, transformations, and destinations across rules
  • Convention flexibility - SQL, BDD, Kafka Streams, and MapReduce naming in one framework
  • Production-ready validation - Struct validation and extensive linting (50+ rules)

Use Cases

Event-Driven Microservices

  • HTTP request routing and handling
  • gRPC stream processing
  • WebSocket event distribution

Stream Processing

  • Message queue consumers (Kafka, RabbitMQ, NATS)
  • Real-time chat processing
  • Log aggregation and filtering

System Monitoring

  • Process lifecycle tracking
  • File system watching
  • Performance metric collection

Business Process Automation

  • Workflow orchestration
  • Rule-based decision engines
  • Event-driven ETL pipelines

Interactive Systems

  • Game input handling
  • UI event processing
  • Hardware device integration

License

See LICENSE file.