A type-safe event processing framework for Go. Build composable event pipelines with conditional execution, middleware support, and multiple naming conventions.
Choose the convention that fits your team — SQL, BDD (Given-When-Then), Kafka Streams, or MapReduce — all express the same pipeline.
Applications process events from various sources (HTTP requests, message queues, timers, system events, user input) and react with side effects. Without a structured approach, event handling becomes scattered across the codebase, difficult to test, hard to modify, and impossible to compose or reuse.
Firehose provides a declarative framework for event processing pipelines. Four rule types express the same pipeline — source → input condition → action → output condition → destination:
| Stage | SQLRule |
ScenarioRule (BDD) |
StreamRule (Kafka Streams) |
MapReduceRule |
|---|---|---|---|---|
| Source | From |
When |
Source |
Source |
| Input condition | Where |
Given |
Filter |
Filter |
| Action | Select |
Then |
Map |
Map |
| Reduce | — | — | — | Reduce |
| Output condition | Having |
GivenOutput |
FilterOutput |
FilterOutput |
| Destination | Into |
To |
Sink |
Sink |
SQLRule, ScenarioRule, and StreamRule share the same engine — they can be mixed freely in the same
pipeline and use identical components for sources, conditions, actions, and destinations. MapReduceRule adds
a Reduce stage for stateful accumulation across events.
rule := &fh.SQLRule[OrderEvent, Email]{
ID: "order_confirmation",
Select: CreateEmail{},
Into: emailService,
From: orderSource,
Where: condition.Cond[OrderEvent](`amount > 100`),
Having: condition.Cond[Email](`to != ""`),
}Extend functionality with Middlewares that wrap callbacks, actions, and destinations.
Events are any Go type. No interface requirements. Optionally implement boolexpr.Symbols to expose attributes
for condition evaluation:
type OrderPlaced struct {
OrderID string
Amount float64
UserTier string
}
func (o OrderPlaced) Get(key string) (any, error) {
switch key {
case "amount":
return o.Amount, nil
case "tier":
return o.UserTier, nil
default:
return nil, fmt.Errorf("unknown symbol: %s", key)
}
}Rules enforce type safety between pipeline stages. Input event type flows from source through transformation to destination:
// HTTP request events → Order events
SQLRule[HTTPRequest, OrderPlaced]
// Order events → Email notifications
SQLRule[OrderPlaced, EmailSent]
// Timer events → Timer events (identity transformation)
SQLRule[TimerTick, TimerTick]The compiler ensures transformations match: Action[I, O] must accept the source's event type and produce the
destination's event type.
Register multiple rules with the same source instance. The framework detects this and starts the source only once, fanning events out to all rules that share it:
kafkaSource := &KafkaConsumer{Topic: "orders"}
// Both rules share kafkaSource - it starts once, events fan out
head, _ = Add(ctx, head, &SQLRule[OrderEvent, Email]{From: kafkaSource, ...})
head, _ = Add(ctx, head, &SQLRule[OrderEvent, Metrics]{From: kafkaSource, ...})Different source instances (even of the same type) start independently.
Middlewares intercept and wrap three points in the pipeline: callbacks (event reception), actions (transformation), and destinations (output). Apply cross-cutting concerns like logging, metrics, retry logic, or rate limiting:
type LoggingMiddleware[I, O any] struct{}
type loggingAction[I, O any] struct {
ruleID string
next Action[I, O]
}
func (a loggingAction[I, O]) Process(ctx context.Context, event I, syms boolexpr.Symbols) (O, error) {
log.Printf("Processing event in rule %s", a.ruleID)
out, err := a.next.Process(ctx, event, syms)
log.Printf("Rule %s completed (err=%v)", a.ruleID, err)
return out, err
}
func (m LoggingMiddleware[I, O]) WrapAction(
_ context.Context,
rule Rule,
action Action[I, O],
) (Action[I, O], error) {
return loggingAction[I, O]{ruleID: rule.GetID(), next: action}, nil
}Middlewares compose in registration order (first middleware wraps later ones).
Rules never panic — they return typed errors that you classify with errors.As:
ConditionError— failure while evaluatingWhere/Given/FilterActionError— failure insideAction.ProcessDestinationError— failure insideDestination.SendReduceError— failure insideReducer.Reduce(MapReduceRule)
Conditions that evaluate to false are normal control flow, signalled by ErrInputNoMatch and ErrOutputNoMatch.
Each error is wrapped in a RuleError carrying the rule ID:
var ruleErr fh.RuleError
if errors.As(err, &ruleErr) {
fmt.Printf("rule %s failed: %v", ruleErr.Rule, ruleErr.Err)
}- ✅ Type-safe event pipelines - Generic types ensure compile-time correctness
- ✅ Any Go type as event - No interface requirements, works with existing types
- ✅ Declarative conditions - Boolean expressions via
boolexprlibrary - ✅ Four naming conventions - SQL, BDD, Kafka Streams, and MapReduce
- ✅ Unified middleware - Single interface for callbacks, transformations, destinations
- ✅ Source fanout optimization - Shared sources start once, distribute to all rules
- ✅ Context propagation - Full context.Context support for cancellation and values
- ✅ Typed error classification - Structured error handling via
errors.As - ✅ Struct validation - Declarative validation with
go-playground/validator
Firehose ships with reusable building blocks under subpackages.
condition.Func[I](func(ctx, event, syms) (bool, error) { ... })— function adapter for conditionscondition.Cond[I]("...")— expression-based filtering usingboolexpr&condition.RateLimit[I]{Limit: ..., Burst: ...}— throttle event processing&condition.Once[I]{Duration: ..., Cache: ...}— deduplicate events byEventIDcondition.Conditions[I]{...}— evaluate multiple conditions in sequence
actions.Func[I, O](func(ctx, event, syms) (O, error) { ... })— function adapter for actions&actions.Cache[I, O]{Action: ..., Cache: ..., TTL: ...}— memoize action output per eventactions.Chain[I, M, O]{...}— compose two actions (I -> M -> O)actions.Chain3[I, A, B, O]{...}— compose three actionsactions.Chain4[I, A, B, C, O]{...}— compose four actionsactions.Chain5[I, A, B, C, D, O]{...}— compose five actions&actions.RoundRobin[I, O]{Actions: ...}— dispatch events across actions in sequence&actions.Random[I, O]{Actions: ...}— dispatch events to a random action
destinations.Func[T](func(ctx, event) error { ... })— function adapter for destinations&destinations.Accumulator[T]{}— accumulate output events in memory (Items() []T)destinations.Fanout[T]{Destinations: ...}— send to all destinations&destinations.RoundRobin[T]{Destinations: ...}— send in round-robin order&destinations.Random[T]{Destinations: ...}— send to a random destinationdestinations.FromChan[T]{Into: ...}— consumechan T, forward each item toIntodestinations.FromSlice[T]{Into: ...}— consume[]T, forward each item toIntodestinations.ToChan[T]{Into: ...}— wrapTas one-itemchan T, forward toIntodestinations.ToSlice[T]{Into: ...}— wrapTas one-item[]T, forward toInto
sources.Func[T](func(ctx, cb) (<-chan struct{}, error) { ... })— function adapter for sources&sources.Manual[T]{}— manual source; callEmit(ctx, event)to trigger events
cache.NewMemory[O](defaultTTL, cleanupInterval)— in-memory cache backend
&middlewares.Panic[I, O]{}— panic recovery for callback/action/destination&middlewares.Slog[I, O]{}— structured event/logging vialog/slog&middlewares.Parallel[I, O]{Runner: runner.Basic{}}— run same-source rules in parallel
Rules can be enabled only in selected environments:
rule := &fh.SQLRule[Event, Out]{
ID: "billing/charge",
Environments: []string{"production", "staging"},
Select: action,
Into: destination,
From: source,
}Add includes such a rule only when ENV matches one of Environments.
If Environments is empty, the rule is always active.
Each rule execution returns an error to the source's error handler:
type ErrorHandler func(err error)Common rule errors:
ErrInputNoMatch— input condition evaluated to false (normal control flow)ErrOutputNoMatch— output condition evaluated to false (normal control flow)ConditionError— failure while evaluatingWhereorHavingActionError— failure insideAction.ProcessDestinationError— failure insideDestination.SendReduceError— failure insideReducer.Reduce
Use errors.Is / errors.As to classify errors in monitoring code.
The framework defines three core interfaces you implement for custom event processing.
Sources produce events and send them to a callback function:
type Source[T any] interface {
Start(ctx context.Context, cb Callback[T]) (done <-chan struct{}, err error)
}Example - HTTP event source:
type HTTPSource struct {
Addr string
}
func (s HTTPSource) Start(ctx context.Context, cb fh.Callback[HTTPRequest]) (<-chan struct{}, error) {
server := &http.Server{Addr: s.Addr}
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
event := HTTPRequest{Method: r.Method, Path: r.URL.Path}
cb(r.Context(), event, func(err error) {
if err != nil {
log.Printf("rule failed: %v", err)
}
})
})
go server.ListenAndServe()
return ctx.Done(), nil
}Actions transform input events to output events:
type Action[I, O any] interface {
Process(ctx context.Context, event I, syms boolexpr.Symbols) (O, error)
}Example - Extract user data:
type ExtractUser struct{}
func (a ExtractUser) Process(
ctx context.Context,
req HTTPRequest,
syms boolexpr.Symbols,
) (User, error) {
userID := req.Headers.Get("X-User-ID")
return fetchUser(userID), nil
}Destinations consume events and produce side effects:
type Destination[T any] interface {
Send(ctx context.Context, event T) error
}Example - Database writer:
type DBWriter struct {
DB *sql.DB
}
func (d DBWriter) Send(ctx context.Context, user User) error {
_, err := d.DB.ExecContext(ctx, "INSERT INTO users ...", user.ID, user.Name)
return err
}Process timer events during business hours:
package main
import (
"context"
"errors"
"fmt"
"log"
"os"
"os/signal"
"time"
"github.com/emad-elsaid/boolexpr"
fh "github.com/emad-elsaid/firehose"
"github.com/emad-elsaid/firehose/condition"
)
// 1. Define your event type
type Tick struct {
Time time.Time
}
// 2. Make it conditionally evaluable (optional)
func (t Tick) Get(key string) (any, error) {
if key == "hour" {
return t.Time.Hour(), nil
}
return nil, fmt.Errorf("unknown symbol: %s", key)
}
// 3. Implement an event source
type Timer struct {
Interval time.Duration
}
func (t Timer) Start(ctx context.Context, cb fh.Callback[Tick]) (<-chan struct{}, error) {
go func() {
ticker := time.NewTicker(t.Interval)
defer ticker.Stop()
for {
select {
case <-ctx.Done():
return
case now := <-ticker.C:
cb(ctx, Tick{Time: now}, func(err error) {
if err != nil {
log.Printf("rule failed: %v", err)
}
})
}
}
}()
return ctx.Done(), nil
}
// 4. Implement a transformation
type FormatTime struct{}
func (FormatTime) Process(ctx context.Context, t Tick, _ boolexpr.Symbols) (string, error) {
return t.Time.Format("15:04:05"), nil
}
// 5. Implement a destination
type Printer struct{}
func (Printer) Send(ctx context.Context, msg string) error {
println(msg)
return nil
}
func main() {
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt)
defer stop()
// 6. Define a rule with a condition
rule := &fh.SQLRule[Tick, string]{
ID: "print_business_hours",
Select: FormatTime{},
Into: Printer{},
Where: condition.Cond[Tick]("hour >= 9 and hour < 17"),
From: Timer{Interval: 1 * time.Second},
}
// 7. Register and start
head, _ := fh.Add(ctx, nil, rule)
errHandler := func(err error) {
if err != nil && !errors.Is(err, context.Canceled) {
log.Printf("engine error: %v", err)
}
}
doneChannels := fh.Start(ctx, head, errHandler)
for _, ch := range doneChannels {
<-ch
}
}Use condition.Cond to filter events based on their attributes. Events must implement boolexpr.Symbols interface:
type OrderEvent struct {
Amount float64
Country string
Premium bool
}
func (o OrderEvent) Get(key string) (any, error) {
switch key {
case "amount":
return o.Amount, nil
case "country":
return o.Country, nil
case "premium":
return o.Premium, nil
default:
return nil, fmt.Errorf("unknown symbol: %s", key)
}
}
// Only process high-value orders
Where: condition.Cond[OrderEvent]("amount > 1000")
// Geographic filtering
Where: condition.Cond[OrderEvent](`country = "US" or country = "CA"`)
// Complex conditions
Where: condition.Cond[OrderEvent]("premium = true and amount > 500")Operators: =, ==, !=, <, <=, >, >=, contains, excludes, starts_with, ends_with
Logic: and, or, (...)
Types: Numbers, strings, booleans, slices
See boolexpr documentation for complete syntax.
// SQLRule defines a complete event processing pipeline
type SQLRule[I, O any] struct {
ID string // Unique identifier
Environments []string // Active only when ENV matches; empty = all environments
Select Action[I, O] // Event transformation
Into Destination[O] // Output handler
From Source[I] // Event source
Where Condition[I] // Optional input condition
Having Condition[O] // Optional output condition
Middlewares []Middleware[I, O] // Pipeline interceptors
}
// ScenarioRule is the BDD (Given-When-Then) variant
type ScenarioRule[I, O any] struct {
ID string
Environments []string
When Source[I] // Event source
Given Condition[I] // Optional input condition
Then Action[I, O] // Event transformation
GivenOutput Condition[O] // Optional output condition
To Destination[O] // Output handler
Middlewares []Middleware[I, O]
}
// StreamRule is the Kafka Streams variant
type StreamRule[I, O any] struct {
ID string
Environments []string
Source Source[I]
Filter Condition[I]
Map Action[I, O]
FilterOutput Condition[O]
Sink Destination[O]
Middlewares []Middleware[I, O]
}
// MapReduceRule adds a stateful Reduce stage (I -> M -> Out)
type MapReduceRule[I, M, Out any] struct {
ID string
Environments []string
Source Source[I]
Filter Condition[I]
Map Action[I, M]
Reduce Reducer[M, Out]
FilterOutput Condition[Out]
Sink Destination[Out]
Middlewares []Middleware[I, M]
}
// Source produces events
type Source[T any] interface {
Start(ctx context.Context, cb Callback[T]) (done <-chan struct{}, err error)
}
// Callback receives source events and an error sink function
type Callback[I any] func(context.Context, I, ErrorHandler)
// ErrorHandler receives errors from rule execution
type ErrorHandler func(err error)
// Action transforms events
type Action[I, O any] interface {
Process(ctx context.Context, event I, syms boolexpr.Symbols) (O, error)
}
// Destination consumes events
type Destination[T any] interface {
Send(ctx context.Context, event T) error
}
// Condition filters events based on conditions
type Condition[I any] interface {
Evaluate(ctx context.Context, event I, syms boolexpr.Symbols) (bool, error)
}
// Reducer combines an intermediary value with an accumulator
type Reducer[M, Out any] interface {
Reduce(ctx context.Context, value M, accumulator Out) (Out, error)
}
// Middleware intercepts pipeline components
type Middleware[I, O any] interface {
WrapCallback(ctx context.Context, rule Rule, callback Callback[I]) (Callback[I], error)
WrapAction(ctx context.Context, rule Rule, action Action[I, O]) (Action[I, O], error)
WrapDestination(ctx context.Context, rule Rule, destination Destination[O]) (Destination[O], error)
}
// RuleError wraps an error with the rule ID that produced it
type RuleError struct {
Rule string
Err error
}// Add registers one or more rules and returns the updated head
func Add(ctx context.Context, head Rule, rules ...Rule) (Rule, error)
// Start activates all registered event sources and returns done channels
func Start(ctx context.Context, head Rule, errFunc ErrorHandler) []<-chan struct{}Events optionally implement this interface for conditional processing:
type Symbols interface {
Get(key string) (any, error)
}For convenience, you can embed boolexpr.SymbolsMap which implements this interface:
type MyEvent struct {
boolexpr.SymbolsMap
}
event := MyEvent{
SymbolsMap: boolexpr.SymbolsMap{
"count": 42,
"name": "example",
},
}- Event-first architecture - Everything revolves around event types and their flow
- Minimal core concepts - Five interfaces: Source, Condition, Action, Destination, Middleware
- Complete type safety - Generics ensure correctness from source to destination
- Separation of concerns - Components define logic, rules define composition
- Declarative over imperative - Describe event flows, not execution details
- Reusability by default - Share sources, transformations, and destinations across rules
- Convention flexibility - SQL, BDD, Kafka Streams, and MapReduce naming in one framework
- Production-ready validation - Struct validation and extensive linting (50+ rules)
Event-Driven Microservices
- HTTP request routing and handling
- gRPC stream processing
- WebSocket event distribution
Stream Processing
- Message queue consumers (Kafka, RabbitMQ, NATS)
- Real-time chat processing
- Log aggregation and filtering
System Monitoring
- Process lifecycle tracking
- File system watching
- Performance metric collection
Business Process Automation
- Workflow orchestration
- Rule-based decision engines
- Event-driven ETL pipelines
Interactive Systems
- Game input handling
- UI event processing
- Hardware device integration
See LICENSE file.