Allow tapping into Broker logs

[Neverlord]

Currently, Broker is mostly a black box to Zeek. Broker can generate valuable log output that helps understanding the system behavior, but Broker writes logs into its own file (or console) if configured. In Zeek, we do have a broker.log which would be the natural place to look at in case of something is off in Broker land, but this log only contains a few high-level events at the moment.

We could follow the example of other message broker APIs for allowing users to tap into the log stream. For example, librdkafka offers an event API that users can tap into by passing a callback object to the library. This is a quite common technique, so it seems natural to do something similar in Broker.

Here's a draft for a logging interface in Broker:

/// Carries information about internal events to the user that are crucial for
/// understanding the system behavior.
class event {
public:
  /// Encodes the severity of the emitted event.
  enum class severity_level {
    /// The reported event is most likely fatal. After a critical event, normal
    /// operation has most likely broken down.
    critical,
    /// The event signals an unrecoverable error in Broker. For example, this
    /// might signal a broken network connection or a data store that is no
    /// longer functional after losing connection to the master node.
    error,
    /// The event signals an unexpected or conspicuous system state that may
    /// still be recoverable.
    warning,
    /// Signals a noteworthy event during normal system operation such as a new
    /// peering connection.
    info,
    /// The event carries information relevant only for troubleshooting and
    /// debugging purposes.
    debug,
  };

  /// Encodes the component that has emitted the event.
  enum class component_type {
    /// The Broker core has emitted the event.
    core,
    /// A Broker data store running in master mode has emitted the event.
    master_store,
    /// A Broker data store running in clone mode has emitted the event.
    clone_store,
    /// The Broker network layer has emitted the event.
    network,
    /// A Broker backend has emitted the event. This could be a SQLite database
    /// or an in-memory store.
    store_backend,
  };

  /// The time when the event has been emitted.
  broker::timestamp timestamp;

  /// Stores the severity for this event.
  severity_level severity;

  /// Stores which Broker component has emitted this event.
  component_type component;

  /// A unique identifier for the event.
  std::string_view identifier;

  /// A human-readable description of the logged event.
  std::string description;
};

/// A smart pointer holding an immutable ::event.
using event_ptr = std::shared_ptr<const event>;

/// An interface for observing internal events in Broker.
class event_observer {
public:
  /// Called by Broker to notify the observer about a new event.
  /// @param what The event that Broker has emitted.
  /// @note This member function is called from multiple threads and thus must
  ///       be thread-safe.
  virtual void observe(event_ptr what) = 0;

  /// Returns true if the observer is interested in events of the given severity
  /// and component type. Returning false will cause Broker to not generate
  /// filtered events.
  virtual bool accepts(event::severity_level severity,
                       event::component_type) const = 0;
};

/// A smart pointer holding an ::event_observer.
using event_observer_ptr = std::shared_ptr<event_observer>;

The implementation itself would be quite straightforward.

Most of the work in Broker would be replacing our current logging calls with a new API that would dispatch the events to the installed observer.

In Zeek, we would implement a subclass of broker::event_observer and would forward the events to Zeek's logging system.