Comments on pull-based introspection protocol

[raulk]

Given that the original desire was for a backpressure-aware protocol (not a protocol that avoids backpressure), I'd like to see a rationale of tradeoffs of each approach.

• Push-based with backpressure signalling (we had talked about a "loss of fidelity" warning message).
• Pull-based with no possibility of backpressure.

IMO, the pull-based model simplifies the state reporting side of things, but it makes real-time events firehoses very cumbersome, as now the server is obliged to buffer events in memory and wait until the client requests them.

Drawing from industry de-facto standards, Prometheus is pull-based, and Jaeger is push based.

[mcollina]

The reason why we opted for a pull-based approach is to avoid buffering on the libp2p side to minimize memory usage and overhead. In a pull-based protocol, the server can maintain a full state/snapshot of the data and send that (or a subset) on demand. In other terms, it does not have to maintain a full queue of elements to send in case the socket is full.

In the workshop, we talked about having a resolution of 1s. I would not classify 1 snapshot/s a firehose of data, so a pull-based approach seemed extremely easier than a signaling protocol.

[raulk]

Yeah, for state pull-based is fine, as the pace is determined by the consumer side. I’m worried about event firehoses, where the pace is determined by the producing side. Were you planning to retain events in memory on the libp2p side?