Provide an EventId to correlate raw messages and parsed records#348
Provide an EventId to correlate raw messages and parsed records#348alnkesq wants to merge 1 commit intodrasticactions:developfrom
Conversation
|
You can add it to this PR, but as a heads-up, I'm getting to the point where I'm going to stop work on this repo. I'm nearly at the point with a fully source gen implementation that I'm happy with. You should probably check that out and see if you can use that instead of(and implement whatever changes you want there). I think it can basically meet your needs (Although it needs an XRPC generator to be "complete" with this). |
By "it" do you mean Also I've just noticed that So I cannot add an If you're ok with adding a
|
|
Yeah, go for it. Given that you're the only one actively using those events AFAIK, and that this is a relatively minor breaking change, it's fine. Do what you need to do. e. To be clear, again, though, I'm planning on stopping development on this. Once I merge this, I'll probably do one more bump of the lexicon bindings, release that as stable, and then stop making any changes to this repo and move any future work to Carpanet. This codebase is still a good base for what you need, though, so if you don't want to use my newer library (which I would totally understand, especially in your case), you should probably fork this outright or add it directly to AppViewLite. |
|
Superseded by #349 |
Because firehose records are parsed in parallel on a threadpool, results can arrive out-of-order.
This PR adds a way of linking them back to their original raw messages, so that a consumer can:
Note: this PR only handles JetStreams, not ATProto sockets.
If you are ok with this change, I will implement the same for ATProto sockets (either here or in a separate PR)
For context
The way I currently solve the cursor checkpointing problem in AppViewLite is an ugly hack that involves "pausing" the threadpool until it drains and I can then capture the last definitely processed cursor. However this hack is prone to deadlocks and unnecessary slowdowns.
By contrast, this is an example of how the new
EventIds can be used to reliably keep track of the last definitely processed record: OutOfOrderCompletionTracker.cs