Binance Orderbook Streaming implementation doesn't take the initial snapshot

[Arshaku]

Binance API documentation gives instructions how to manage local orderbook correctly:
https://github.com/binance-exchange/binance-official-api-docs/blob/master/web-socket-streams.md

How to manage a local order book correctly

1. Open a stream to wss://stream.binance.com:9443/ws/bnbbtc@depth
2. Buffer the events you receive from the stream
3. Get a depth snapshot from **https://www.binance.com/api/v1/depth?symbol=BNBBTC&limit=1000"
4. Drop any event where u is <= lastUpdateId in the snapshot
5. The first processed should have U <= lastUpdateId+1 AND u >= lastUpdateId+1
6. While listening to the stream, each new event's U should be equal to the previous event's u+1
7. The data in each event is the absolute quantity for a price level
8. If the quantity is 0, remove the price level
9. Receiving an event that removes a price level that is not in your local order book can happen and is normal.

As of xchange-stream 4.3.1 it doesn't follow the recommendation by Binance API Docs. As you can see the recommendation is to get the snapshot of full orderbook on 3rd step, and then apply updates on it. The current implementation doesn't take the snapshot. As a result the local orderbook doesn't reflect the reality of Binance orderbook on serverside (though it slowly gets closer over time).

[hlbkin]

In current code I think it is possible to do it in a way:

1. Make a map with sequences inside BinanceStreamingMarketDataService and boolean of whether snapshot was received
2. Subscribe to increments
3. If boolean==false, make request to Snapshot via rest and update order book. Make boolean= true
4. Continue with increments that satisfy requirement on sequence

[daniel-cohen]

I've noticed the same exact problem and I am currently working on implementing the Binance recommended initialization. I.E subscribing to the orderbook update events first, buffering them and then removing anything older than the snapshot from the buffer before processing events in the buffer.

I got it to a somewhat functional state, but not ready to be contributed back.See:
https://github.com/daniel-cohen/XChange-stream/tree/binance-orderbook-init

There are still a few things I'm trying to work out, so if anyone can give me a hand it would be greatly appreciated:

In "BinanceStreamingMarketDataService.java" I was trying to skip one orderbook event before making the call to "InitializeOrderBook", but I couldn't make it work, so till I figure it out , I've added a 3 second sleep inside "InitializeOrderBook":

.skip(1) 
.skipUntil(InitializeOrderBook(currencyPair));

[traviscollins]

FYI, I have discovered that the current implementation of the Binance streaming order book depth updates creates an inconsistent orderbook over time.

https://github.com/traviscollins/xchange-stream-tester

Build the above project, and run the "BinanceStreamTesterMain". Then let it sit for a day or two. You'll see messages saying "Ask of X is lower than highest bid of Y". It's a simple evaluation of whether a single orderbook contains an ask thats lower than it's highest bid (which should never happen, because that would be a negative spread).

[Josi1337]

+1
I had the same issue as traviscollins. Could someone fix that?

[Saoish]

I think the problem traviscollins discovered is on binance end. If you try to listen for depth stream of a low volume trading symbol, you would find out the data is not been pushed every second, and each new event's U is not guaranteed equal to the previous event's u+1.

[mingrui]

interesting thanks for this, I will have to manage my own orderbook

[davidjirovec]

I've modified DepthCacheExample from https://github.com/binance-exchange/binance-java-api to do the same check as @traviscollins does in his tester and I get errors there too:
Ask of 5882.87000000 is lower than highest bid of 5885.38000000

[traviscollins]

We moved to a completely custom solution for Binance using a binary sort update of a simple list and found that Binance’s stream is sane. Our custom implementation works. Something about this Xchange-stream implementation does not.

…

On Jun 24, 2018, at 9:38 AM, David Jirovec ***@***.***> wrote: I've modified DepthCacheExample from https://github.com/binance-exchange/binance-java-api to do the same check as @traviscollins does in his tester and I get errors there too: Ask of 0.01311900 is lower than highest bid of 0.01312300 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[davidjirovec]

That's good to know. But still I am surprised from the result of my experiment, because it does not contain xchange-stream at all. https://github.com/binance-exchange/binance-java-api seems to be reference implementation of java client for binance api.

[cowwoc]

@traviscollins Can you please elaborate on how your code works? How do you handle receiving events out of order? Do you also see that a "new event's U is not guaranteed equal to the previous event's u+1"?

[Flemingjp]

Not positive if this is the best place, but as per @Arshaku instructions, I've had a play with RxJava operators to produce a solution that follow the recommendations. I used the Binance Java API to gather the data, but the logic of manipulating the data should carry across. I've outlined it partially on this gist. The logic of the code gets the snapshot of the orderbook, after the updates begin to queue and buffers the updates such that various validation checks can be made.

[afibur]

Hello!
I'm trying to understand how to build but don't familiar with java...
The idea is to update snapshot with data received from stream.
But how entries from orderbook removed? I mean that trading continues, so orders should be fulfilled or canceled.
Thanks in advance!

[badgerwithagun]

Here's my first stab at the problem properly. A few things I'm uncertain about I've only just started testing, but it implements all the bits, and makes an attempt at what exactly to do when the stream gets out of sync:

badgerwithagun@5fd6345

Thoughts welcome.

[badgerwithagun]

Positive results so far. My trick of re-syncing any time we get an out-of sequence update does indeed seem to work:

INFO  [2018-10-06 13:50:05,672] info.bitrich.xchangestream.binance.BinanceStreamingMarketDataService: Orderbook snapshot for BTC/USDT out of date (last=260118666, U=260118675, u=260118679) {} 
INFO  [2018-10-06 13:50:05,672] info.bitrich.xchangestream.binance.BinanceStreamingMarketDataService: Fetching initial orderbook snapshot for BTC/USDT 
INFO  [2018-10-06 13:50:06,670] info.bitrich.xchangestream.binance.BinanceStreamingMarketDataService: Orderbook snapshot for BTC/USDT out of date (last=260118680, U=260118680, u=260118685) {} 
INFO  [2018-10-06 13:50:06,670] info.bitrich.xchangestream.binance.BinanceStreamingMarketDataService: Fetching initial orderbook snapshot for BTC/USDT 
INFO  [2018-10-06 13:50:07,668] info.bitrich.xchangestream.binance.BinanceStreamingMarketDataService: Orderbook snapshot for BTC/USDT out of date (last=260118688, U=260118686, u=260118698) {} 
INFO  [2018-10-06 13:50:07,668] info.bitrich.xchangestream.binance.BinanceStreamingMarketDataService: Fetching initial orderbook snapshot for BTC/USDT 
INFO  [2018-10-06 13:50:08,668] info.bitrich.xchangestream.binance.BinanceStreamingMarketDataService: Orderbook snapshot for BTC/USDT out of date (last=260118703, U=260118699, u=260118708) {} 
INFO  [2018-10-06 13:50:08,668] info.bitrich.xchangestream.binance.BinanceStreamingMarketDataService: Fetching initial orderbook snapshot for BTC/USDT 
INFO  [2018-10-06 13:50:09,667] info.bitrich.xchangestream.binance.BinanceStreamingMarketDataService: Orderbook snapshot for BTC/USDT out of date (last=260118713, U=260118709, u=260118716) {} 
INFO  [2018-10-06 13:50:09,667] info.bitrich.xchangestream.binance.BinanceStreamingMarketDataService: Fetching initial orderbook snapshot for BTC/USDT 
INFO  [2018-10-06 13:50:10,668] info.bitrich.xchangestream.binance.BinanceStreamingMarketDataService: Orderbook snapshot for BTC/USDT out of date (last=260118718, U=260118717, u=260118724) {} 
INFO  [2018-10-06 13:50:10,668] info.bitrich.xchangestream.binance.BinanceStreamingMarketDataService: Fetching initial orderbook snapshot for BTC/USDT 

[badgerwithagun]

Also survived a socket reconnect:

INFO  [2018-10-06 14:17:48,864] info.bitrich.xchangestream.binance.BinanceStreamingService: Reopening websocket because it was closed by the host
INFO  [2018-10-06 14:17:51,068] info.bitrich.xchangestream.binance.BinanceStreamingService: Connecting to wss://stream.binance.com:9443/stream
INFO  [2018-10-06 14:17:52,111] info.bitrich.xchangestream.service.netty.WebSocketClientHandler: WebSocket Client connected!
WARN  [2018-10-06 14:17:52,111] info.bitrich.xchangestream.binance.BinanceStreamingService: Resubscribing channels
INFO  [2018-10-06 14:17:53,035] info.bitrich.xchangestream.binance.BinanceStreamingMarketDataService: Orderbook snapshot for BTC/USDT out of date (last=260124793, U=260124805, u=260124810)
INFO  [2018-10-06 14:17:53,035] info.bitrich.xchangestream.binance.BinanceStreamingMarketDataService: Fetching initial orderbook snapshot for BTC/USDT 

[badgerwithagun]

Will continue to test this in the coming days.

If all goes well I'll submit a pull request as soon as we have progress on #224. I doubt this will even work on develop without all the unmerged PRs I run against.