Why data should be ordered by exchange timestamp

[leftys]

Hi, I am implementing support for https://crypto-lake.com/ order book data and wonder why the data have to be ordered by exchange timestamp. If we want to be realistic, we should order the data as we would receive them and that means that exchange timestamps can be mixed due to exchange delays on individual data types. Imho data rows should be only ordered by exchange timestamp among the same data type (eg. trade and trade).

Are there any hftbacktest internals relying on this order? Can i safely disable this part of data validation?

Example:

found 79183 rows that exch_timestamp is ahead of the previous exch_timestamp

[leftys]

For reference, this is my WIP version:
https://github.com/crypto-lake/analysis-sharing/blob/hftbacktest/hftbacktest.ipynb
https://github.com/crypto-lake/analysis-sharing/blob/hftbacktest/lake_to_hftbacktest.py

Some interesting points:

• faster data conversion using numba
• tqdm support during conversion using numba_progress package

[nkaz001]

Data should be ordered by both exchange and local timestamps.

To accurately simulate exchange(order fills), it should be ordered by exchange timestamp since it's the actual event sequence. Also, hftbacktest ensures that the user (local) sees data in the order of local timestamp for a more realistic backtest using a different index. But, as you pointed out, in the case of Binance, timestamps can be mixed across stream types. So, the validation util inserts rows that are only valid for the exchange and only valid for the local by duplicating the mixed timestamped rows.

Please see the following in detail.
https://hftbacktest.readthedocs.io/en/latest/data.html#validation
https://github.com/nkaz001/hftbacktest/blob/master/hftbacktest/data/validation.py#L212

There is a limitation if you use the snapshot by converting it into the diff even through the exchange provides the raw diff. It's a lot better to use the most granular data that the exchange provides.