The old collect-binancefutures and collector data issue

[yiduohuang]

Hello, I have three questions about the data in version V2.

1, I downloaded data from https://reach.stratosphere.capital/data/usdm/, it gives the following IndexError when running binancefutures.convert().

IndexError Traceback (most recent call last)
Cell In[49], line 4
1 import numpy as np
2 from hftbacktest.data.utils import binancefutures
----> 4 data = binancefutures.convert(
5 'btcusdt_20240901.gz',
6 combined_stream=True
7 )

File ~/anaconda3/lib/python3.11/site-packages/hftbacktest/data/utils/binancefutures.py:121, in convert(input_filename, output_filename, opt, base_latency, combined_stream, buffer_size)
119 exch_timestamp = int(transaction_time) * timestamp_mul
120 for px, qty in data['b']:
--> 121 tmp[row_num] = (
122 DEPTH_EVENT | BUY_EVENT,
123 exch_timestamp,
124 local_timestamp,
125 float(px),
126 float(qty),
127 0,
128 0,
129 0
130 )
131 row_num += 1
132 for px, qty in data['a']:

IndexError: index 100000000 is out of bounds for axis 0 with size 100000000

2, I obtained data using old collect-binancefutures repo, it gives the following ValueError when running binancefutures.convert().

ValueError Traceback (most recent call last)
Cell In[29], line 4
1 import numpy as np
2 from hftbacktest.data.utils import binancefutures
----> 4 data = binancefutures.convert(
5 'btcusdt_20241016.gz',
6 combined_stream=True
7 )

File ~/anaconda3/lib/python3.11/site-packages/hftbacktest/data/utils/binancefutures.py:89, in convert(input_filename, output_filename, opt, base_latency, combined_stream, buffer_size)
87 if not line:
88 break
---> 89 local_timestamp = int(line[:timestamp_slice])
90 message = json.loads(line[timestamp_slice + 1:])
91 if combined_stream:

ValueError: invalid literal for int() with base 10: b'1729075130402739 {"'

3, When I use the new data collector written in Rust from this repository (https://github.com/nkaz001/hftbacktest/tree/master/collector/src), by running the command: cargo run -- binancefutures btcusdt. I receive a warning saying that the depth feed is not supported. However, I am still able to get the depth data, and it works correctly in binancefutures.convert() without any errors.
Thanks

[nkaz001]

1. You need to set the bigger buffer size.
   Please see https://hftbacktest.readthedocs.io/en/py-v2.1.0/reference/hftbacktest.data.utils.binancefutures.html

2. The old collector uses microseconds, but the new one uses nanoseconds. If you want to keep using the old one, you need to modify the code accordingly.

3. What's the exact warning message?