Issue running in production

[amustapha]

Looks like the function requires a start_date, however when looking at the code, couldn't quite figure out why we'd want a start date on an api request to get candles... This is especially tricky since we'd ideally want to get the last n(window_size) candles.

        ^^^^^^^^^
  File "/Users/bots/investing-algorithm-framework/venv/lib/python3.11/site-packages/schedule/__init__.py", line 691, in run
    ret = self.job_func()
          ^^^^^^^^^^^^^^^
  File "/Users/bots/investing-algorithm-framework/venv/lib/python3.11/site-packages/investing_algorithm_framework/services/strategy_orchestrator_service.py", line 76, in run_strategy
    .get_data(identifier=data_id.get_identifier())
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bots/investing-algorithm-framework/venv/lib/python3.11/site-packages/investing_algorithm_framework/services/market_data_source_service/market_data_source_service.py", line 90, in get_data
    return market_data_source.get_data(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bots/investing-algorithm-framework/venv/lib/python3.11/site-packages/investing_algorithm_framework/infrastructure/models/market_data_sources/ccxt.py", line 481, in get_data
    raise OperationalException(
investing_algorithm_framework.domain.exceptions.OperationalException: start_date should be set for CCXTOHLCVMarketDataSource

[MDUYN]

Hi, could you show me how you create your CCXTOHLCVMarketDataSource class?

[amustapha]

I took the SMA cross over strategy from https://github.com/coding-kitties/investing-algorithm-framework/tree/main/examples/crossover_moving_average_trading_bot and made a few adaptations to it.

here's what my data sources look like

symbols = ['NEAR/FDUSD', 'BTC/FDUSD']
timeframe = '15m'
ohlcvs = [
    CCXTOHLCVMarketDataSource(
        identifier=f"{pair}-ohlcv",
        market="BINANCE",
        symbol=pair,
        time_frame=timeframe,
        window_size=200,
    )
    for pair in symbols
]

tickers = [
    CCXTTickerMarketDataSource(
        identifier=f"{pair}-ticker",
        market="BINANCE",
        symbol=pair,
        backtest_time_frame=timeframe,
    )
    for pair in symbols
]

sources = ohlcvs + tickers

And here's what my code looks like in production.py

app.add_algorithm(algorithm)
app.add_portfolio_configuration(
    PortfolioConfiguration(
        market="BINANCE",
        trading_symbol=fiat,
        initial_balance=500,
    )
)
for source in sources:
    app.add_market_data_source(source)


API_KEY = os.getenv("API_KEY")
API_SECRET = os.getenv("API_SECRET")

# Configure your market credentials here
binance_market_credentials = MarketCredential(
    api_key=API_KEY, secret_key=API_SECRET, market="BINANCE"
)
app.add_market_credential(binance_market_credentials)

if __name__ == "__main__":
    app.run()

[MDUYN]

I solved the issue, you are right. Ideally you want to create a CCXTOHLCVMarketDataSource with just a window size. The expected behaviour is then that on every run it will always take current datetime as end date and create a start start based on on the given window size (So if 200 is the window size and the timeframe is 2h, we want 200 2h date points subtracted from the end datetime). This was also the intended behaviour, however, with recent releases the code was unfortunately altered. With the linked PR I added tests to for both start and end datetime params and windows size only param.

[amustapha]

Cool. However ideally since you're using ccxt I think simply providing end time as now and window size, it would return the most recent candles.

…

On Sun, Dec 29, 2024, 12:04 AM Marc van Duyn ***@***.***> wrote: I solved the issue, you are right. Ideally you want to create a CCXTOHLCVMarketDataSource with just a window size. The expected behaviour is then that on every run it will always take current datetime as end date and create a start start based on on the given window size (So if 200 is the window size and the timeframe is 2h, we want 200 2h date points subtracted from the end datetime). This was also the intended behaviour, however, with recent releases the code was unfortunately altered. With the linked PR I added tests to for both start and end datetime params and windows size only param. — Reply to this email directly, view it on GitHub <#282 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADMMBCCBT5OPJG5AKXPFMV32H4VATAVCNFSM6AAAAABUGWZVBSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNRUGU2TEMRTGY> . You are receiving this because you authored the thread.Message ID: ***@***.*** .com>

[MDUYN]

Yes, that is also true, and that is also what is done under the hood as can be seen here:

# Code located at investing_algorithm_framework.infrastructure.services.market_service.ccxt_market_service
        from_time_stamp = exchange.parse8601(
            from_timestamp.strftime(datetime_format)
        )

        if to_timestamp is None:
            to_timestamp = exchange.milliseconds()
        else:
            to_timestamp = exchange.parse8601(
                to_timestamp.strftime(datetime_format)
            )
        data = []

        while from_time_stamp < to_timestamp:
            ohlcv = exchange.fetch_ohlcv(symbol, time_frame, from_time_stamp)

The data download happens through the call: ohlcv = exchange.fetch_ohlcv(symbol, time_frame, from_time_stamp)

The reason for using from_timestamp and to_timestamp is that many exchanges and brokers impose rate limits on data requests. For instance, if you request data starting from 2014 with a 2-hour timeframe, the exchange may not provide all the data in a single response.

To handle this, the data is downloaded in stages. For a small window size (e.g., 200), you likely won't encounter rate limits. However, when requesting larger datasets, rate limits may apply. Hence, start and end timestamps are calculated based on the window size to ensure the data can be retrieved efficiently without exceeding these limits.

[MDUYN]

I will resolve this issue with the next release. You can see the linked pull request. Currently making sure that all the tests still work.