Indicators Optimization #1172

xandortelvanyx · 2024-09-21T21:38:55Z

In the past 15 days, I have watched Mr. Chad Thackray's playlist on YouTube twice and read the Backtesting.py API Reference Documentation. I primarily work with dictionaries because I find NumPy and pandas a bit complicated, so my experience is working on dictionaries.

So far, I have solved many problems and successfully integrated my strategy into Backtesting.py. However, I am currently facing a final issue that I have tried to resolve in every possible way, but I still cannot solve it.

Functions

prepare()

Initializes the data for Backtesting.py.

ohlc()

Retrieves chart data.

indicators()

Obtains analysis data.

TestAnalytics.start()

Initializes the main analytics core.

TestAnalytics.run()

Retrieves live data or data from a CSV file.
Applies strategies and indicators.
Saves the processed data in data.csv and returns it.

`data.csv` Preview

time	low	high	open	close	volume	spread	support_and_resistance	market_structure_shift	break_of_structure	sma	signal_buy	signal_sell	signal_neutral	action	stop_loss	limit_price	take_profit
1698105600	1.06609	1.06734	1.06686	1.06721	4693	1				1.06436
1698134400	1.06439	1.06947	1.06789	1.06501	17555	0	resistance	HH	up	1.0667	2	0	1	buy	1.04624	1.05563	1.06501
1698148800	1.06122	1.0651	1.06497	1.06158	14720	0				1.06572
1698163200	1.05829	1.06243	1.06157	1.05866	24104	0				1.06407
1698177600	1.05844	1.05941	1.05867	1.05886	7136	0				1.0624

`optimaze.py` Code

from backtesting import Backtest, Strategy
from app.analyst.testanalytics import TestAnalytics
import pandas as pd
import numpy as np

TestAnalytics.start(
    login=5030094614,
    password='DlCg+gH0',
    server='MetaQuotes-Demo'
)

def prepare(data):
    df = pd.DataFrame(data[1:], columns=data[0]) if isinstance(data, list) else data
    df.rename(columns={
        'time': 'Date',
        'low': 'Low',
        'high': 'High',
        'open': 'Open',
        'close': 'Close',
        'volume': 'Volume'
    }, inplace=True)
    df.replace('', np.nan, inplace=True)
    return df

def ohlc(support_and_resistance_candleback=18, sma_candleback=20, rsi_candleback=14):
    raw_data = TestAnalytics.run(
        support_and_resistance_candleback=support_and_resistance_candleback,
        sma_candleback=sma_candleback,
        rsi_candleback=rsi_candleback
    )
    df = prepare(raw_data)
    
    required_columns = ['Date', 'Low', 'High', 'Open', 'Close', 'Volume']
    df = df[required_columns]
    df.replace('', np.nan, inplace=True)
    df['Date'] = pd.to_numeric(df['Date'], errors='coerce')
    df['Date'] = pd.to_datetime(df['Date'], unit='s', errors='coerce')
    df.set_index('Date', inplace=True)
    
    numeric_columns = ['Low', 'High', 'Open', 'Close', 'Volume']
    df[numeric_columns] = df[numeric_columns].apply(pd.to_numeric, errors='coerce')

    df.to_csv('_debug/ohlc.csv', index=False)
    # print(df)
    return df

def indicators(support_and_resistance_candleback=18, sma_candleback=20, rsi_candleback=14):
    raw_data = TestAnalytics.run(
        support_and_resistance_candleback=support_and_resistance_candleback,
        sma_candleback=sma_candleback,
        rsi_candleback=rsi_candleback
    )
    df = prepare(raw_data)

    required_columns = ['action', 'limit_price', 'stop_loss', 'take_profit']
    df = df[required_columns]
    df.replace('', pd.NA, inplace=True)
    numeric_columns = ['limit_price', 'stop_loss', 'take_profit']
    df[numeric_columns] = df[numeric_columns].apply(pd.to_numeric, errors='coerce')
    
    if 'action' in df.columns:
        action_mapping = {'buy': 1, 'sell': -1, 'neutral': 0}
        df['action'] = df['action'].replace(action_mapping)
        for col in required_columns:
            df[col] = df[col].fillna(0)
        df['action'] = df['action'].astype(int)

    df.to_csv('_debug/indicators.csv', index=False)
    # print(df)
    return df


class LimitOrderStrategy(Strategy):
    trade_size = 1000
    support_and_resistance_candleback = 18
    sma_candleback = 20
    rsi_candleback = 14
    counter = 0

    def init(self):
        self.counter = 0
        self.indicators = self.I(
            indicators, 
            self.support_and_resistance_candleback, 
            self.sma_candleback, self.rsi_candleback
            )
        np.set_printoptions(threshold=np.inf)
        indicator_array = np.array(self.indicators.data)
        # print(indicator_array)
        if indicator_array.shape[0] == 4:
            indicator_array = indicator_array.T
        if indicator_array.size > 0 and not np.all(np.isnan(indicator_array)):
            indicator_data = pd.DataFrame(
                indicator_array, 
                columns=['action', 'limit_price', 'stop_loss', 'take_profit']
                )
            indicator_data.to_csv('_debug/indicators.csv', index=False)
            # print(indicator_data)
            self.actions = indicator_data['action'].to_numpy()
            self.limit_prices = indicator_data['limit_price'].to_numpy()
            self.stop_losss = indicator_data['stop_loss'].to_numpy()
            self.take_profits = indicator_data['take_profit'].to_numpy()

            # print(self.actions)

    def next(self):
        action = int(self.actions[self.counter])
        limit_price = self.limit_prices[self.counter]
        stop_loss = self.stop_losss[self.counter]
        take_profit = self.take_profits[self.counter]
        # print(action)
        if action in [1, -1]:
            if action == 1:
                self.buy(size=self.trade_size, limit=limit_price, stop=stop_loss, tp=take_profit)
            elif action == -1:
                self.sell(size=self.trade_size, limit=limit_price, stop=stop_loss, tp=take_profit)
        self.counter += 1

def main():
    df = ohlc()
    bt = Backtest(
        df, 
        LimitOrderStrategy, 
        cash=10000, 
        commission=0.000, 
        trade_on_close=False, 
        exclusive_orders=True
    )
    stats, heatmap = bt.optimize(
        sma_candleback=range(5, 20),  
        maximize='Win Rate [%]',  
        return_heatmap=True
    )

    print(stats)
    print(heatmap)
if __name__ == "__main__":
    main()

Debugging

ohlc() return

                         Low     High     Open    Close  Volume
Date
2023-09-26 00:00:00  1.05805  1.05963  1.05928  1.05842    2786
2023-09-26 04:00:00  1.05781  1.05929  1.05841  1.05926    3077
2023-09-26 08:00:00  1.05700  1.06000  1.05926  1.05959    8667
2023-09-26 12:00:00  1.05895  1.06091  1.05960  1.06058    7775
2023-09-26 16:00:00  1.05674  1.06065  1.06058  1.05692   12413
...                      ...      ...      ...      ...     ...
2024-01-22 04:00:00  1.08979  1.09093  1.09018  1.09015   13536
2024-01-22 08:00:00  1.08871  1.09096  1.09015  1.08970   29556
2024-01-22 12:00:00  1.08797  1.08994  1.08970  1.08917   36801
2024-01-22 16:00:00  1.08816  1.09000  1.08917  1.08961   43915
2024-01-22 20:00:00  1.08801  1.08965  1.08961  1.08820   23934

[499 rows x 5 columns]

indicators() return

     action  limit_price  stop_loss  take_profit
0         0          0.0        0.0          0.0
1         0          0.0        0.0          0.0
2         0          0.0        0.0          0.0
3         0          0.0        0.0          0.0
4         0          0.0        0.0          0.0
..      ...          ...        ...          ...
494       0          0.0        0.0          0.0
495       0          0.0        0.0          0.0
496       0          0.0        0.0          0.0
497       0          0.0        0.0          0.0
498       0          0.0        0.0          0.0

[499 rows x 4 columns]

`indicator_array` in init()

[[ 0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       1.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.      -1.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   1.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       1.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       1.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.      -1.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       1.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.      -1.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.     ]
 [ 0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       1.05563  0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       1.06094  0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   1.06579  0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       1.07624  0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       1.07872  0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       1.08764  0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       1.09382
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       1.09982
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.       0.       0.       0.       0.       0.
   0.       0.       0.     ]
 [...][...]]

`indicator_data` in init()

     action  limit_price  stop_loss  take_profit
0       0.0          0.0        0.0          0.0
1       0.0          0.0        0.0          0.0
2       0.0          0.0        0.0          0.0
3       0.0          0.0        0.0          0.0
4       0.0          0.0        0.0          0.0
..      ...          ...        ...          ...
494     0.0          0.0        0.0          0.0
495     0.0          0.0        0.0          0.0
496     0.0          0.0        0.0          0.0
497     0.0          0.0        0.0          0.0
498     0.0          0.0        0.0          0.0

[499 rows x 4 columns]

`self.actions` in init()

[ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.
  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.
  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.
  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.
  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.
  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.
  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  1.  0.  0.  0.
  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.
  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0. -1.
  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  1.  0.  0.  0.
  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.
  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.
  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.
  0.  0.  0.  0.  0.  0.  0.  1.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.
  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.
  0.  0.  0.  0.  0.  0.  1.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.
  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.
  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0. -1.  0.  0.
  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.
  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.
  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.
  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.
  0.  0.  0.  1.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.
  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.
  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.
  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.
  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0. -1.  0.  0.  0.  0.  0.  0.
  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.]

`action` in next()

Using `[-1]` in next()

All values are 0

Using `self.counter` in next()

The values are correct, but they do not update with each change in sma_candleback during optimization.

Output

Using `[-1]` in next()

Code

action = int(self.actions[[-1]])
limit_price = self.limit_prices[[-1]]
stop_loss = self.stop_losss[[-1]]
take_profit = self.take_profits[[-1]]

Result

Start                     2023-09-26 00:00:00
End                       2024-01-22 20:00:00
Duration                    118 days 20:00:00
Exposure Time [%]                         0.0
Equity Final [$]                      10000.0
Equity Peak [$]                       10000.0
Return [%]                                0.0
Buy & Hold Return [%]                2.813628
Return (Ann.) [%]                         0.0
Volatility (Ann.) [%]                     0.0
Sharpe Ratio                              NaN
Sortino Ratio                             NaN
Calmar Ratio                              NaN
Max. Drawdown [%]                        -0.0
Avg. Drawdown [%]                         NaN
Max. Drawdown Duration                    NaN
Avg. Drawdown Duration                    NaN
# Trades                                    0
Win Rate [%]                              NaN
Best Trade [%]                            NaN
Worst Trade [%]                           NaN
Avg. Trade [%]                            NaN
Max. Trade Duration                       NaN
Avg. Trade Duration                       NaN
Profit Factor                             NaN
Expectancy [%]                            NaN
SQN                                       NaN
_strategy                 LimitOrderStrate...
_equity_curve                             ...
_trades                   Empty DataFrame
...
dtype: object
sma_candleback
1                NaN
2                NaN
3                NaN
4                NaN
5                NaN
6                NaN
7                NaN
8                NaN
9                NaN
10               NaN
11               NaN
12               NaN
13               NaN
14               NaN
15               NaN
16               NaN
17               NaN
18               NaN
19               NaN
Name: Win Rate [%], dtype: float64

Using `self.counter` in next()

Code

action = int(self.actions[[self.counter]])
limit_price = self.limit_prices[[self.counter]]
stop_loss = self.stop_losss[[self.counter]]
take_profit = self.take_profits[[self.counter]]

Result

Start                     2023-09-26 00:00:00
End                       2024-01-22 20:00:00
Duration                    118 days 20:00:00
Exposure Time [%]                   99.398798
Equity Final [$]                     10030.01
Equity Peak [$]                      10053.34
Return [%]                             0.3001
Buy & Hold Return [%]                2.813628
Return (Ann.) [%]                    0.975993
Volatility (Ann.) [%]                0.786113
Sharpe Ratio                         1.241542
Sortino Ratio                        1.970413
Calmar Ratio                         3.542233
Max. Drawdown [%]                    -0.27553
Avg. Drawdown [%]                   -0.069088
Max. Drawdown Duration       25 days 12:00:00
Avg. Drawdown Duration        5 days 18:00:00
# Trades                                    1
Win Rate [%]                            100.0
Best Trade [%]                       2.832201
Worst Trade [%]                      2.832201
Avg. Trade [%]                       2.832201
Max. Trade Duration         118 days 08:00:00
Avg. Trade Duration         118 days 08:00:00
Profit Factor                             NaN
Expectancy [%]                       2.832201
SQN                                       NaN
_strategy                 LimitOrderStrate...
_equity_curve                             ...
_trades                      Size  EntryBa...
dtype: object
sma_candleback
1                 100.0
2                 100.0
3                 100.0
4                 100.0
5                 100.0
6                 100.0
7                 100.0
8                 100.0
9                 100.0
10                100.0
11                100.0
12                100.0
13                100.0
14                100.0
15                100.0
16                100.0
17                100.0
18                100.0
19                100.0
Name: Win Rate [%], dtype: float64

Other attempts

I tried assigning the data to sefla.data in init(), but it doesn't work because the library takes the data only at the first initialization, not during every optimization loop.
I'm trying to use self.counter but I think it's not working.

Goal

I want to optimize my strategies and indicators.

The text was updated successfully, but these errors were encountered:

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Indicators Optimization #1172

Indicators Optimization #1172

xandortelvanyx commented Sep 21, 2024

Indicators Optimization #1172

Indicators Optimization #1172

Comments

xandortelvanyx commented Sep 21, 2024

Functions

data.csv Preview

optimaze.py Code

Debugging

ohlc() return

indicators() return

indicator_array in init()

indicator_data in init()

self.actions in init()

action in next()

Using [-1] in next()

Using self.counter in next()

Output

Using [-1] in next()

Code

Result

Using self.counter in next()

Code

Result

Other attempts

Goal

`data.csv` Preview

`optimaze.py` Code

`indicator_array` in init()

`indicator_data` in init()

`self.actions` in init()

`action` in next()

Using `[-1]` in next()

Using `self.counter` in next()

Using `[-1]` in next()

Using `self.counter` in next()