about conbine_AFF shift

[wangtianqi1993]

Hi, in conbime_AFF.py shift value is 21, this is beceuse "target = Ref(vwap,-21)/Ref(vwap,-1)-1"?
If change the target, the value of shift should also be modified accordingly. Hope to get your answer, thank you.

[DulyHao]

If you use 'close' you could use Ref(close,-20)/Ref(vwap,-1) -1.
Because, you can assume you trade on close price after you acquire the whole day data. This is possible because there is a Collective bidding after the market close.
If open and vwap, you could not use that, because, this will assume you use today's data and trade on the second day's open or vwap price.

[wangtianqi1993]

Thank you, I understand your point. I would like to further inquire about line 160 in the combine_AFF.py file, where shift = 21 is set. My understanding is that this is to prevent today's y from leaking into the model during the training process by using data from the previous 21 days. Would setting the shift to a smaller value (such as 2) be better, as it would use more recent data for training, making the model more timely? @DulyHao

[DulyHao]

When the future return is calculated used the data 21 days later, shift=21 is the way to prevent information leakage or future function. It is actually prevent today's x is calculated with tomorow's data.
You could not assign a shift value smaller than 21

[DulyHao]

In another way, in the combine_AFF.py, the last 21 days data does not have an exact y(there is not enough future data), so we have to get the first (n_leng-21) data to evaluate our model's performance.

[wangtianqi1993]

In another way, in the combine_AFF.py, the last 21 days data does not have an exact y(there is not enough future data), so we have to get the first (n_leng-21) data to evaluate our model's performance.

However, based on the logic in the following code from combine_AFF.py, even if shift is set to 0, the data in x and y does not include y_true or to_pred, so there is no data leakage issue. I'm not sure if this understanding is correct. As for what you mentioned about "the last 21 days data does not have an exact y," moving the end_time of the data forward by a certain period can also avoid this issue.

prepare the linear regression data

x = fct_tensor[begin:cur-shift,:,good_idx]
y = tgt_tensor[begin:cur-shift,]#.flatten()
to_pred = fct_tensor[cur,:,good_idx]
y_true = tgt_tensor[cur,]
@DulyHao

[DulyHao]

target = Ref(vwap,-21)/Ref(vwap,-1)-1
tgt_tensor = exprs2tensor([target],data_all,normalize=False)

when running target.evaluate(data), it will automatically shift the future data to the current.
For example,
vwap in data: [t0,t1,t2,t3,...,t100]
target.evaluate(data) will be:
[t21/t1-1, t22/t2-1,...t121/t101-1]
Here t121 does not exist, so the last n_shift items in tgt_tensor will be Nan:
tgt_tensor: [t21/t1-1, t22/t2-1, ...,Nan, Nan]

when you get
y = tgt_tensor[begin:cur-shift]

The final n_shift Nans will be droped, and y will be:
[t21/t1-1,...,t100/t80-1]

[wangtianqi1993]

ok, understand , thanks!