Extracting non-optimal balances and tie-breaking criteria

[Dalcon2704]

First of all, thank you for creating selbal and for the support you provide to the community.

I am using the selbal.cv function with dichotomous response (n ≈ 50 samples). To avoid the warning regarding species appearing in less than 20% of samples, I have pre-filtered my data, resulting in a dataframe with 43 species.

This is how I call the function:

AUC_max <- selbal.cv(x = df_X, y = Y,
                       n.iter = 30,
                       n.fold = 5,
                       maxV = 30,
                       opt.cri = "max")

I have a few questions regarding the selbal.cv output and its internal logic.

1. How to extract data for a non-optimal number of variables?

The selbal.cv function identifies an "optimal" number of variables (2 in my case). However, I am interested in obtaining balances that include more variables than the suggested optimum. I cannot get the user_numVar argument to work.

In the accuracy.nvar object, the plot shows the performance for a range of variables broader than the optimal number. Since the algorithm calculates these values for the representation, is there a way to extract the specific balances or the raw data for these other points?

2. How does the method handle "ties" in pairwise balances?

When used the method published to obtain all the pairwise balances manually, I found multiple pairs of species whose balance prediction is equally correct (identical AUC).

In these cases, how does the method select the "best" balance among several candidates with the same performance? Does it follow the original order of the variables in the dataframe, or is there an additional internal criterion?
Thank you in advance!

[IkaStat]

Hi @Dalcon2704!

First of all, we want to thank you for your kind words about selbal and hope it can be helpful in your research.

The two questions you ask are very interesting, although I believe your case is an exceptional one. I will briefly answer your questions and then go into more detail about your case:

QUESTION 1

The main objective of selbal.cv() is to find the optimal balance in terms of:

• Greater accuracy (better prediction)
• Fewer variables or species in your case (greater simplicity)

In selbal, all the parameters that refer to the number of variables (maxV, user_numVar) actually determine the maximum number of variables to include in the balance. Thus, two situations may occur:

• If, before reaching user_numVar, the algorithm finds no improvement by adding new variables to the balance, it stops because it cannot improve the accuracy.
• If it reaches user_numVar, it doesn’t matter if there are variables that, when included in the balance improve the accuracy; since the user has set a limit, the algorithm stops.

Later, when I discuss your specific case, I will go into more detail on this aspect.

---

QUESTION 2

When there are ties in the accuracy of different balances, selbal selects the first one it finds. This is because we use which.max function, and in case of ties, it returns the position of the first element with the highest accuracy. It is possible, but also difficult for two balances two give exactly the same accuracy.

---

YOUR PARTICULAR EXAMPLE

Regarding your specific case, the attached graph shows:

• An accuracy of exactly 1 regardless of the number of variables selected
• A MSE of exactly 0 regardless of the number of variables selected

This leads me to believe that each and every one of your explanatory variables for the analysis is “perfect” or at least 30 of them as the graph shows). In other words, regardless of the subset of species you select, you will obtain a balance with an accuracy of 1. I suppose we are in front of one of the following situations:

• The link between your response variable and some species follows a perfect presence-absence relationship; that is, all samples with a response variable 1 show the presence of the species and the rest have a 0 abundance (or vice versa).
• The link between your response variable and some species follows an increasing-decreasing abundance relationship; that is, the abundances of the group with a 1 as a response variable are greater than those with a response variable 0 (or vice versa).

So, going back to your questions, since an optimal balance is achieved with just two variables, anything else results in a more complex balance (more variables) but as efficient as the simplest ones .

---

I don't know if this answers your questions or at least clarifies some of the doubts you had. In any case, I think if you share the second graph returned byselbal.cv function , perhaps I could clarify your case a bit more.