Add Fold Weights for Variable Resample Weighting

[tjburch]

Closes #990

This PR implements support for variable fold weights in hyperparameter tuning. This is useful in cases where folds may have differing numbers of observations, and you want proportional contribution to hyperparameter selection.

The implementation adds two main functions: add_fold_weights() to attach custom weights to rset objects, and calculate_fold_weights() to automatically compute weights proportional to fold sizes. Weights are stored as .fold_weights attributes and should flow through the existing tuning pipeline.

Core changes are in estimate_tune_results() which now detects weights and uses weighted statistics (weighted mean, weighted standard deviation, effective sample size) when aggregating metrics. Implementation should be backwards compatible and non-breaking.

[topepo]

Hey @tjburch. Thanks for the PR.

We're doing a pretty invasive update the this package that will take some time. We'll look at the PR after things are settled there but it might be another 2-3 weeks.

Is this something time-sensitive for you?

[tjburch]

Nope. Just had some bandwidth staying awake on paternity leave. Review at your leisure, let me know if I can assist otherwise.

[topepo]

Just had some bandwidth staying awake on paternity leave

I estimate that 5% of all my work has been while waiting at the bustop or for some sort of practice to end 😄

[tjburch]

Got around to deploying this into a local project and running into some odd errors. Will circle back.

[github-actions]

Remaining comments which cannot be posted as a review comment to avoid GitHub Rate Limit

air

---

[air] _{reported by reviewdog 🐶}

tune/R/utils.R

Line 393 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/R/utils.R

Line 399 in 27a5ab5

#'

---

[air] _{reported by reviewdog 🐶}

tune/R/utils.R

Line 408 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/R/utils.R

Line 411 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/R/utils.R

Line 414 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/R/utils.R

Line 419 in 27a5ab5

#'

---

[air] _{reported by reviewdog 🐶}

tune/R/utils.R

Line 427 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/R/utils.R

Line 430 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/R/utils.R

Line 436 in 27a5ab5

#'

---

[air] _{reported by reviewdog 🐶}

tune/R/utils.R

Line 440 in 27a5ab5

#'

---

[air] _{reported by reviewdog 🐶}

tune/R/utils.R

Line 466 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/R/utils.R

Line 481 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-checks.R

Line 547 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-checks.R

Line 550 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-checks.R

Line 555 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-checks.R

Line 561 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-checks.R

Line 566 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-checks.R

Line 578 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-checks.R

Line 583 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-checks.R

Line 590 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-checks.R

Line 594 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-checks.R

Lines 596 to 600 in 27a5ab5

	unweighted_results <- fit_resamples(simple_wflow, folds,
	control = control_resamples(save_pred = FALSE))
	weighted_results_equal <- fit_resamples(simple_wflow, weighted_folds_equal,
	control = control_resamples(save_pred = FALSE))

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-checks.R

Line 604 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-checks.R

Lines 606 to 607 in 27a5ab5

	expect_equal(unweighted_metrics$mean, weighted_metrics_equal$mean, tolerance = 1e-10)

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-checks.R

Line 609 in 27a5ab5

unequal_weights <- c(0.1, 0.3, 0.6) # Higher weight on last fold

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-checks.R

Lines 611 to 613 in 27a5ab5

	weighted_results_unequal <- fit_resamples(simple_wflow, weighted_folds_unequal,
	control = control_resamples(save_pred = FALSE))

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-checks.R

Line 615 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-checks.R

Lines 617 to 618 in 27a5ab5

	expect_false(all(abs(unweighted_metrics$mean - weighted_metrics_unequal$mean) < 1e-10))

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-checks.R

Line 621 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-checks.R

Line 626 in 27a5ab5

expect_equal(sum(calculated_weights), 1) # Should sum to 1 now

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-checks.R

Line 631 in 27a5ab5

skip_if_not_installed("parsnip")

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-checks.R

Line 635 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-checks.R

Line 640 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-checks.R

Line 644 in 27a5ab5

tune_mod <- parsnip::linear_reg(penalty = tune()) %>%

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-checks.R

Line 649 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-checks.R

Line 652 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-checks.R

Line 656 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-checks.R

Lines 658 to 661 in 27a5ab5

	weighted_tune_results <- tune_grid(tune_wflow, weighted_folds,
	grid = simple_grid,
	control = control_grid(save_pred = FALSE))

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-checks.R

Lines 663 to 664 in 27a5ab5

	expect_s3_class(weighted_tune_results, "tune_results")

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-checks.R

Line 669 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-checks.R

Lines 671 to 673 in 27a5ab5

	unweighted_tune_results <- tune_grid(tune_wflow, folds,
	grid = simple_grid,
	control = control_grid(save_pred = FALSE))

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-checks.R

Line 675 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-checks.R

Line 677 in 27a5ab5

expect_false(all(abs(weighted_metrics$mean - unweighted_metrics$mean) < 1e-10))

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Lines 2 to 3 in 27a5ab5

	if (rlang::is_installed(c("rsample", "parsnip", "yardstick", "workflows", "recipes", "kknn"))) {

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 15 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 26 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 31 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 36 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 41 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 51 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 54 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 62 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 72 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 74 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 85 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 95 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 97 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 108 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 119 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 122 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 131 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 135 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 139 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 143 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 148 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 150 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 161 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 171 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 182 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 186 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 191 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 195 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 204 in 27a5ab5

c(1/6, 1/3, 1/2) # normalized to sum to 1

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 206 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 209 in 27a5ab5

c(0.2, 0.3, 0.5) # already normalized to sum to 1

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 216 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 219 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 225 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 230 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 234 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 237 in 27a5ab5

expect_true(is.na(tune:::.weighted_sd(c(1), c(1)))) # single value

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 243 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 246 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 248 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 259 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 267 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 269 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 280 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Lines 283 to 284 in 27a5ab5

	expect_true(nrow(individual_metrics) >= 3) # At least one metric per fold

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 292 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 297 in 27a5ab5

resamples = folds, # No weights

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 303 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 314 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 317 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 328 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 337 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 341 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 351 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 355 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 364 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 366 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 377 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 380 in 27a5ab5

expected_weights <- weights / sum(weights) # normalized

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 389 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Lines 398 to 400 in 27a5ab5

	splits1 <- rsample::make_splits(x = mtcars[1:20,], assessment = mtcars[21:32,])
	splits2 <- rsample::make_splits(x = mtcars[1:15,], assessment = mtcars[16:32,])

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 405 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 408 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 411 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 417 in 27a5ab5

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Lines 422 to 423 in 27a5ab5

	}

[github-actions]

Remaining comments which cannot be posted as a review comment to avoid GitHub Rate Limit

air

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-checks.R

Line 677 in 7fa521d

expect_false(all(abs(weighted_metrics$mean - unweighted_metrics$mean) < 1e-10))

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Lines 2 to 3 in 7fa521d

	if (rlang::is_installed(c("rsample", "parsnip", "yardstick", "workflows", "recipes", "kknn"))) {

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 15 in 7fa521d

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 26 in 7fa521d

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 31 in 7fa521d

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 36 in 7fa521d

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 41 in 7fa521d

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 51 in 7fa521d

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 54 in 7fa521d

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 62 in 7fa521d

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 72 in 7fa521d

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 74 in 7fa521d

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 85 in 7fa521d

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 95 in 7fa521d

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 97 in 7fa521d

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 108 in 7fa521d

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 119 in 7fa521d

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 122 in 7fa521d

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 131 in 7fa521d

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 135 in 7fa521d

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 139 in 7fa521d

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 143 in 7fa521d

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 148 in 7fa521d

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 150 in 7fa521d

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 161 in 7fa521d

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 171 in 7fa521d

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 182 in 7fa521d

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 186 in 7fa521d

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 191 in 7fa521d

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 195 in 7fa521d

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 204 in 7fa521d

c(1/6, 1/3, 1/2) # normalized to sum to 1

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 206 in 7fa521d

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 209 in 7fa521d

c(0.2, 0.3, 0.5) # already normalized to sum to 1

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 216 in 7fa521d

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 219 in 7fa521d

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 225 in 7fa521d

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 230 in 7fa521d

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 234 in 7fa521d

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 237 in 7fa521d

expect_true(is.na(tune:::.weighted_sd(c(1), c(1)))) # single value

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 243 in 7fa521d

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 246 in 7fa521d

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 248 in 7fa521d

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 259 in 7fa521d

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 267 in 7fa521d

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 269 in 7fa521d

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 280 in 7fa521d

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Lines 283 to 284 in 7fa521d

	expect_true(nrow(individual_metrics) >= 3) # At least one metric per fold

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 292 in 7fa521d

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 297 in 7fa521d

resamples = folds, # No weights

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 303 in 7fa521d

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 314 in 7fa521d

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 317 in 7fa521d

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 328 in 7fa521d

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 337 in 7fa521d

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 341 in 7fa521d

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 351 in 7fa521d

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 355 in 7fa521d

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 364 in 7fa521d

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 366 in 7fa521d

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 377 in 7fa521d

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 380 in 7fa521d

expected_weights <- weights / sum(weights) # normalized

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 389 in 7fa521d

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Lines 398 to 400 in 7fa521d

	splits1 <- rsample::make_splits(x = mtcars[1:20,], assessment = mtcars[21:32,])
	splits2 <- rsample::make_splits(x = mtcars[1:15,], assessment = mtcars[16:32,])

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 405 in 7fa521d

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 408 in 7fa521d

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 411 in 7fa521d

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Line 417 in 7fa521d

---

[air] _{reported by reviewdog 🐶}

tune/tests/testthat/test-weights.R

Lines 422 to 423 in 7fa521d

	}

[tjburch]

Sorry for the noise here - didn't realize Air was part of ci/cd now.

fwiw I've been using my fork here for a while in a production process and it seems fine. I'm going to dust it off and get it ready to go asap

[tjburch]

Alright, I think this is back ready for review.

[tjburch]

── Error ('test-checks.R:673:3'): fold weights with tune_grid ──────────────────
Error in `tune_grid(tune_wflow, weighted_folds, grid = simple_grid, control = control_grid(save_pred = FALSE))`: Package install is required for glmnet.

Looks like I'll need to replace the model here to something the ci tools have

[EmilHvitfeldt]

Looks like I'll need to replace the model here to something the ci tools have

yes that is correct! {xgboost} is a engine you can use, it is used in other tests

[topepo]

Looks pretty solid! Thanks for the contribution and your patience through our busy season.

I think the main thing to do is to make a standalone file for weighted statistics.

It could be helpful for you to veryify that your weighting functions get the same results as what is currently in recipes.

[topepo]

Should we check for equal weights too and return NULL if that is the case? I think that will be the case in some situations, and it would slightly increase efficiency to avoid work whenever possible.

[topepo]

Why not use stats ::weighted.mean()

[topepo]

So looking further down, we do the same with the variance calculation. We do have an unexported API in recipes wt_calcs() that does a lot of this.

We can copy it over here (to avoid duplication) and think about moving that core code to hardat so that we get it everywhere from one source.

[topepo]

Erm actually, we should probably put those weighting functions in a "standalone file" here, and then recipes and anyone else can import that.

@EmilHvitfeldt do you have any thoughts on that?

[EmilHvitfeldt]

I would be happy to have a standalone file for weighted functions

[topepo]

I'll amend my comments a third time. We don't export wt_calcs() but we do export the single statistic versions like recipes::averages() and recipes::variances() so maybe it's that simple.

[topepo]

These are all imported so no need for these skips

[topepo]

Same as ☝️ etc

[topepo]

The testing issue for:

namespace 'workflowsets' is not available and has been replaced

is a false alarm. That is fixed in #1104.

[topepo]

@tjburch I think that I'd like to change "fold weights" to "resample weights" to be more general. Is the former common terminology for these?

[tjburch]

@tjburch I think that I'd like to change "fold weights" to "resample weights" to be more general. Is the former common terminology for these?

I'm unaware if there is a common/formal terminology. This came out of a use-case, and that's just what I've been calling it :) . Resample weights is fine by me.

[topepo]

I moved one of the tests away from glmnet to svm. @tjburch The glmnet package is a difficult thing to have as a formal dependency. They sometimes make breaking changes that affect us so we test them in a different non-CRAN repo.

[topepo]

With printing in the rset objects, I think that we'll have to move this code into an rsample PR.

Most of the rsets have a superseding class (like v_fold) and the print method for that class happens first.

Using v_fold as an example, its print method removes the first two classes (v_fold and rset) so that the remaining printing is the default tibble printing. That's why there is not a rsample:::print.rset method.

I'm going to temporarily remove the print methods and make this comment an issue for rsample, and then Hannah can take a look (who is out of office next week 😿)

#' @export
print.rset <- function(x, ...) {
  fold_weights <- attr(x, ".fold_weights")

  if (!is.null(fold_weights)) {
    # Create a tibble with fold weights as a column
    x_tbl <- tibble::as_tibble(x)
    x_tbl$fold_weight <- fold_weights
    print(x_tbl, ...)
  } else {
    # Use default behavior
    NextMethod("print")
  }
}

#' @export
print.manual_rset <- function(x, ...) {
  fold_weights <- attr(x, ".fold_weights")

  if (!is.null(fold_weights)) {
    # Create a tibble with fold weights as a column
    x_tbl <- tibble::as_tibble(x)
    x_tbl$fold_weight <- fold_weights
    print(x_tbl, ...)
  } else {
    # Use default behavior for manual_rset
    NextMethod("print")
  }
}