Skip to content

Commit

Permalink
add support for parallelism with future (#208)
Browse files Browse the repository at this point in the history
* add support for parallelism with future
* use future's RNG when using doFuture
  • Loading branch information
simonpcouch authored Mar 14, 2024
1 parent f4696d8 commit d768536
Show file tree
Hide file tree
Showing 4 changed files with 22 additions and 9 deletions.
6 changes: 4 additions & 2 deletions DESCRIPTION
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
Package: stacks
Title: Tidy Model Stacking
Version: 1.0.3.9000
Version: 1.0.3.9001
Authors@R: c(
person("Simon", "Couch", , "[email protected]", role = c("aut", "cre")),
person("Max", "Kuhn", , "[email protected]", role = "aut"),
Expand All @@ -19,8 +19,10 @@ Depends:
Imports:
butcher (>= 0.1.3),
cli,
doFuture,
dplyr (>= 1.1.0),
foreach,
future,
generics,
ggplot2,
glmnet,
Expand Down Expand Up @@ -57,4 +59,4 @@ Config/testthat/edition: 3
Encoding: UTF-8
LazyData: true
Roxygen: list(markdown = TRUE)
RoxygenNote: 7.2.3.9000
RoxygenNote: 7.3.1
2 changes: 2 additions & 0 deletions NEWS.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# stacks (development version)

* Introduced support for parallel processing using the [future](https://www.futureverse.org/) framework. The stacks package previously supported parallelism with foreach, and users can use either framework for now. In a future release, stacks will begin the deprecation cycle for parallelism with foreach, so we encourage users to begin migrating their code now. See [the _Parallel Processing_ section in the tune package's "Optimizations" article](https://tune.tidymodels.org/articles/extras/optimizations.html#parallel-processing) to learn more (#866).

* Improved error message for unsupported model modes (#152).

# stacks 1.0.3
Expand Down
19 changes: 14 additions & 5 deletions R/fit_members.R
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,8 @@
#' model contains the necessary components to predict on new data.
#'
#' @details
#' To fit members in parallel, please register a parallel backend function.
#' See the documentation of [foreach::foreach()] for examples.
#' To fit members in parallel, please create a plan with the future package.
#' See the documentation of [future::plan()] for examples.
#'
#' @template note_example_data
#'
Expand Down Expand Up @@ -117,15 +117,24 @@ fit_members <- function(model_stack, ...) {
dplyr::full_join(metrics_dict, by = c("value" = ".config"), multiple = "all")
}

if (foreach::getDoParWorkers() > 1) {
`%do_op%` <- foreach::`%dopar%`
if (foreach::getDoParWorkers() > 1 || future::nbrOfWorkers() > 1) {
`%do_op%` <- switch(
# note some backends can return +Inf
min(future::nbrOfWorkers(), 2),
foreach::`%dopar%`,
doFuture::`%dofuture%`
)
} else {
`%do_op%` <- foreach::`%do%`
}

# fit each of them
member_fits <-
foreach::foreach(mem = member_names, .inorder = FALSE) %do_op% {
foreach::foreach(
mem = member_names,
.inorder = FALSE,
.options.future = list(seed = TRUE)
) %do_op% {
asNamespace("stacks")$fit_member(
name = mem,
wflows = model_stack[["model_defs"]],
Expand Down
4 changes: 2 additions & 2 deletions man/fit_members.Rd

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

0 comments on commit d768536

Please sign in to comment.