-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathbackends.Rmd
73 lines (57 loc) · 3.57 KB
/
backends.Rmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
---
title: "Parallel Backends"
description: "Quickly change how and where your parallel code runs"
preview: images/site_preview.png
---
By default, future-based code runs sequentially, but with a _single
line of code_, we easily switch to run the exact same code in
parallel. The most common approach is to parallelize on the local
machine, but we have also the option to harness the CPUs of other
local or remote machines. For example, to parallelize on the local
machine, the end-user can call:
```r
plan(multisession)
```
After this, all of Futureverse, including **[future.apply]**,
**[furrr]**, and **[doFuture]**, and any package that use these, will
run the code in parallel.
To switch back to sequential processing, we can call:
```r
plan(sequential)
```
If you have Secure Shell (SSH) access to other machines on your local
network, or remote machines, call:
```r
plan(cluster, workers = c("n1", "n1", "n2", "remote.server.org"))
```
This will set up four parallel workers, where two run on the local
'n1' machine, another on the local 'n2' machine, and the fourth on the
remote 'remote.server.org' machine.
In addition to the above built-in parallel backends, more are provided
by other R packages, as show in the table below.
| Parallel backend | Description
|:-----------------------|:----------------------------------------------
| `sequential` | Run future-based code sequentially (default).
| `multisession` | Parallelize on the local machine via persistent background R processes.
| `cluster` | Parallelize across local and remote machines via persistent background R processes.
| `multicore` | Parallelize on the local machine via transient, forked R processes.
| `callr` | Parallelize on the local machine via transient background R processes. Available in the **[future.callr]** package.
| `mirai_multisession` | Parallelize on the local machine via persistent background R processes. Available in the **[future.mirai]** package.
| `mirai_cluster` | Parallelize across local and remote machines via persistent background R processes. Available in the **[future.mirai]** package.
| `batchtools_lsf` | Parallelize via the high-performance-compute (HPC) scheduler Load Sharing Facility (LSF). Available in the **[future.batchtools]** package.
| `batchtools_openlava` | Parallelize via the high-performance-compute (HPC) scheduler OpenLava. Available in the **[future.batchtools]** package.
| `batchtools_pbs` | Parallelize via the high-performance-compute (HPC) scheduler TORQUE/PBS. Available in the **[future.batchtools]** package.
| `batchtools_sge` | Parallelize via the high-performance-compute (HPC) scheduler Son/Sun/Oracle/Univa Grid Engine (SGE). Available in the **[future.batchtools]** package.
| `batchtools_slurm` | Parallelize via the high-performance-compute (HPC) scheduler Slurm. Available in the **[future.batchtools]** package.
It is straightforward to implement new backends that leverage other
ways to harness available compute resources. As soon as a new backend
has been validated to be compliant with the Future API specifications,
which can be done by the **[future.tests]** package, then it can be
used anywhere future-based code is used.
[doFuture]: https://doFuture.futureverse.org
[furrr]: https://furrr.futureverse.org
[future.apply]: https://future.apply.futureverse.org
[future.batchtools]: https://future.batchtools.futureverse.org
[future.callr]: https://future.callr.futureverse.org
[future.mirai]: https://future.mirai.futureverse.org
[future.tests]: https://future.tests.futureverse.org