Add vignette map example using native serialization

[shikokuchuo]

The custom functions vignette examples currently all involve converting to and from R objects.

With polars using mirai's serialization registry, this is not always going to be necessary.

It might be an idea to add some examples of how users can take advantage of this, e.g.:

library(polars)
library(purrr)
library(mirai)
daemons(6)

dat <- as_polars_df(mtcars)
dat |>
  as.list() |>
  map(in_parallel(\(x) x$sort())) |>
  as_polars_df()

There's no conversion to R types anywhere in this (you can verify line by line that the result is a polars object).

Not the best example, so can I leave it with you to find a more natural one?

Also happy to contribute something, just let me know.

[eitsupi]

Thanks for checking that!
How about this?

r-polars/vignettes/custom-functions.Rmd

Lines 111 to 132 in a9102a9

	In this case, instead of using `as.data.frame()` to convert all data to R,
	we can use `as.list()` to convert the polars DataFrame to a list of Series
	and only convert some Series to R, thus avoiding unnecessary conversions between
	Polars and R[^serialization].
	[^serialization]: In technical terms, polars Series and other objects are wrappers around external pointers,
	which usually cannot be sent to other processes. However, this package has built-in integration with `{mirai}`,
	which registers the serialization and deserialization functions for Series and other objects
	when the package is loaded. This allows users to treat Series and other polars objects like regular R objects.
	```{r, eval=purrr_parallel_available}
	dat |>
	as.list() |> # Convert to a list of Series, not converting each column to R vector yet.
	map_at(
	c("carb", "mpg"),
	# Since the Series are sent to each daemon,
	# we need to convert them to R vectors first by `$to_r_vector()`.
	# Finally, we convert them back to Series by `as_polars_series()` (in this case, optional).
	in_parallel(\(s) scale(s$to_r_vector())[, 1] |> polars::as_polars_series())
	) |>
	as_polars_df()
	```

In fact, if we are simply processing columns, I don't think using purrr will improve the speed.

One powerful use of daemon is to send lazyframes to other computers for aggregation, but this is not related to parallel processing and is not included in the article.

[shikokuchuo]

Lines 126-127, it's because scale() only works on R vectors that we need to do the conversion. Otherwise looks fine.

I can see that for this type of use case, it is indeed not such an advantage.

If you think of a more compelling example involving lazyframes or otherwise doing some lengthy calculation which actually uses the native format, do let me know and I'd be happy to update the mirai serialization vignette with a separate section for polars, highlighting the integration.

[etiennebacher]

The custom functions vignette examples currently all involve converting to and from R objects.
With polars using mirai's serialization registry, this is not always going to be necessary.

Lines 126-127, it's because scale() only works on R vectors that we need to do the conversion.

otherwise doing some lengthy calculation which actually uses the native format,

I think I'm missing something regarding native serialization. The main usecase is to be able to send only some of the Series to a daemon, as shown in the vignette, but not to apply polars functions on this Series. To me, if the custom function you want to apply can be written entirely with polars syntax, then there's no point using mirai or purrr for parallel processing. You should just write a custom function and use this in with_columns(), select(), etc. and polars will take care of optimizing the queries, for instance by doing common subplan elimination.

It seems to me that using purrr or mirai for parallel processing should be a last resort that happens when your code cannot be translated into polars syntax, for instance because it comes from an external package. We took the example of scale(), which would be simple enough to translate, but this could be any function from any R package. If the function in question doesn't accept polars Series (like the vast majority of functions), then we have to convert to R types.

One powerful use of daemon is to send lazyframes to other computers for aggregation

Would be great to have some usecases like this!

[eitsupi]

I think it would be better to add the documentation to integration-mirai instead of to create a new the vignette.

Just as a note, if we load polars after creating daemons like this, we'll get a strange error because mirai::register_serial doesn't propagate to the already created daemons.
It's probably worth checking to see if the daemons are already created and raising a warning:

mirai::daemons(1)
#> [1] 1

series <- polars::as_polars_series(1)$cast(polars::pl$Int128)

list(series) |>
  mirai::mirai_map(\(x) x * 2L) |>
  _[][[1]]
#> 'miraiError' chr Error in x$len(): Evaluation failed in `$len()`.

^{Created on 2025-07-24 with reprex v2.1.1}