[parallel] Introduce data parallelism abstraction #2621

clabby · 2025-12-26T00:03:32Z

Overview

Inspired by #2187, introduces commonware-parallel, which offers a thin abstraction over a Strategy that can perform a foldor join operation either sequentially (with Iterator) or in parallel (with rayon's ParallelIterator.) These strategies are objects, rather than marker traits, so that types can be given a Strategy to be used in any operations attached to it. The Parallel strategy owns a commonware_runtime::ThreadPool (Arc<rayon::ThreadPool>).

In addition, fixes commonware-runtime's create_pool function to work with the deterministic runtime by using rayon's ThreadPoolBuilder::use_current_thread option. When set, this enables the current thread to steal queued work when yielded to.

TODO:

coding
math
cryptography
Expose Strategy across the board? There's quite a few places where the concurrency argument was not exposed, and this PR tries to honor that. It may be beneficial, though, to allow passing a Strategy through every function / type that can make use of it? -> Exposed it in the areas I think make sense.

closes #1540
closes #2187

cloudflare-workers-and-pages · 2025-12-26T01:33:26Z

Deploying monorepo with Cloudflare Pages

Latest commit:	`92d9540`
Status:	✅ Deploy successful!
Preview URL:	https://8246b305.monorepo-eu0.pages.dev
Branch Preview URL:	https://cl-parallel-crate.monorepo-eu0.pages.dev

View logs

patrick-ogrady · 2025-12-27T06:25:22Z

math/src/algebra.rs

    ///
    /// For empty slices, the result should be [`Additive::zero`];
-    fn msm(points: &[Self], scalars: &[R], _concurrency: usize) -> Self {
+    fn msm<S: ParStrategy>(points: &[Self], scalars: &[R], _strategy: &S) -> Self {


nit: can we just use Strategy in places like this? I tend to only rename if there is an import conflict?

There is an import conflict, w/ proptest's Strategy.

_: &impl Parallel would be nice here, eh?

math/src/algebra.rs

cloudflare-workers-and-pages · 2025-12-29T16:10:02Z

Deploying with Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Updated (UTC)
✅ Deployment successful! View logs	commonware-mcp	`92d9540`	Jan 06 2026, 08:47 PM

consensus/src/simplex/scheme/bls12381_threshold.rs

cronokirby

The main RC is to move from a trait with open ended impls, to a concrete struct which dynamically knows how to parallelize things.

I think we should also eschew providing something as low level as join, in favor of focusing on "fold_init" as the source of the difference in behavior.

cronokirby · 2026-01-05T17:22:16Z

parallel/src/lib.rs

+    /// assert_eq!(sum, 55);
+    /// assert_eq!(product, 3628800);
+    /// ```
+    fn join<L, LO, R, RO>(&self, left: L, right: R) -> (LO, RO)


Are there places, already, that need this granular level of parallelism?

I think exposing this could be fine, but if so, we should use it to base other methods off of, to avoid burdening the trait too much.

Right now, no. The primary reason I added this was for future flexibility in areas where we may want to use parallel divide-and-conquer, for example in the bisection-style verification for BLS.

Glad to remove this for now given that no one uses it, if you'd prefer.

Removed this for now.

parallel/src/lib.rs

cronokirby · 2026-01-05T17:28:00Z

parallel/src/lib.rs

+    }
+}
+
+#[cfg(test)]


One idea to simplify the test suite is to enforce that:

parallel fold_init does the same thing as sequential fold_init,

sequential "other method" does the same thing as the expression in terms of fold_init

I think you could probably streamline the test suite a lot with that.

I think we could do that in a follow up PR though, I think the existing tests provide enough coverage.

Updated these while removing IntoStrategyIterator 👍

cronokirby · 2026-01-05T19:27:31Z

parallel/src/lib.rs

+/// let range = 0..100;
+/// let _iter = range.into_iter();
+/// ```
+pub trait IntoStrategyIterator: IntoIterator {


I think having a cfg_if defined body of the trait is really hacky.

Also, having rayon leak into the public interface of the module seems less than ideal.

With a concrete struct strategy, one approach would be to have this trait represent the concept of being able to call fold on itself (or whatever the ur-method ends up being) using the strategy as an argument. Then, for feature gating, we would just create a nice impl which hooks into a concrete method on strategy which has parallel iterator as a bound. This would avoid having to know about Rayon to implement this for types that don't need parallelism.

With having rayon leak into the public interface, what are the issues you foresee there? IntoStrategyIterator intentionally bridges regular iterators with rayon's parallel iterators (when in an std context.) Hiding rayon would allow us to use other backends for data parallelism, but I couldn't think of another that we may want to use.

This would avoid having to know about Rayon to implement this for types that don't need parallelism.

The way I was thinking about it is that you would only use Strategy for situations where you want to allow for parallelism via rayon specifically. If you don't need parallelism, I was assuming you likely wouldn't use Strategy for the type at all. (This was also the intention behind the IntoStrategyIterator bridge requiring that T: IntoParallelIterator outside of no_std contexts.)

Is "use Strategy when you may or may not want to parallelize T's operations with rayon" too narrow?

Re point 2: you need to use Strategy if you want consumers of your code to be able to parallelize your code. It's correct that there aren't situations where you'd use Strategy if you never want parallelism. However, there are many situations where you need to use Strategy, while also wanting your code to be run Sequentially in, e.g. WASM, or other constrained environments. In that case, you need a public API that's easy to satisfy regardless of what capabilities you have, since that kind of code will have to assume the least possible. You want to avoid the virality induced by having a public API required just to call a function on strategy be feature flagged. Instead, the feature flagging should really only affect what means you have to construct a strategy. Once you have one, it should be usable even if no_std environments.

Re point 1: the main issue is that it makes using the crate more annoying and require reading more documentation needlessly. The user shouldn't have to understand both this crate and read up docs on rayon on how to construct things or what types implement the trait we're linking them to, it should just be a complete interface. It would also be nice to have the option to use other backends, especially if the limitation there comes from exposing implementation details, rather than a higher level conceptual blocker.

However, there are many situations where you need to use Strategy, while also wanting your code to be run Sequentially in, e.g. WASM, or other constrained environments. In that case, you need a public API that's easy to satisfy regardless of what capabilities you have, since that kind of code will have to assume the least possible. You want to avoid the virality induced by having a public API required just to call a function on strategy be feature flagged. Instead, the feature flagging should really only affect what means you have to construct a strategy. Once you have one, it should be usable even if no_std environments.

This makes sense. I think the existing API is relatively friendly to deal with here personally, but def open to disagreement. Users should never manually implement IntoStrategyIterator for themselves, their code would look something like:

struct Thing([u8; 16]); impl IntoIterator for Thing { /* ... */ } #[cfg(feature = "std")] impl IntoParallelIterator for Thing { /* ... */ }

[dependencies] rayon = { workspace = true, optional = true } [features] std = ["dep:rayon"]

where feature flagging would be necessary anyways due to including rayon in the std feature. Thing here would automatically implement commonware_parallel::IntoStrategyIterator, whether std was enabled or not.

the main issue is that it makes using the crate more annoying and require reading more documentation needlessly. The user shouldn't have to understand both this crate and read up docs on rayon on how to construct things or what types implement the trait we're linking them to, it should just be a complete interface.

I mainly didn't want to prematurely abstract here, didn't know of any other backends we may want to use. Are there any that we're thinking of adding?

I agree that whatever trait bounds we require for a collection, they should be automatically implemented on any type a user will realistically provide, and it can be more complex to implement it manually.

It's less that we would consider using something other than rayon, but more so that we don't want to commit to a public API based on rayon, because then you're stuck with its way of modelling things forever.

Even more importantly, you want to avoid a situation in which to understand and use this crate, you also need to read a bunch of rayon docs first to figure out how to construct the things you need to pass in here, or what types this crate is asking for, etc. In other words, we want to abstract over rayon, not provide a slightly different interface in addition to it.

patrick-ogrady

Should create_pool in the runtime return Parallel (rather than its own type)?

In any case, we should probably have Arc<rayon::ThreadPool> live in parallel now?

patrick-ogrady · 2026-01-06T14:48:24Z

coding/src/benches/bench.rs

                    minimum_shards: min as u16,
                    extra_shards: (chunks - min) as u16,
                };
+                let pool = Arc::new(ThreadPoolBuilder::new().num_threads(conc).build().unwrap());


Can we create a helper for this in parallel?

patrick-ogrady · 2026-01-06T14:57:22Z

coding/src/benches/bench.rs

+                                        &Parallel::new(pool.clone()),
+                                    )
+                                } else {
+                                    S::encode(&config, data.as_slice(), &Sequential)


A shame that there isn't a single variable we could use to house Sequential/Parallel to avoid rewriting the function.

I suppose we could have an enum of sorts that we just provide conc to but maybe that defeats the purpose.

We should use Box<dyn Strategy>

This is the discussion around using a concrete type (or Box<dyn Strategy>) vs. the trait.

From what we spoke about on slack - Strategy (now Parallel) cannot be (usefully) dyn-compatible because each generic method requires Self: Sized and therefore cannot be included in the trait object's vtable.

examples/bridge/src/bin/validator.rs

examples/reshare/src/validator.rs

patrick-ogrady · 2026-01-06T15:13:04Z

parallel/README.md

+[![Crates.io](https://img.shields.io/crates/v/commonware-parallel.svg)](https://crates.io/crates/commonware-parallel)
+[![Docs.rs](https://docs.rs/commonware-parallel/badge.svg)](https://docs.rs/commonware-parallel)
+
+Abstract data parallelism over iterators and tasks.


nit: We should make the first word a verb

(maybe Parallelize...)

patrick-ogrady · 2026-01-06T15:14:55Z

runtime/src/utils/mod.rs

    }

+    #[test_traced]
+    fn test_create_pool_deterministic() {


cronokirby

Some more comments:

I have a naming suggestion, to make the use of Strategy clearer.

I think we should also consistently use &impl Strategy rather than an explicit generic ; consistency matters more, and I think this creates much less noise.

We also want to avoid storing a thread pool and creating the strategy on demand, in favor of storing the strategy.

parallel/src/lib.rs

cronokirby · 2026-01-06T16:13:21Z

math/src/algebra.rs

    ///
    /// For empty slices, the result should be [`Additive::zero`];
-    fn msm(points: &[Self], scalars: &[R], _concurrency: usize) -> Self {
+    fn msm<S: ParStrategy>(points: &[Self], scalars: &[R], _strategy: &S) -> Self {


_: &impl Parallel would be nice here, eh?

cronokirby · 2026-01-06T16:17:57Z

coding/src/benches/bench.rs

+                                        &Parallel::new(pool.clone()),
+                                    )
+                                } else {
+                                    S::encode(&config, data.as_slice(), &Sequential)


We should use Box<dyn Strategy>

Uses `ParallelBridge` from the `rayon` crate to convert iterators into parallel iterators, simplifying the codebase by removing the custom `IntoStrategyIterator` trait. There's a slight performance hit, but the API is much more ergonomic and doesn't bind to `rayon` specifics.

…ential` -> `ParallelNone`

cronokirby

Awesome!

codecov · 2026-01-06T21:53:26Z

Codecov Report

❌ Patch coverage is 96.58444% with 18 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.87%. Comparing base (1e0d536) to head (92d9540).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
parallel/src/lib.rs	92.96%	9 Missing ⚠️
runtime/src/lib.rs	86.95%	3 Missing ⚠️
runtime/src/utils/cell.rs	0.00%	3 Missing ⚠️
consensus/src/simplex/scheme/bls12381_threshold.rs	98.97%	1 Missing ⚠️
cryptography/src/bls12381/scheme.rs	0.00%	1 Missing ⚠️
runtime/src/deterministic.rs	93.33%	1 Missing ⚠️

@@            Coverage Diff             @@
##             main    #2621      +/-   ##
==========================================
+ Coverage   92.82%   92.87%   +0.04%     
==========================================
  Files         361      362       +1     
  Lines      107102   107278     +176     
==========================================
+ Hits        99416    99631     +215     
+ Misses       7686     7647      -39

Files with missing lines	Coverage Δ
coding/src/lib.rs	`92.30% <100.00%> (+0.05%)`	⬆️
coding/src/no_coding.rs	`94.02% <100.00%> (ø)`
coding/src/reed_solomon.rs	`97.39% <100.00%> (ø)`
coding/src/zoda.rs	`94.63% <100.00%> (+2.94%)`	⬆️
consensus/src/simplex/elector.rs	`100.00% <100.00%> (ø)`
...tography/src/bls12381/certificate/threshold/mod.rs	`99.19% <100.00%> (+<0.01%)`	⬆️
cryptography/src/bls12381/dkg.rs	`95.28% <100.00%> (+0.01%)`	⬆️
cryptography/src/bls12381/primitives/group.rs	`91.71% <100.00%> (ø)`
cryptography/src/bls12381/primitives/ops/batch.rs	`96.98% <100.00%> (+0.05%)`	⬆️
cryptography/src/bls12381/primitives/ops/mod.rs	`99.03% <100.00%> (+<0.01%)`	⬆️
... and 20 more

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1e0d536...92d9540. Read the comment docs.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

clabby self-assigned this Dec 26, 2025

clabby added the breaking-api This PR modifies the public interface of a function. label Dec 26, 2025

clabby added this to Tracker Dec 26, 2025

clabby moved this to In Progress in Tracker Dec 26, 2025

clabby force-pushed the cl/parallel-crate branch 3 times, most recently from f82cc6b to 9ec19df Compare December 26, 2025 16:07

patrick-ogrady reviewed Dec 27, 2025

View reviewed changes

math/src/algebra.rs Outdated Show resolved Hide resolved

clabby force-pushed the cl/parallel-crate branch 2 times, most recently from 5e5fafb to dca457f Compare December 29, 2025 16:09

clabby force-pushed the cl/parallel-crate branch 5 times, most recently from dc44339 to ddbcf2c Compare January 5, 2026 15:01

clabby commented Jan 5, 2026

View reviewed changes

consensus/src/simplex/scheme/bls12381_threshold.rs Outdated Show resolved Hide resolved

clabby marked this pull request as ready for review January 5, 2026 16:15

clabby moved this from In Progress to Ready for Review in Tracker Jan 5, 2026

cronokirby requested changes Jan 5, 2026

View reviewed changes

clabby force-pushed the cl/parallel-crate branch 2 times, most recently from 3c2b385 to 9611aec Compare January 5, 2026 22:07

patrick-ogrady reviewed Jan 6, 2026

View reviewed changes

cronokirby reviewed Jan 6, 2026

View reviewed changes

clabby force-pushed the cl/parallel-crate branch 4 times, most recently from a1d9872 to 753268c Compare January 6, 2026 17:04

clabby added 19 commits January 6, 2026 15:42

Disable default features for parallel in workspace

9b2624b

review

3a203f9

remove Strategy::join

9b931b0

simplify tests

59fb94e

Upfront allocation over par_bridge

3c204ad

Add helper for constructing a thread pool

5f19a75

Use impl Strategy in Space::msm

90e46b9

Rename Strategy -> Parallel, Parallel -> ParallelRayon, `Sequ…

c24ec88

…ential` -> `ParallelNone`

new description

3952835

review

a9384e0

Move create_pool to {tokio, deterministic}::Context

848f48f

change names (again.)

9d0ff05

Add create_strategy helper

674bcba

rename trait (again)

c2d8af5

Remove dangling create_pool methods

da4d93a

doc tests

6cc61ae

rm deterministic comment on dedicated threads

075777b

rebase

92d9540

clabby force-pushed the cl/parallel-crate branch from d85f023 to 92d9540 Compare January 6, 2026 20:46

patrick-ogrady approved these changes Jan 6, 2026

View reviewed changes

clabby requested a review from cronokirby January 6, 2026 21:09

cronokirby approved these changes Jan 6, 2026

View reviewed changes

patrick-ogrady merged commit 63a20d7 into main Jan 6, 2026
127 checks passed

patrick-ogrady deleted the cl/parallel-crate branch January 6, 2026 21:53

github-project-automation bot moved this from Ready for Review to Done in Tracker Jan 6, 2026

This was referenced Jan 8, 2026

[consensus/simplex] Fix Seedable impl for bls12381_threshold scheme #2756

Merged

[consensus] Improve consensus' usage of Strategy #2762

Merged

patrick-ogrady mentioned this pull request Jan 9, 2026

[cryptography/bls12381] Migrate to using rayon::ThreadPool #903

Closed

[parallel] Introduce data parallelism abstraction #2621

[parallel] Introduce data parallelism abstraction #2621

Uh oh!

Conversation

clabby commented Dec 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Uh oh!

cloudflare-workers-and-pages bot commented Dec 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploying monorepo with Cloudflare Pages

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cloudflare-workers-and-pages bot commented Dec 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploying with Cloudflare Workers

Uh oh!

Uh oh!

cronokirby left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

clabby Jan 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

patrick-ogrady left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cronokirby left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

clabby commented Dec 26, 2025 •

edited

Loading

cloudflare-workers-and-pages bot commented Dec 26, 2025 •

edited

Loading

cloudflare-workers-and-pages bot commented Dec 29, 2025 •

edited

Loading

clabby Jan 5, 2026 •

edited

Loading