Yes I am planning both (multiple tables in parallel and supporting multiple generators to writers).

Yes I am planning both (multiple tables in parallel and supporting multiple generators to writers).

I would love to help

I have been thinking about "how would we use multiple threads to generate line item data"

What I was thinking about so far is:

1. Make sure all generator / iterators are Clone (so you can .clone() to get a copy)
2. Add a skip_num(num_records: usize) method to each one

Then we could generate output chunks in parallel like

let iter = LineItemGenerator::new(...).iter();

// make first 1000 rows into a buffer
let chunk_1 = generate_lines(iter.clone().take(1000));
// make next 1000 rows ...
let iter = iter.skip_num(1000); // call special skip which just advances the rngs
let chunk_2 = generate_lines(iter.clone().take(1000)
...
let chunk_N = generate_lines(iter.clone().take(1000)

println!("{}", chunk_1);
println!("{}", chunk_2);
...
println!("{}", chunk_n);

I think skip_num would be pretty fast as it just advances the random number generators (and could skip all actual row generation logic)

So this solution requires advancing each random number generator 2x, but I think it could be put all the cores to use pretty effectively

I kind of reached the same conclusion; although I am more inclined to hide skip_num logic here. My line of thought is that library users should transparently be able to have parallel and non-parallel iterators (like how rayon exposes par_iter). But I'll have to tinker about it.

I kind of reached the same conclusion; although I am more inclined to hide skip_num logic here. My line of thought is that library users should transparently be able to have parallel and non-parallel iterators (like how rayon exposes par_iter). But I'll have to tinker about it.

Perhaps we can start just using the normal iterator skip (which will do a bunch of work that is not strictly necessary). If that turns out to be too slow we can look into a skip_num optimization -- it could well be a premature optimization

I actually played around with this this morning -- I think we can use the existing "part count" functionality to do this. I'll monkey with it

I am quite pleased I have a proof of concept that this approach can be made to work (and it generates data crazy fast)

• (POC) Line item generation in parallel #34