MultithreadedEnvs and Trajectories

[HenriDeh]

I'm creating this issue to pin down that MultiThreadedEnvs do not currently work.

It steps each inner env in parallel and stores the traces, but there's nothing that can be done with that currently as we would need to store each separate trajectory (in the seminal sense of "trajectory") contiguously in the replay buffer. Right now with the caching implementation, they can only be mixed up in the storage.

Options I see to address that

1. extend trajectories to have N (number of parallel envs) containers, each will pertain to one env and have a capacity of 1/Nth the total capacity of the trajectory.
2. partition the container into N partitions.
3. Create a small Trajectory for each env and append to the main Trajectory at the end of an episode.

I clearly prefer the first option, it is straightfoward and probably the most efficient.

[Mytolo]

In my opinion, in the first option, each container should have the full capacity? The MultiThreadEnv is simply to speed up experience gathering, so each container should behave as it would be a single environment. User perspective would be to define how much experience should be stored for one environment. That way, there are also no worries about rounding / ceiling for the capacity, since it has to be a whole number.