Package structure and architectural roadmap

[pkienscherf]

Hi all,

I recently started re-implementing some RL projects I've been working on within the RL.jl framework (due to its great auxiliary functions, friendly enforcing of APIs and logging that really helps/saves work!). Starting to dig deeper into the package itself, particularly with a view to implementing some MARL applications in the future, some questions regarding the package architecture arose to me:

Why is RL.jl currently designed as wrapper around sub-packages with own dependencies and versioning? Would it not be easier to use "regular" sub-modules instead? Is the structure mostly due to legacy or by design? I created a fork (pkienscherf/ReinforcementLearning.jl) with an alternative structure of sub-modules that made it easier for me to work with the package in development mode, so I'd be interested to hear what you think about such approach (obviously that was a quick draft not including some of the remaining re-structuring like moving the test).

My second thought concerns the difference between Core and Base. Obviously from code inspection you can find some general patterns, but I find the two terms (semantically) quite interchangeable, especially for RL.jl-newbies who might want to contribute to the package (like myself 😊 ).

Both of these points also refer to a more general question, as to whether there is a mid- to long-term roadmap for the package and its architecture to orient contributors?

I hope I was not inadvertently hypercritical; I really like RL.jl and I'd love to help moving it forward once I understand it better (and have sufficient time on my hands 😊).

[findmyway]

Hi @pkienscherf ,

I recently started re-implementing some RL projects I've been working on within the RL.jl framework (due to its great auxiliary functions, friendly enforcing of APIs and logging that really helps/saves work!). Starting to dig deeper into the package itself, particularly with a view to implementing some MARL applications in the future,

Looking forward to your upcoming works!

Why is RL.jl currently designed as wrapper around sub-packages with own dependencies and versioning? Would it not be easier to use "regular" sub-modules instead? Is the structure mostly due to legacy or by design? I created a fork (pkienscherf/ReinforcementLearning.jl) with an alternative structure of sub-modules that made it easier for me to work with the package in development mode, so I'd be interested to hear what you think about such approach (obviously that was a quick draft not including some of the remaining re-structuring like moving the test).

This is by design. See discussions here.

I bet you've seen the project structure in the README:

+-----------------------------------------------------------------------------------+
|                                                                                   |
|  ReinforcementLearning.jl                                                         |
|                                                                                   |
|      +------------------------------+                                             |
|      | ReinforcementLearningBase.jl |                                             |
|      +----|-------------------------+                                             |
|           |                                                                       |
|           |     +--------------------------------------+                          |
|           +---->+ ReinforcementLearningEnvironments.jl |                          |
|           |     +--------------------------------------+                          |
|           |                                                                       |
|           |     +------------------------------+                                  |
|           +---->+ ReinforcementLearningCore.jl |                                  |
|                 +----|-------------------------+                                  |
|                      |                                                            |
|                      |     +-----------------------------+                        |
|                      +---->+ ReinforcementLearningZoo.jl |                        |
|                            +----|------------------------+                        |
|                                 |                                                 |
|                                 |     +-------------------------------------+     |
|                                 +---->+ DistributedReinforcementLearning.jl |     |
|                                       +-------------------------------------+     |
|                                                                                   |
+------|----------------------------------------------------------------------------+
       |
       |     +-------------------------------------+
       +---->+ ReinforcementLearningExperiments.jl |
       |     +-------------------------------------+
       |
       |     +----------------------------------------+
       +---->+ ReinforcementLearningAnIntroduction.jl |
             +----------------------------------------+

Let me explain the goal of each sub-package first since based on your following comment, it seems the doc doesn't make this point clear.

RLBase only defines a set of standard concepts and interfaces. It should be lightweight with minimal dependencies. For other julia packages which would like to use algorithms in RL.jl or solve problems defined in RL.jl in approaches beyond reinforcement learning, they only need to add a dependency on this package solely.

RLCore defines the skeleton of a typical RL experiment.

RLZoo is to add many different implementations in the Agent part in the above picture and RLEnvs is to add more environments Environments part.

So the answer to your question is now clear. Different packages have different targeting users.

• ReinforcementLearningBase, some other packages which want to communicate with packages under the JuliaReinforcementLearning org should add dependency of this package. For example, POMDPs.jl
• ReinforcementLearningEnvironments, for packages who would like to solve general RL problems in algorithms not provided in RL.jl. For example, AlphaZero.jl
• ReinforcementLearningCore, for people like you who plan to add many independent algorithms without relying on existing algorithms in RLZoo.jl but wish to reuse the general pipeline in the above picture.
• ReinforcementLearningZoo, for packages who would like to use the vast built in algorithms but do not need any environments provided in ReinforcementLearningEnvironments. See SeaPearl.jl (although they added a dependency on RL.jl directly.)
• ReinforcementLearning, simply use all the components in above subpackages. For example ReinforcementLearningAnIntroduction.jl
• ReinforcementLearningExperiments, added many built-in experiments for users to try out some benchmarks. Note that this package contains much more dependencies compared to RL.jl. And generally other packages shouldn't rely on it.

Hope this will make things more clear in your mind.

but I find the two terms (semantically) quite interchangeable,

Emm, I think they are not interchangeable? For interfaces defined in ReinforcementLearningBase, mainly environments and policies, we suggest adding a RLBase prefix when adding a new dispatch, while for others defined in ReinforcementLearningCore, like learners, hooks..., we usually add a RLCore prefix.

Both of these points also refer to a more general question, as to whether there is a mid- to long-term roadmap for the package and its architecture to orient contributors?

TBH I do not have a very clear roadmap in my head. Currently we have several OSPP students working on offline RL and MARL, hoping that we can have a more feature-rich ecosystem. On one hand, I wish this package can be polished further to have a more fluent experience when applying RL algorithms in this package to user's customized problems. I must admit that there's still a large gap compared to many RL packages in the Python world like StableBaselines/RLlib/acme. And I can receive people's feature request every now and then. On the other hand, I plan to bring this package to a new stage with a distributed actor system (some ongoing work Oolong.jl), hoping that this package will have more practical usages.

I hope I was not inadvertently hypercritical; I really like RL.jl and I'd love to help moving it forward once I understand it better (and have sufficient time on my hands 😊).

Of course not. Really wish to have more people on board. 🤗

[pkienscherf]

Great, thanks for the swift and extensive reply and explanations! Things are way clearer for me now. Having standalone packages that can/should be added independently put together in a single repository was just unfamiliar to me (I hadn't found the linked issue before explaining why). I'll see what workflow works best when using RL.jl in development mode. Thanks!

[findmyway]

Also see https://juliareinforcementlearning.org/docs/tips/#How-to-setup-local-development-environment? for setting up local development environment.

[pkienscherf]

That's what I did and didn't seem to work as expected before -- but now it does. Oh well :) Anyway, thanks!