Next Release Plan (v0.11)

[findmyway]

Goal

Improve the interactions between ReinforcementLearning.jl and other ecosystems in Julia.

Why is it important?

In the early days of developing this package, the main goal is to reproduce some popular (deep) RL algorithms. It's still important to keep adding new emerging algorithms into this package. But as an engineer, I always think the higher impact is achieved only when users really apply those powerful RL algorithms to the problems they are interested in. In recent years, many important packages across different domains were developed in Julia and the whole ecosystem improved a lot. Although the interfaces defined in this package are loose and flexible, people are still unsure how to use this package due to lacking concrete examples. Adding more examples and removing some restricted assumptions will greatly encourage more people to try this package. On the other hand, doing so will also improve the quality of this package.

Potential breaking changes

The most important change would be decoupling the training data generation and policy optimization. The state is assumed to be a tensor by default in many cases. This is the main blocking issue when interacting with many other packages. Besides, the async training pipeline will not only improve the performance of existing algorithms on a single node but also provide the foundation of large scale training in future releases (possibly in v0.12)

Key issues to be addressed

Following are some of the existing issues on the top of my mind. Please raise new ones if you wish to be addressed in the next release.

Environments

• Add a dedicated multi-dimensional space type #268
• CI fails with [email protected] #572 Stil no luck to address this issue, so I have to remove OpenSpiel related part in the next release. enable OpenSpiel #691
• Demonstrate that environments can be implemented on not only CPU but also GPU add CUDA accelerated Env #121
• How to display/render AtariEnv? #546
• Take a look at https://github.com/dojo-sim/Dojo.jl
• Take a look at https://github.com/Lyceum/MuJoCo.jl
• More examples with DifferentialEquations.jl
• https://github.com/deepmind/android_env Should be OK through PyCall.jl. Need some examples.
• Look into https://github.com/corail-research/SeaPearl.jl and see if there could be some improvements.

Refactor Existing Policies

• BasicDQN
• JuliaRL_BC_CartPole
• JuliaRL_DQN_CartPole Add JuliaRL_DQN_CartPole #650
• JuliaRL_PrioritizedDQN_CartPole add PrioritizedDQN #698
• JuliaRL_Rainbow_CartPole add rainbow #724
• JuliaRL_QRDQN_CartPole add QRDQN #699
• JuliaRL_REMDQN_CartPole add REMDQN #708
• JuliaRL_IQN_CartPole add IQN #710
• JuliaRL_VMPO_CartPole
• JuliaRL_VPG_CartPole add VPG #733
• JuliaRL_BasicDQN_MountainCar
• JuliaRL_DQN_MountainCar
• JuliaRL_A2C_CartPole
• JuliaRL_A2CGAE_CartPole
• JuliaRL_PPO_CartPole
• JuliaRL_MAC_CartPole
• JuliaRL_DDPG_Pendulum
• JuliaRL_SAC_Pendulum
• JuliaRL_TD3_Pendulum
• JuliaRL_PPO_Pendulum
• JuliaRL_BasicDQN_SingleRoomUndirected

Add New Policies

• Question: Can ReinforcementLearning.jl handle Partially Observed Markov Processes (POMDPs)? #608

• Rename some functions to help beginners navigate source code #326

• Improve the code structure and docs on general utils when defining a network Unify common network architectures and patterns #139

• Add alternatives to Flux.jl Experimental support of Torch.jl #136

• Change clip_by_global_norm! into a Optimizer #193

• Derivative-Free Reinforcement Learning #206

• Reinforcement Learning and Combinatorial Optimization #250

• Model based reinforcement learning #262 and WIP: PETS algorithm from facebook/mbrl #531

• Revisit Support multiple discrete action space #347

• Gain in VPGPolicy does not account for terminal states? #578

• Integrate CFR related algos at https://github.com/WhiffleFish/CounterfactualRegret.jl ?

• Combine transformers and RL #392 Borrow some ideas from https://github.com/facebookresearch/salina

Training pipeline

• Support async training Asynchronous Methods for Deep Reinforcement Learning #142
• Recurrent Models #144
• non-episodic environments Add a way to handle non-episodic environments #613 changes to policies may also be needed.

Documentation

• Missing docs for how to implement a new algorithm #582
• Missing docs for TDLearner #580
• Explain MultiThreadEnv in detail. MultiThreadEnv with custom (continuous) action spaces fails #596

Utils

• Visualization. Leverage Term.jl based on the suggestion here
• Support Tables.jl and PrettyTables.jl for Trajectories #232
• Observation. Integrate OpenTelemetry.jl to provide a more unified approach for recording.
• Setup CI to generate the Docker image

Timeline

I'm not sure I can fix them all. But at least I'll take a deep look into them and then tag a new release at the end of this quarter (around the end of June 2022).

[findmyway]

@JuliaRegistrator register

[JuliaRegistrator]

Registration pull request created: JuliaRegistries/General/61734

After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.

This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:

git tag -a v0.10.1 -m "<description of version>" 2e1de3e5b6b8224f50b3d11bba7e1d2d72c6ef7c
git push origin v0.10.1

[findmyway]

@JuliaRegistrator register subdir=src/ReinforcementLearningBase

[JuliaRegistrator]

Registration pull request created: JuliaRegistries/General/63479

After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.

This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:

git tag -a ReinforcementLearningBase-v0.10.0 -m "<description of version>" b58c7c4e7c7ecb141df135023dd284cab487c6e0
git push origin ReinforcementLearningBase-v0.10.0

[findmyway]

@JuliaRegistrator register subdir=src/ReinforcementLearningEnvironments

[JuliaRegistrator]

Registration pull request created: JuliaRegistries/General/63531

After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.

This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:

git tag -a ReinforcementLearningEnvironments-v0.7.0 -m "<description of version>" 06aa63e06f255654e5ae967ec4737cafd4d77908
git push origin ReinforcementLearningEnvironments-v0.7.0

[findmyway]

@JuliaRegistrator register subdir=src/ReinforcementLearningCore

[JuliaRegistrator]

Registration pull request updated: JuliaRegistries/General/63532

After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.

This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:

git tag -a ReinforcementLearningCore-v0.9.0 -m "<description of version>" 0f00ba42a00aacbf7cd8a31ab6b6e3c4cf14c1ef
git push origin ReinforcementLearningCore-v0.9.0

[jeremiahpslewis]

@JuliaRegistrator register subdir=src/ReinforcementLearningBase

[JuliaRegistrator]

Registration pull request created: JuliaRegistries/General/78892

After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.

This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:

git tag -a ReinforcementLearningBase-v0.11.0 -m "<description of version>" 107ceec40976d8023bc2b17a82861e4a240b7dee
git push origin ReinforcementLearningBase-v0.11.0

[jeremiahpslewis]

@JuliaRegistrator register subdir=src/ReinforcementLearningBase

[JuliaRegistrator]

Registration pull request updated: JuliaRegistries/General/78892

After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.

This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:

git tag -a ReinforcementLearningBase-v0.11.0 -m "<description of version>" b38b9f30c5750bc0d522f423d910750efb6f507b
git push origin ReinforcementLearningBase-v0.11.0

[jeremiahpslewis]

@JuliaRegistrator register subdir=src/ReinforcementLearningCore

[JuliaRegistrator]

Registration pull request created: JuliaRegistries/General/78898

After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.

This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:

git tag -a ReinforcementLearningCore-v0.9.1 -m "<description of version>" a6216b0b6e21fda86b72611228698455ce515b22
git push origin ReinforcementLearningCore-v0.9.1

[jeremiahpslewis]

@JuliaRegistrator register subdir=src/ReinforcementLearningCore

[JuliaRegistrator]

Registration pull request updated: JuliaRegistries/General/78898

After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.

This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:

git tag -a ReinforcementLearningCore-v0.9.1 -m "<description of version>" d49bdd226ccc7e89ec0ebf8eaffee245a5e94346
git push origin ReinforcementLearningCore-v0.9.1

[jeremiahpslewis]

@JuliaRegistrator register subdir=src/ReinforcementLearningEnvironments

[JuliaRegistrator]

Registration pull request created: JuliaRegistries/General/78919

After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.

This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:

git tag -a ReinforcementLearningEnvironments-v0.7.1 -m "<description of version>" bbce4f10c1863b3cae4a98f0a0b1595c34606342
git push origin ReinforcementLearningEnvironments-v0.7.1

[JuliaRegistrator]

Registration pull request created: JuliaRegistries/General/102429

Tip: Release Notes

Did you know you can add release notes too? Just add markdown formatted text underneath the comment after the text
"Release notes:" and it will be added to the registry PR, and if TagBot is installed it will also be added to the
release that TagBot creates. i.e.

@JuliaRegistrator register

Release notes:

## Breaking changes

- blah

To add them here just re-invoke and the PR will be updated.

Tagging

After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.

This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:

git tag -a ReinforcementLearningEnvironments-v0.8.8 -m "<description of version>" a982d70b55691e18ccd4051969613e37497d47bc
git push origin ReinforcementLearningEnvironments-v0.8.8

[jeremiahpslewis]

@JuliaRegistrator register subdir=src/ReinforcementLearningExperiments

[JuliaRegistrator]

Registration pull request created: JuliaRegistries/General/102430

Tip: Release Notes

Did you know you can add release notes too? Just add markdown formatted text underneath the comment after the text
"Release notes:" and it will be added to the registry PR, and if TagBot is installed it will also be added to the
release that TagBot creates. i.e.

@JuliaRegistrator register

Release notes:

## Breaking changes

- blah

To add them here just re-invoke and the PR will be updated.

Tagging

After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.

This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:

git tag -a ReinforcementLearningZoo-v0.11.0 -m "<description of version>" a982d70b55691e18ccd4051969613e37497d47bc
git push origin ReinforcementLearningZoo-v0.11.0

Also, note the warning: Version 0.11.0 skips over 0.9.0
This can be safely ignored. However, if you want to fix this you can do so. Call register() again after making the fix. This will update the Pull request.

[JuliaRegistrator]

Registration pull request created: JuliaRegistries/General/102431

Tip: Release Notes

Did you know you can add release notes too? Just add markdown formatted text underneath the comment after the text
"Release notes:" and it will be added to the registry PR, and if TagBot is installed it will also be added to the
release that TagBot creates. i.e.

@JuliaRegistrator register

Release notes:

## Breaking changes

- blah

To add them here just re-invoke and the PR will be updated.

Tagging

After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.

This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:

git tag -a ReinforcementLearningExperiments-v0.4.0 -m "<description of version>" a982d70b55691e18ccd4051969613e37497d47bc
git push origin ReinforcementLearningExperiments-v0.4.0

[jeremiahpslewis]

@JuliaRegistrator register subdir=src/ReinforcementLearningZoo

[JuliaRegistrator]

Registration pull request created: JuliaRegistries/General/102457

Tip: Release Notes

Did you know you can add release notes too? Just add markdown formatted text underneath the comment after the text
"Release notes:" and it will be added to the registry PR, and if TagBot is installed it will also be added to the
release that TagBot creates. i.e.

@JuliaRegistrator register

Release notes:

## Breaking changes

- blah

To add them here just re-invoke and the PR will be updated.

Tagging

After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.

This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:

git tag -a ReinforcementLearningZoo-v0.9.0 -m "<description of version>" a6623a85110ce5124d575329784dcd3486b2e62a
git push origin ReinforcementLearningZoo-v0.9.0

[jeremiahpslewis]

@JuliaRegistrator register subdir=src/ReinforcementLearningExperiments

[JuliaRegistrator]

Registration pull request updated: JuliaRegistries/General/102431

Tip: Release Notes

Did you know you can add release notes too? Just add markdown formatted text underneath the comment after the text
"Release notes:" and it will be added to the registry PR, and if TagBot is installed it will also be added to the
release that TagBot creates. i.e.

@JuliaRegistrator register

Release notes:

## Breaking changes

- blah

To add them here just re-invoke and the PR will be updated.

Tagging

After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.

This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:

git tag -a ReinforcementLearningExperiments-v0.4.0 -m "<description of version>" a6623a85110ce5124d575329784dcd3486b2e62a
git push origin ReinforcementLearningExperiments-v0.4.0

[jeremiahpslewis]

@JuliaRegistrator register subdir=src/ReinforcementLearningBase

[JuliaRegistrator]

Registration pull request created: JuliaRegistries/General/103658

Tip: Release Notes

Did you know you can add release notes too? Just add markdown formatted text underneath the comment after the text
"Release notes:" and it will be added to the registry PR, and if TagBot is installed it will also be added to the
release that TagBot creates. i.e.

@JuliaRegistrator register

Release notes:

## Breaking changes

- blah

To add them here just re-invoke and the PR will be updated.

Tagging

After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.

This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:

git tag -a ReinforcementLearningBase-v0.13.0 -m "<description of version>" de5893f24ff265143eec5a83a394bed57f1cb296
git push origin ReinforcementLearningBase-v0.13.0

[jeremiahpslewis]

@JuliaRegistrator register subdir=src/ReinforcementLearningCore

[JuliaRegistrator]

Registration pull request created: JuliaRegistries/General/103660

Tip: Release Notes

Did you know you can add release notes too? Just add markdown formatted text underneath the comment after the text
"Release notes:" and it will be added to the registry PR, and if TagBot is installed it will also be added to the
release that TagBot creates. i.e.

@JuliaRegistrator register

Release notes:

## Breaking changes

- blah

To add them here just re-invoke and the PR will be updated.

Tagging

After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.

This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:

git tag -a ReinforcementLearningCore-v0.15.0 -m "<description of version>" de5893f24ff265143eec5a83a394bed57f1cb296
git push origin ReinforcementLearningCore-v0.15.0

[jeremiahpslewis]

@JuliaRegistrator register subdir=src/ReinforcementLearningEnvironments

[JuliaRegistrator]

Registration pull request created: JuliaRegistries/General/103665

Tip: Release Notes

Did you know you can add release notes too? Just add markdown formatted text underneath the comment after the text
"Release notes:" and it will be added to the registry PR, and if TagBot is installed it will also be added to the
release that TagBot creates. i.e.

@JuliaRegistrator register

Release notes:

## Breaking changes

- blah

To add them here just re-invoke and the PR will be updated.

Tagging

After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.

This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:

git tag -a ReinforcementLearningEnvironments-v0.9.0 -m "<description of version>" de5893f24ff265143eec5a83a394bed57f1cb296
git push origin ReinforcementLearningEnvironments-v0.9.0

[jeremiahpslewis]

@JuliaRegistrator register

[JuliaRegistrator]

Registration pull request created: JuliaRegistries/General/103670

Tip: Release Notes

Did you know you can add release notes too? Just add markdown formatted text underneath the comment after the text
"Release notes:" and it will be added to the registry PR, and if TagBot is installed it will also be added to the
release that TagBot creates. i.e.

@JuliaRegistrator register

Release notes:

## Breaking changes

- blah

To add them here just re-invoke and the PR will be updated.

Tagging

After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.

This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:

git tag -a v0.11.0 -m "<description of version>" de5893f24ff265143eec5a83a394bed57f1cb296
git push origin v0.11.0

[jeremiahpslewis]

@JuliaRegistrator register subdir=src/ReinforcementLearningFarm

[JuliaRegistrator]

Registration pull request created: JuliaRegistries/General/103673

Tip: Release Notes

Did you know you can add release notes too? Just add markdown formatted text underneath the comment after the text
"Release notes:" and it will be added to the registry PR, and if TagBot is installed it will also be added to the
release that TagBot creates. i.e.

@JuliaRegistrator register

Release notes:

## Breaking changes

- blah

To add them here just re-invoke and the PR will be updated.

Tagging

After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.

This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:

git tag -a ReinforcementLearningFarm-v0.0.1 -m "<description of version>" de5893f24ff265143eec5a83a394bed57f1cb296
git push origin ReinforcementLearningFarm-v0.0.1

[jeremiahpslewis]

So, this is done. Thanks for the great teamwork @HenriDeh!!!

[HenriDeh]

It was all you! I wish I had more time to allocate to RL, but I must focus on something else at the moment.