How about making this package compatible with DifferentialEquations.jl?

[JinraeKim]

According to this blog, the simulation flow of ReinforcemetLearning.jl is very flexible by calling hooks at any stages as follows.

To me, it looks like callbacks in DifferentialEquations.jl.
How about making this package based on DifferentialEquations.jl?
Although it may cause high dependencies, it would provide extensive functionality and extensibility for continuous time simulation (including ODE, SDE, RODE, and so on). Also, DifferentialEquations.jl can handle discrete time simulations.
In addition, DifferentialEquations.jl is a widely used package for scientific machine learning. For example, it is also compatible with other ML packages, for example, see DiffEqFlux.jl for neural ODE.
Note that this proposal might be biased for continuous-time simulation of dynamical systems (as I'm interested in them).

(Note)
I'm not sure what an appropriate form of Base.run would be for continuous-time simulation cuz there might be no difference between policy(PRE_ACT_STAGE, ...), env(action), and policy(POST_ACT_STAGE, ...) (they would be integrated into one callback). Probably the action is injected into the simulator as parameters, e.g., parameterised simulation within one time step.
(Note2)
Of course, there is an alternative way; custom usage of DifferentialEquations.jl only for env(action) (that is, system propagation).

[00krishna]

I think the idea of a closer connection between DifferentialEquations.jl and ReinforcementLearning.jl is a good idea. If there were some easier helper functions to work with Diffeq types, or perhaps if there was an interface built into ModellingToolkit to build up a Reinforcement Learning problem. I am not sure the best approach.

The algorithms for RL are very different than those from Diffeq, so not sure how easy it would be to connect them. However there are other nice things already created in Diffeq, such as the setup of types for CUDA and parallelization, etc. So it is something to consider.

[findmyway]

I'm not very familiar with DifferentialEquations.jl. So maybe @JinraeKim have some words on it.

[JinraeKim]

Thanks for calling me, @findmyway.
I'm not an expert on DiffEq.jl and RL.jl neither a developer so I'm little bit worried about giving advice :)
Here's my opinion:

• Advantage of DiffEq.jl
  DiffEq.jl is merely a solver for various DEs (differential equation). For example, ordinary DE (ODE), stochastic DE (SDE), etc.
  So I prefer to use DiffEq.jl as dynamical systems I usually consider are in continuous-time settings.
  I used to try to use RL.jl for my research but now I usually use DiffEq.jl for this reason.

• How to integrate them?
  For RL.jl, it may be possible to use DiffEq.jl for propagating system dynamics (transition).
  I've heard that DiffEq.jl also supports discrete-time transitions.
  Also, there are many callbacks from DiffEq.jl.
  To preserve the callbacks of RL.jl, there should be an intermediate interface bridging the callbacks from the two packages.

• Some worries

1. Discrete-time dynamical systems
   I'm a bit worried about the implementation of discrete-time dynamics.
   First, in RL.jl, there are callbacks applied "before and after when action is applied". It would require some intermediate interface as I mentioned above. Also, I barely perform discrete-time dynamical system simulation with DiffEq.jl so I'm not sure the functionality of DiffEq.jl is enough to cover the requirements of RL.jl.
2. The way dealing with "state"

• To say, DiffEq.jl works as "separating state from environments". In other words, one would define DE and inject the initial condition of state (namely, x0) and the solver gives a trajectory x(t) (or, x(t) where t in ts for specified time instants). On the other hand, RL.jl takes a standard form of reinforcement learning setting: define an environment, and the env object "contains its state". I think that it's just a difference of concept. For RL researchers, current RL.jl's approach would be much familiar. If RL.jl takes DIffEq.jl as a backend, there would be some choices including "providing an interface for keeping current concept".

[00krishna]

@findmyway @JinraeKim good points here. I suppose there are a few points to include. I think you all have done exceptional work in building up the package and such so congrats on a job well done.

I think that there is a compelling case for closer integration with the SciML julia organization because there is a lot of opportunity for integration and reducing the implementation of the same algorithms.

So I have some familiar with the DifferentialEquations.jl and related ModellingToolkit.jl and DiffEqFlux.jl packages. These libraries provide solvers for ODEs and PDEs as @JinraeKim said. So that is all fine. You can include an ODE in an RL environment and then use that to solve the MDP, etc. BUT, there is a lot lot more to these packages. The DiffEqFlux package is as well as ModellingToolkit.jl are working on implementing a lot of tools from control theory, so they are solving a lot of classical and modern state space control problems using Flux and neural networks. So solving a control problem using a neural network is essentially an RL algorithm. There are a lot of tools in there for developing custom adjoints and adjoint sensitivity analysis, which is also another control and RL strategy. So it probably makes a bit of sense to look at the intro tutorial.

In terms of @JinraeKim's question about discrete dynamics, yes I personally use DifferentialEquations.jl for a lot of modelling for discrete dynamics. So all of that is there. I can simulate random walks, biased random walks, jump processes, different noise processes, etc. But the nice thing is that all of this can be integrated back into the adjoint sensitivity analysis for estimating parameters from data, or solving control problems.

My basic point is that there is a lot of connection between ReinforcementLearning.jl and all of the SciML ecosystems. So it is good to perhaps talk to those guys and figure out compatibility between packages. I think you will learn a lot in the process. Also, you should certainly watch the SciML talks at the current JuliaCon that is starting today :).

[findmyway]

Also, you should certainly watch the SciML talks at the current JuliaCon that is starting today :).

Great suggestion. I think I may also need to have a discussion with @YingboMa at some time.