ChanceStyle is a little confusing

[ShuhuaGao]

In the Guide, there is

If there's no rng in the environment, everything is deterministic afer taking each action, then we call the ChanceStyle of these environments are of Deterministic. Otherwise, we call them Stochastic

I am confused about the meaning of ChanceStyle.

• Does ChanceStyle only apply to multi-agent RL? Since the documentation of ChanceStyle always mentions a game and a chance player.
• In the above Guide, there is no rng field in the struct LotteryEnv, but the ChanceStyle shown therein is Stochastic().
• Consider a single-agent RL. If the state transition is deterministic, that is, given (s, a), the resultant (s', r) is fixed, what should the ChanceStyle be according to your definition? Such deterministic transition is commonly assumed in practice, like a robot.
• Does an enviroment's ChanceStyle matter much? Possibly some algorithms will make use of it.

[findmyway]

In the Guide, there is

If there's no rng in the environment, everything is deterministic afer taking each action, then we call the ChanceStyle of these environments are of Deterministic. Otherwise, we call them Stochastic

I am confused about the meaning of ChanceStyle.

• Does ChanceStyle only apply to multi-agent RL? Since the documentation of ChanceStyle always mentions a game and a chance player.

From the API designer's level, no the ChanceStyle can be applied to both single and multi agent environments. But in this package, we mainly used it in multi-agent environments.

• In the above Guide, there is no rng field in the struct LotteryEnv, but the ChanceStyle shown therein is Stochastic().

Nice catch! Actually we used the Random.GLOBAL_RNG in that game. I didn't add the rng field in the definition of that game on purpose to keep that example simple enough for first time users. But yes, to obey the definition of interfaces strickly, we should add a rng in that game.

• Consider a single-agent RL. If the state transition is deterministic, that is, given (s, a), the resultant (s', r) is fixed, what should the ChanceStyle be according to your definition? Such deterministic transition is commonly assumed in practice, like a robot.

Then just define it as DETERMINISTIC

• Does an enviroment's ChanceStyle matter much? Possibly some algorithms will make use of it.

Until now, only some multiaget algorithms require that the games are deterministic or explicit stochastic.

[findmyway]

I added some checks here:

ReinforcementLearning.jl/src/ReinforcementLearningBase/src/base.jl

Lines 161 to 165 in 196f02a

	if ChanceStyle(Y) ∉ (DETERMINISTIC, EXPLICIT_STOCHASTIC)
	s = 888
	Random.seed!(Y, s)
	Random.seed!(X, s)
	end

These checks are optional though.

[ShuhuaGao]

But I do not see where this field rng is ever checked for ChanceStyle

I actually mean with the above words that there is no check like hasfield(TEnv, :rng). Did you refer to the existence of the field rng when saying

If there's no rng in the environment, everything is deterministic afer taking each action, then we call the ChanceStyle of these environments are of Deterministic. Otherwise, we call them Stochastic

[findmyway]

What I mean by the rng here is more general than just a hasfield(Env, :rng) check. In some cases, the environment is just a wrapper and does not have such a rng field. But semantically, environments of STOCHASTIC chance style should have implemented the Random.seed! interface to allow setting the random seed.

[ShuhuaGao]

That makes sense. In that case, the documentation may better state that a dispatch of ChanceStyle(::Type{<:AbstractEnv}) should be provided for a custom environment if ChanceStyle matters for that environment.

[findmyway]

Fair enough, I update that section slightly 3ccf6ed . But feel free to make further updates.