Replace step! with act! and observe

[zsunberg]

After discussion in #20, I'm now leaning toward having act! and observe to replace step! since it makes significantly more sense in a 2-player environment.

Please weigh in!

If accepted, this will

• close Support for multiple players games #20
• close observe(env) #12
• close Functions that depend on the environment state vs functions of the environment itself #25 (by adding statements to the docstrings)

[zsunberg]

@jonathan-laurent I can't request a review from you since you are not a JuliaReinforcementLearning member, but I think it would be good for you to weigh in since this was prompted by #20

[jonathan-laurent]

@findmyway Could you please add me to JuliaReinforcementLearning?

[findmyway]

Looks good to me. My only concern is the name of observe.

First, apologize if my comments in #12 (comment) and JuliaReinforcementLearning/ReinforcementLearning.jl#68 (comment) caused any confusions there. Inspired by all your discussions, I'm experimenting with a wild idea to remove observe(env) (at least in single agent environment). So let's forget those discussions temporarily.

From the example here, the observe function returns a state. But I guess it is assumed to return an observation in POMDP environments? Then how do we get the inner state in such environments? (With a state(env)?) And I think @jonathan-laurent mentioned another two functions set_state!(env, state) or Env(state) somewhere. So I think people with different backgrounds may have a different understanding of state or observation. We'd better to clearly define these concepts explicitly somewhere in the doc.

[findmyway]

@findmyway Could you please add me to JuliaReinforcementLearning?

It seems you are already in 😉

[jonathan-laurent]

Should we consider only returning the reward or is this for a next PR?

[zsunberg]

Right - I'll change it depending on the outcome of #20

[jonathan-laurent]

Maybe we should insist that valid_action_mask must be defined with respect to actions(env) and not actions(env, player(env)).

Also, is there a way to make implementing actions(env) mandatory but actions(env, player) optional?

[zsunberg]

Yes, currently actions(env) is mandatory, but actions(env, player) is optional.

[zsunberg]

We'd better to clearly define these concepts explicitly somewhere in the doc.

Yes. Documentation #24 will be important after we get the rest of the basic questions about the interface answered.

Informally, the definitions are as follows:
observe returns all of the information given to the agent on that step to make decisions.
state returns the Markov state of the environment. The behavior of two environments with the same static parameters that start from the same state should be statistically identical.

We may want a trait that indicates whether the environment is fully observable, in which case observe and state return the same information.