[OSPP-Week4 & 5] Start to implement new Multi-agent Algorithm(maddpg)

[peterchen96]

Mission for this week

• Fix possible bugs in the implementation of NFSP.
• Consider implementing a new multi-agent algorithm in RLZoo.jl:
  • Recently, I start to read the paper Computing Approximate Equilibria in Sequential Adversarial Games by Exploitability Descent which proposed Exploitability Descent(ED) algorithm.
  • I'll try to create a reading report and a rough implementation idea for this algorithm in this discussion on the weekend.
  • Or maybe other suggestions for a new algorithm? @findmyway

[findmyway]

Could you implement MADDPG first? It would be great if you can add it into RL.jl in the next two weeks so that you can cooperate with @Sid-Bhatia-0 before the end of GSoC to provide a non-trivial benchmark for those environments he had implemented..

[peterchen96]

Thanks for your suggestion! I'll read the paper Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments these days and try to have a rough realization idea on the weekend.

[Sid-Bhatia-0]

There is CollectGemsMultiAgentUndirected from the GridWorlds.jl package that provides a simple multi-agent environment. @peterchen96 let me know if you need any help understanding or modifying this environment.

[peterchen96]

@Sid-Bhatia-0 Thanks for your reminder! I’ll ask you if I have any questions about the environment. 😄

[peterchen96]

Idea about MADDPG.

From my view, each agent's network structure and its update method in the MADDPG is largely the same as DDPG, besides the following point(or hypothesis):

• each agent's critic can know other agents' policies, including their current personal information states and actions. So the network's input includes all agents' state information and actions.

Implementaton about MADDPG.

Based on the idea, the MADDPG can be constructed like the following:

agent = Agent(
    policy = DDPGPolicy,
    trajectory = CircularArraySARTTrajectory,
    )
agents = MADDPGManager(
    Dict((player, deepcopy(agent)) 
        for player in players(env) if player != chance_player(env)),
    batch_size,
    update_freq,
    step_counter,
    rng
    )

Where each agent 's trajectory collects its own information in the self-play process, and each critic will assemble all agents' trajectory when updating its own network.

Attempt the algorithm on KuhnPoker.

Here I implement the simplest version of MADDPG, which only works for simultaneous games with discrete action space. However, the continuous action space version can add a space type judgment when output the actions just like the following(I don't know how to judge one action space whether it is discrete 😞 ).

function (π::MADDPGManager)(env::AbstractEnv)
    while current_player(env) == chance_player(env)
        env |> legal_action_space |> rand |> env
    end
    if is_discrete(action_space(env)) # how to judge the type of the env's action space?
        Dict((player, ceil(agent.policy(env))) for (player, agent) in π.agents)
    else
        Dict((player, agent.policy(env)) for (player, agent) in π.agents)
    end
end

And I attempt this implementation on the KuhnPoker game(#444). Since the original KuhnPoker is a sequential game, I use ActionTransformedEnv to generate a dummy action for the other agent. Here use the reward of player 1 to evaluate the performance. Since the algorithm is deterministic, nash_conv may not be suitable here.

Asking for suggestions...