[OSPP-Week1] Make the design of core components of Offline Reinforcement Learning algorithms.

[Mobius1D]

@findmyway
My plans for this week are the following:

1. To implement an Offline RL algorithm using JuliaRL components so that I would be able to understand the pipline in a better manner.
2. A simple version of CQL or any other algorithm after discussion with @pilgrimygy will be implemented and tested using standard experiments. (this is just for testing purposes). The actual implementation will be given as a PR in the coming week.
3. To study different RL libraries like rllib and rlkit to understand the implementation and the components required.
4. Provisions for loading datasets into trajectories will be made using a wrapper for d4rl and d4rl-pybullet.
5. The main components of the library will be finalised by this week so that they can be built effectively in the coming weeks. A report will be made in the form of a discussion over the course of the week so that all ideas are taken into account for getting the big picture.

[pilgrimygy]

Dear, I think that CQL is very simple algorithm. In fact, it only defines a regularizer to make Q-value conservative. In my opinion, it can be defined as a common component, which can be used in various other algorithms. I prefer to command discrete BCQ as a benchmark.
@findmyway What do you think?

[pilgrimygy]

I have completed a preliminary version for reference.

[Mobius1D]

Great. It can be a component itself. I am implementing it for continuous cases and also the goal is to understand the pipeline.

Would be great if you could DM your preliminary version or make a github repo.

I will be posting my understanding of the components after I implement one myself so that we can brainstorm and make the basic design ready. Just busy with a few things for now but will get the design within a couple of days.

[pilgrimygy]

It seems like:

"""
    calculate_CQL_loss(q_value, action; method)
See the paper: [Conservative Q-Learning for Offline Reinforcement Learning](https://arxiv.org/abs/2006.04779)
"""
function calculate_CQL_loss(q_value::Matrix{T}, action::Vector{R}; method = "CQL(H)") where {T, R}
    if method == "CQL(H)"
        cql_loss = mean(log.(sum(exp.(q_value), dims=1)) .- q_value[action])
    else
        @error Wrong method parameter
    end
    return cql_loss
end

Original Code

[pilgrimygy]

When most algorithms calculate loss, this function can be additionally called to calculate CQL loss.

[Mobius1D]

Thanks. Got it. There will be some subtle variations for policy gradients tho. Not that much tbh, we would have to pass in the policy and calculate the actions and pass it into the q function update.