Add ConvLSTM2D instead of Conv2D layer for better performance?

[talentoscope]

I've been very interested in this project, playing with it locally.
I think it would be helpful to make at least one of the layers (perhaps all of the Conv2D ones) Convolutional 2D LSTM layers. I believe this will help find temporal-spacial relationships within the game screen better and result in better network models.

[Naereen]

Hi,

Thanks for the suggestion! Yes it could, possibly, who knows…
Could you try and see if it gives any improvement on the performance? (see keras documentation)

So far, all my experiments have been huge failures, with up-to 4 days of training leading to zero improvement over the random policy.

[Naereen]

I got an error while trying to replace Conv2D with ConvLSTM2D without any other modification, and I have the feeling that it won't be trivial to do the change.
Please try on your own, and submit a PR if you succeed (I won't have time to experiment right now).

FYI:

Traceback (most recent call last):
  File "run-mario.py", line 204, in <module>
    mario_main(N=PARALLEL_EMULATORS)
  File "run-mario.py", line 127, in mario_main
    name=dqn_model_name
  File "/home/lilian_besson/gym-nes-mario-bros.git/src/dqn/model.py", line 164, in __init__
    self.base_model = q_model(input_shape, num_actions)
  File "/home/lilian_besson/gym-nes-mario-bros.git/src/dqn/model.py", line 107, in q_model
    inputs, outputs = q_function(input_shape, num_actions)
  File "/home/lilian_besson/gym-nes-mario-bros.git/src/dqn/model.py", line 80, in q_function
    out = ConvLSTM2D(filters=32, kernel_size=8, strides=(4, 4), padding=padding, activation='relu')(image_input)
  File "/usr/local/lib/python3.5/dist-packages/keras/layers/convolutional_recurrent.py", line 277, in __call__
    return super(ConvRNN2D, self).__call__(inputs, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/keras/layers/recurrent.py", line 499, in __call__
    return super(RNN, self).__call__(inputs, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/keras/engine/topology.py", line 575, in __call__
    self.assert_input_compatibility(inputs)
  File "/usr/local/lib/python3.5/dist-packages/keras/engine/topology.py", line 474, in assert_input_compatibility
    str(K.ndim(x)))
ValueError: Input 0 is incompatible with layer conv_lst_m2d_1: expected ndim=5, found ndim=4

[Nindaime]

The LSTM algorithm learns patterns from data across time, this means the LSTM aims to understand the relationship between data from one instance in time and another instance, while still learning relationship between inputs and output. To achieve this, the LSTM needs a time step which is bassically grouping rows of your dataset and feeding to the neural network. So you will need to reshape your dataset which current is in 4 dimensions to 5 dimensons which is what the last line of the error means, the ValueError

Assuming your dataset shape is as follows:
dataset.shape
=> (1023, 4, 28, 28)

def reshape_input(data, time_steps):
    # Converting dataframe to numpy array
    if isinstance(data, pd.DataFrame):
        data = data.values
    
    # Reshape the data into a 5D array
    num_samples = data.shape[0] 
    image_width = 28
    image_height = 28
    grouping = 4
    
    # Define empty numpy array 
    input = np.zeros((num_samples, time_steps, grouping, image_width, image_height))
    for i in range(num_samples):
        if i+time_steps < num_samples:
            input[i, :, :, :, :] = data[i:i+time_steps, :, :, :]
            
    return input

# Reshape the dataset with time step of 6
dataset = reshape_input(dataset, 6)
dataset.shape

=> (1023, 6, 4, 28, 28)

You may also employ the numpy reshape method but I prefer it this way

[Naereen]

Hi @Nindaime,
Thanks for your suggestion.
As you might have seen, this project hasn't been updated in the last... 5 years! And I don't plan on updating it, sorry.
I'll leave your interesting comment, if someone ever wants to explore this project in this direction.
Regards.