Comparative Analysis of Classical and Quantum Reinforcement Learning Algorithms

Description of Project (Abstract)

The development of quantum computational techniques has advanced greatly in recent years alongside the continued growth within the field of artificial intelligence. These improvements have paved the way for their conjunction into a subfield called quantum reinforcement learning to explore innovative solutions. Reinforcement learning is an area of artificial intelligence that focuses on refining the learning process of agents, which are entities that interact with their surroundings to identify and adopt the most optimal strategies.

This work explores the potential of quantum computing to facilitate the inefficiencies of the traditional RL algorithms, specifically focusing on $\epsilon$-Greedy Q-learning and its issue with slow learning convergence rates. Quantum computing offers an improvement to existing classical computation solutions due to its leveraging of quantum mechanics. This project implements variational quantum circuits, which is a form of quantum machine learning, by developing a hybrid quantum-classical algorithm. It investigates the enhancement of the decision-making capabilities of an agent within two environments, Frozen Lake and Maze World, that are designed.

The aim is to achieve a faster convergence to the optimal learning policy with the application of the hybrid algorithm. The results indicate a speed-up in the convergence of rewards across the tested environments, including a reduction in the average steps taken to reach the goal by around 20% with the implementation of the designed VQA. The success of this work displays a great potential to improve and build upon the relationship between the domains of quantum and classical reinforcement learning.

Dependencies and Installations

To run this project ensure you have Python installed on your system. This project is built with Python 3.8 or above.

For in-built algorithm metrics collection and display simply install:

• matplotlib 3.8.1

For the scripts with quantum Q-learning algorithm make sure you have Qiskit installed.

• qiskit 0.45.1
• qiskit-aer 0.12.1
• qiskit-algorithms 0.3.0

Algorithms and Environments

Main Project Execution: The primary project file is the quantum_qlab.py file. Execute this file to select and run the desired environment and algorithm.

Maze World Classical Q-learning: The maze.py file generates a random maze where an agent applies the classical Q-learning algorithm to find a solution.

Frozen Lake Environment: The frozen_lake.py supports both classical and quantum algorithms but lacks a built-in UI. Users must manually select the desired algorithm at the end of the file.

Variational Quantum Algorithm: The VQA.py file contains the implementation of the Variational Quantum Algorithm, which is utilized in the main quantum_qlab.py file.

Testing

The testing conducted with Tensorboard, to run the scripts yourself and see the algorithm results ensure you have installed:

• tensorboard 2.15.2
• tensorboard-data-server 0.7.2
• tensorboard-plugin-wit 1.8.1
• tensorflow 2.15.0

The files have the testing lines commented for ease of the user too, you can simply uncomment them and run to see results on Tensorboard.

Logs of the tests conducted are included under the directory - Test Logs Directory.

This directory has the collected metrics over different hyperparameter values for the classical RL agent, and the hybrid quantum-classical RL agent.

Lastly the tests that were conducted specifically for the traditional Q-learning algorithm are under classical_experiments directory.

Acknowledgements

This project used environments based on Maze Grid World and Frozen Lake, which have been altered accordingly; however, the main code used from the two environments are:

• Solving Frozen Lake using Q-learning from scratch on Kaggle
• Quantum Reinforcement Learning Tutorial on YouTube