FINAL YEAR PROJECT - NTU EEE

• By Tran Thuy Dung
• Supervisor: Assoc. Prof. Andy Khong :D

Project Title: Automatic Question Generation with Natural Language Processing

NTU EEE FINAL YEAR PROJECT

Automatic Question Generation with Natural Language Processing

by Tran Thuy Dung

Table of Contents

1. About The Project
   • Built With
2. Getting Started
   • Installation
3. Usage
4. Model Performance
5. Contact
6. Acknowledgments

About The Project

This project uses transfer learning to train a T5 model for Automatic Question Generation (AQG). A React Web Application with a SQLite backend is also created to demo the model!

Main project objectives:

• Generate a model pipeline to train T5 for AQG with Pytorch and SQuAD data
• Frontend web application with React
• Backend with FastAPI, SQLAlchemy and Pydantic

(back to top)

Built With

Major frameworks/libraries used:

• React.js
• Bootstrap
• JQuery
• FastAPI
• Pytorch
• SQLalchemy
• Pydantic

(back to top)

Getting Started

Below are the steps to run the scripts. There are 3 main folders:

• /model: contains the pipeline for the model, which can be run on a terminal
• /fyp-app: frontend scripts for the React app
• /backend: backend scripts containing the API and the database utils.

(back to top)

Installation

Clone this repo

git clone [email protected]:yoongtr/fyp-scripts.git

Create a conda environment and install necessary libraries. Navigate to the root folder and run:

• conda/pip environment

  conda create -yn aqg-app python=3.7 # create a new environment
  conda activate aqg-app
  conda install --file requirements.txt
  pip install -r requirements.txt

The requirements.txt file stores the libraries and corresponding versions for the repo.
To run the app:

• To run the backend: navigate to /backend

  cd backend
  uvicorn main:app --reload # uvicorn watches changes and reload the app when neccessary

• After backend is inititalised, navigate to frontend folder to run the frontend app:

  cd ../fyp-app
  npm install # install neccessary npm packages
  npm start # start the web server

The frontend runs at http://localhost:3000/home while the backend is at http://0.0.0.0:8000/docs#. Open these links in the browser.

(back to top)

Usage

Model pipeline

Training model or inference from terminal:

• SQuAD v2.0 is used as training data. The context and answer are treated as input, and the question as output.
• During training, the answer is given. However for inference, only the context is given and an answer is extracted from the context by T5 for summarisation.
• Training pipeline:
• Inference pipeline:
• Navigate to /model
• To create JSON data from SQuAD-like format that can be read by the pipeline:

    python create_eval_data.py # change the file names and paths accordingly

• To train the model. The trained model is saved as .ckpt file.

    python pipeline.py train path-to-processed-json-data checkpoints best-checkpoint t5-base epochs-no batch-size learning-rate

for example

    python pipeline.py train ../SQuAD-2.0/train-v2.0.json checkpoints best-checkpoint t5-base 10 4 0.0001

• Inference: infer-with-file parses a txt file, infer-with-ans requires answer input, and infer-without-ans does not require answer input.

python pipeline.py infer-without-ans t5-base 20220113/best-checkpoint.ckpt # Change path to ckpt file accordingly

• For evaluation, there are two types of dev sets for inference - one is the original dev set from SQuAD 2.0 and the other one is the Xinya Du's extracted SQuAD. I already generated the two files as squad_for_eval.json and squad_du_for_eval.json respectively. Use this repo and follow its instructions to evaluate inference texts.

(back to top)

Quiz Generation Web App

• The full framework for the webapp is as below

(back to top)

• Once the app runs, you should see a home page like this where a paragraph can be submitted and a question + answer key is generated:

• Clicking into "Sign Up" or "Log In" pages allows you to see more functions at the homepage. Texts can be submitted to create a quiz.

• Going into My Account allows you to see the user information, a leaderboard and past quizzes.

(back to top)

Model performance

• I used Du's SQuAD dataset to evaluate my model and metric scripts from https://github.com/tuetschek/e2e-metrics.
• Below is a comparison between my model and other models (other models scores are from Lopez et al.'s paper)

Tables	Answer Provided?	BLEU-4	METEOR	ROUGE_L
Du et al. (2017)	No	12.28	16.62	39.75
Du and Cardie (2018)	Yes	15.16	19.12	-
Zhao et al. (2018)	No	4.8	12.52	30.11
Zhao et al. (2018)	Yes	16.38	20.25	44.48
Dong et al. (2019)	Yes	22.12	25.06	51.07
Lopez et al. (2021)	No	8.26	21.2	44.38
My model (T5 summarization +AQG)	No	16.37	20.4	41.50

(back to top)

Contact

Yoong (Thuy Dung) - [email protected] - [email protected]

Project Link: https://github.com/yoongtr/fyp-scripts

(back to top)

Acknowledgments

Relevant papers

• Attention is all you need: Proceedings of the 31st International Conference on Neural Information Processing Systems
• Learning to ask: Neural question generation for reading comprehension
• Harvesting paragraph-level question-answer pairs from Wikipedia
• Paragraph-level neural question generation with Maxout pointer and gated self-attention networks
• Unified language model pre-training for natural language understanding and generation
• Simplifying paragraph-level question generation via transformer language models

(back to top)