GCN-SOAN: Improving Peer Assessment with Graph Convolutional Networks

Table of Contents:

• Introduction
• Baselines
• How to Run Experiments

Introduction

GCN-SOAN is a semi-supervised graph convolutional network, leveraging the power of GCNs and the flexibility of multi-relational weighted networks, coupled with experts' evaluations as ground truths.

Baselines

1. Average: Take the average of the peer assessments to estimate ground-truth valuations.
2. Median: Take the median of the peer assessments to estimate ground-truth valuations.
3. PeerRank: The PeerRank model is an iterative model motivated by the Google PageRank algorithm. In PeerRank, the grade of a user relies on the grade of its graders, and the grade of the graders also relies on the grade of other graders.
4. TunedModel: In TunedModels, the authors formulate peer assessment as a probabilistic model by defining users' true score, graders' bias, and graders' reliability with peer grades as the only inputs of the model. They framed the probabilistic model as a posterior distribution problem over the latent variables given on observed peer grades.
5. RankwithTA: RankwithTA is an iterative algorithm that incorporates experts' grades as ground truth for a portion of the submissions. In RankwithTA, the final grade of each user depends on both their performance in answering their own submission and the quality of their peer assessments.
6. Vancouver: Vancouver is a crowdsourcing algorithm that aggregates peer grades leveraging a reputation system that captures users' grading accuracy. The Vancouver algorithm is motivated by expectation maximization (EM) techniques and uses the variance minimization principle to estimate students' performances in their grading ability to control the influence each grader exerts on the final consensus grade of each submission.

How to Run Experiments?

Required Packages

The easy/recommend way: We recommend to use a virtualenv to avoid conflicting with your previously installed packages. To create a virtualenv and install all required packages use the following commands:

1. Navigate to the repository folder using cd GCN-SOAN
2. Create a new virtual environment for python: virtualenv gcn_soan_env
3. Activate the created virtual environment: source gcn_soan_env/bin/activate
4. Install required packages: ./setup.sh in Linux OS, and ./setup_windows.sh in Windows OS

* Make sure to have execute permission for setup.sh file: chmod +x setup.sh

* If you get any error during installing Pytorch on your system, the Pytorch community is the best place to look for: Pytorch Community

The hard way: Install the following packages separately using pip

matplotlib==3.3.4
networkx==2.5.1
numpy==1.19.5
pandas==1.1.5
scikit-learn==0.24.2
scipy==1.5.4
torch==1.8.2+cu102
torch-cluster==1.5.9
torch-geometric==2.0.1
torch-scatter==2.0.9
torch-sparse==0.6.12
torch-spline-conv==1.2.1
torchaudio==0.8.2
torchvision==0.9.2+cu102

Run all studies with one command

1. Give execute permission for run_studies.sh file: chmod +x run_studies.sh
2. Run the bash script: run_studies.sh
3. The results of simulated studies will be saved in data/simulated_studies/study*/*.pdf
4. The results of real-world studies will be saved in data/real_data_processed/rmse_models*.csv

Run simulated studies in paper

There are 6 studies conducted in the paper, each investigating the impact of one factor on the results. To run each of them separately:

1. First you need to update the set_study.py file with a unique identifier defined for each study.
2. Then you should run: python3 main.py

For example, to run the first study which examines the impact of number of graders per item, you should update the set_study.py file with: study = 'number_of_graders' and then run: python3 main.py

Run a customized simulated study

1. First you need to update the set_study.py file with a unique identifier called "custom".
2. Then, choose your desired parameters in set_study.py for "custom_study" variable.
3. Finally, you should run: python3 main.py

The above command generates corresponding synthetic datasets, run GCN-SOAN and all baselines, and save the results in data/studies directory. The plots also will be generated and saved in pdf file in the corressponding directory.

Run experiments on real data

1. The dataset is not publicly available, so you should request it from the following link: Request for real dataset
2. You need to locate the 4 main files of the dataset containing peer grades, self grades, TA grades, and max grades in data/real_data_raw directory.
3. Set the study in set_study to the desired experiment you want, it can be one of the followings:
   real_data_peer_evaluation
real_data_peer_and_self_evaluation
4. Run python3 main.py
5. The results will be saved in the following file: data/real_data_processed/rmse_models.csv

List of unique identifiers for studies:

• Study 1 - Number of graders per item: The unique identifier for this study is "number_of_graders"
• Study 2 - Bias parameter: The unique identifier for this study is "graders_bias"
• Study 3 - Ground truth mean: The unique identifier for this study is "ground_truth_distribution"
• Study 4 - Reliability parameter: The unique identifier for this study is "working_impact_grading"
• Study 5 - Strategic peer grade generation: The unique identifier for this study is "erdos"
• Study 6 - Homophily: The unique identifier for this study is "homophily"
• Study 7 - Real data peer evaluation: The unique identifier for this study is "real_data_peer_evaluation"
• Study 8 - Real data peer and self evaluation: The unique identifier for this study is "real_data_peer_and_self_evaluation"
• Study 9 - Custom: The unique identifier for this study is "custom"

Cite us

Arxiv link: https://arxiv.org/abs/2111.04466

@misc{namanloo2021improving,
  title={Improving Peer Assessment with Graph Convolutional Networks}, 
  author={Alireza A. Namanloo and Julie Thorpe and Amirali Salehi-Abari},
  year={2021},
  eprint={2111.04466},
  archivePrefix={arXiv},
  primaryClass={cs.CY}
}