On the Effectiveness of Parameter Efficient Methods: Sparsity and FFNs Matter

Repo for preprint paper .

Environment

Here are the steps to create the environment from scratch. We presuppose the installation of conda.

# create conda env hgf, change name if needed
conda env create -n hgf python=3.8.13 -y
conda activate hgf
# install packages
conda install pytorch==1.11.0 cudatoolkit=11.3 -c pytorch
pip install transformers==4.22.0 datasets==2.1.0
# for easy debugging
# pip install debugpy==1.6.0
# it is also applicable to install packages from `requirements.txt`
# pip install -r requirements.txt

GLUE tasks

Based on the script run_glue.py.

Fine-tuning the library models for sequence classification on the GLUE benchmark: General Language Understanding Evaluation.

Main Results

MRPC Results

GLUE Results

Scale-up Results

Usage

Useful scripts are under the scripts directory. Sample usage with respect to adapters is as follows. Scripts for other parameter efficient methods have similar usage. Each script is highly customizable. For help on arguments defined in the script, please resort to the respective argument file under argument_classes.

bash scripts/adapter_hyper.sh