LLMs for Extremely Low-Resource Finno-Ugric Languages

This repository contains the implementation used for training and evaluating language models for extremely low-resource Finno-Ugric languages.

Models

Pre-trained:

• tartuNLP/Llama-SMUGRI-7B

Instruction-tuned:

• tartuNLP/Llama-SMUGRI-7B-Instruct-MTI (SupInst+TrAlpaca)
• Llama-SMUGRI-7B-Instruct-MTI-Tr (SupInst+TrAlpaca+TrInst)
• tartuNLP/Llama-SMUGRI-7B-Instruct-LLMTI (SupInst+LLMTrAlpaca)
• tartuNLP/Llama-SMUGRI-7B-Instruct-LLMTI-Tr (SupInst+LLMTrAlpaca+TrInst)

Evaluation

Belebele-SMUGRI:

• https://huggingface.co/datasets/tartuNLP/belebele-smugri

SIB-SMUGRI:

• https://huggingface.co/datasets/tartuNLP/sib-smugri

Usage

Scripts for launching training are provided in:

• scripts/training

LM-eval-harness configurations:

• scripts/evaluation/lm_eval_harness_configs

Citation

@misc{purason2024llmsextremelylowresourcefinnougric,
      title={LLMs for Extremely Low-Resource Finno-Ugric Languages}, 
      author={Taido Purason and Hele-Andra Kuulmets and Mark Fishel},
      year={2024},
      eprint={2410.18902},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2410.18902}, 
}

Acknowledgements

The implementation is built on github.com/TartuNLP/llammas.