Archive Benchmark System

Benchmark System Documentation

This page documents how to use the TranslateBookWithLLM benchmark system to test LLM translation quality.

Quick Start

# Run quick benchmark (19 languages)
python -m benchmark.cli run --openrouter-key YOUR_KEY

# Run full benchmark (40+ languages)
python -m benchmark.cli run --full --openrouter-key YOUR_KEY

# Publish results to this wiki
python -m benchmark.cli wiki-publish

Prerequisites

Requirement	Description
Ollama	Running locally with at least one model installed
OpenRouter API Key	For translation evaluation (get one here)
Git	For publishing to the wiki

CLI Commands Reference

Running Benchmarks

python -m benchmark.cli run [OPTIONS]

Option	Description
-m, --models	Models to test (auto-detects if omitted)
-l, --languages	Language codes to test (e.g., fr de ja)
--full	Test all 40+ languages
--openrouter-key	OpenRouter API key
--evaluator	Evaluator model (default: anthropic/claude-haiku-4.5)
--ollama-endpoint	Custom Ollama URL
--resume	Resume interrupted run by ID

Managing Results

Command	Description
list	Show all benchmark runs
show RUN_ID	Display run details (-d for detailed stats)
export RUN_ID	Export to CSV (-o path)
delete RUN_ID	Delete a run (-f to skip confirmation)
merge RUN_IDs	Combine runs (--publish to auto-publish)

Publishing

Command	Description
wiki	Generate wiki pages locally
wiki-publish	Generate and push to GitHub wiki

Test Languages

Quick Benchmark (19 languages)

Category	Languages
European	French, German, Spanish, Italian, Portuguese, Polish
Asian	Chinese (Simplified/Traditional), Japanese, Korean, Vietnamese, Thai
South Asian	Hindi, Bengali, Tamil
Cyrillic	Russian, Ukrainian
Semitic	Arabic, Hebrew

Full Benchmark (40+ languages)

Adds: Dutch, Swedish, Danish, Norwegian, Finnish, Greek, Romanian, Hungarian, Czech, Indonesian, Malay, Filipino, Bulgarian, Serbian, Latin, Ancient Greek, Sanskrit, Welsh, Basque, Catalan, Galician, Irish, Scottish Gaelic, Icelandic, Maltese

Evaluation Metrics

Each translation is scored 1-10 on:

Metric	What it measures
Accuracy	Meaning preservation
Fluency	Natural expression in target language
Style	Literary style/tone preservation
Overall	Combined quality score

Score Indicators

Score	Indicator	Quality
9-10	🟢	Excellent
7-8	🟡	Good
5-6	🟠	Acceptable
3-4	🔴	Poor
1-2	⚫	Failed

Reference Texts

5 classic literature excerpts (~500 chars each):

1. Pride and Prejudice (Austen, 1813) - Ironic prose
2. The Picture of Dorian Gray (Wilde, 1890) - Sensory description
3. A Study in Scarlet (Doyle, 1887) - Dialogue
4. Walden (Thoreau, 1854) - Philosophical prose
5. Moby-Dick (Melville, 1851) - Archaic narrative

Testing New Models

# 1. Install model
ollama pull new-model:tag

# 2. Run benchmark
python -m benchmark.cli run -m new-model:tag --openrouter-key KEY

# 3. View results
python -m benchmark.cli show RUN_ID --detailed

# 4. Merge with existing results
python -m benchmark.cli merge old_run new_run --publish

Configuration

Set in .env:

API_ENDPOINT=http://localhost:11434/api/generate
OPENROUTER_API_KEY=your_key
WIKI_REPO_URL=https://github.com/username/repo.wiki.git

Troubleshooting

Issue	Solution
No models found	Run ollama list and ollama pull model
Wiki push fails	Create at least one wiki page on GitHub first
Slow benchmarks	Use quick benchmark or fewer models

---

← Back to Home | View All Languages | View All Models