Source Verifier - Citation Checking for Wikipedia

An AI-powered Wikipedia user script that helps editors verify whether citations actually support the claims they're attached to. Clicking a citation number opens a sidebar that fetches the source, sends the claim plus the source text to an LLM, and returns a verdict: Supported, Partially Supported, Not Supported, or Source Unavailable.

Inspired by User:Polygnotus/Scripts/AI_Source_Verification.js and User:Phlsph7/SourceVerificationAIAssistant.js.

Features

• Click any [N] citation in an article to verify the associated claim
• Batch-verify every citation in an article and generate a wiki-markup report of failed citations
• Multiple LLM providers with a unified interface:
  • Lift Wing (default, free, no API key — Qwen3.8-27B via Wikimedia Toolforge)
  • HuggingFace (optional, currently not the default; set an HF API key to use HF-hosted models)
  • PublicAI (free, no API key — Qwen-SEA-LION / OLMo / Apertus)
  • Claude (Anthropic)
  • Gemini (Google)
  • ChatGPT (OpenAI)
• Source content fetched through a CORS proxy so the script runs purely in the browser
• Benchmark suite for evaluating provider accuracy against 76 human-labeled citation pairs

Installation

The script is deployed as a Wikipedia user script at User:Alaexis/AI_Source_Verification and auto-updates via USync.

To install, add the following line to your common.js:

importScript('User:Alaexis/AI_Source_Verification.js');

API keys for paid providers are stored in localStorage and configured from the sidebar UI. Lift Wing (the default) works out of the box with no key. Existing saved HuggingFace selections are migrated to Lift Wing once; users can reselect HuggingFace explicitly.

Usage

1. Open any Wikipedia article
2. Click the "Verify Sources" toggle to open the sidebar
3. Click any citation number [N] in the article — the claim and source will be extracted, sent to the selected LLM, and a verdict displayed
4. Or click Verify all citations to batch-check the whole article and generate a wiki-markup report

Command-line interface (ccs verify)

The CLI reuses core/ to verify a single citation from the terminal — the same verification the userscript performs in-page, minus the UI.

Install (from a clone)

git clone https://github.com/alex-o-748/citation-checker-script.git
cd citation-checker-script
npm install

No global install needed. Use the npx form shown below; npm exposes the ccs bin directly from node_modules/.bin.

Usage

npx ccs verify <wikipedia-url> <citation-number> [--provider <name>] [--no-log]

Example:

npx ccs verify https://en.wikipedia.org/wiki/Great_Migration_(African_American) 14

Run npx ccs --help for the full option and exit-code table.

Providers and API keys

Provider	Flag value	Env var required
Lift Wing (default)	--provider liftwing	none (Qwen3.8 through Toolforge)
HuggingFace	--provider huggingface	none (routed via the worker proxy; HF_API_KEY opts into direct calls)
PublicAI	--provider publicai	none (routed via the worker proxy)
Claude	--provider claude	CLAUDE_API_KEY
Gemini	--provider gemini	GEMINI_API_KEY
OpenAI	--provider openai	OPENAI_API_KEY

The CLI calls the same publicai-proxy.alaexis.workers.dev endpoint as the userscript for source fetching and PublicAI routing; other providers are called directly using your env-var API key.

Logging

By default the CLI POSTs a log entry to the worker proxy's /log endpoint (same schema the userscript uses). Pass --no-log to skip.

URL forms

Supported:

• https://en.wikipedia.org/wiki/<Title>
• https://en.wikipedia.org/wiki/<Title>?oldid=<rev>

Not supported in Phase 1:

• https://en.wikipedia.org/w/index.php?title=<Title> form
• ?curid=<pageid> form
• non-en Wikipedias

Verify HTTP API

POST /v1/verify at https://citation-verifier.toolforge.org checks one claim against one source over HTTP, with no API key, using the same pipeline as the userscript and CLI. It is for tools that can't import core/ directly. Request shape, limits, fair use and the status dashboard (GET /status): docs/verify-api.md. The server is api/server.js.

Batch CSV output

The batch runner keeps its complete findings CSV and automatically creates a second, reviewer-ready file with -clean before the extension:

node service/run-sweep.js --titles-file articles.txt --out findings.csv
# writes findings.csv and findings-clean.csv

The clean file excludes every check made against truncated source content. If a collective check was completed for an adjacent citation group, it also excludes that group's individual checks so the group is represented once.

To clean a CSV from an earlier run without changing the original file:

node service/clean-csv.js previous-findings.csv
# writes previous-findings-clean.csv

node service/clean-csv.js previous-findings.csv --out ready-to-share.csv

Post-run checks

When the CSV is written, the runner also checks it for signs that the run went wrong and writes three small files beside it:

File	Holds
findings-checks.md	One line per check, PASS / WARN / FAIL: missing articles, fetch failures caused by our own fetcher, outage windows, ERROR rows, truncated sources, markup in claims, over-long claims, table-like articles, rationales in the wrong language, answers that break the prompt's format, mixed models or prompt versions
findings-checks.json	The same, plus metrics (rows per article, verdicts, fetch failures by cause) for comparing runs
findings-review.csv	The rows worth reading: up to 20 examples of each kind of suspect row (a row a check picked out on its own, e.g. CSS in the claim) plus 15 random rows of each verdict, with a review_reason column

A run that finishes but fails a check exits with code 5, so a Toolforge job submitted with --emails onfailure emails you about it. --skip-checks turns the checks off. To check an existing CSV, including one from a killed run:

node service/run-checks.js previous-findings.csv --titles-file articles.txt
# writes previous-findings-checks.md, -checks.json and -review.csv

The file is streamed, so its size doesn't matter (an 80,000-row, 133 MB CSV takes about 8 seconds and under 200 MB of memory).

Repository Layout

main.js                          Wikipedia user script (single-class IIFE)
core/                            Pure-logic ESM modules spliced into main.js
bin/, cli/                       `npx ccs verify` CLI entry point
benchmark/                       Benchmark suite (Node.js)
  extract_dataset.js               Extract claim/source pairs from Wikipedia
  run_benchmark.js                 Run LLM verification on the dataset
  analyze_results.js               Metrics and confusion matrices
  generate_comparison.js           CSV comparison across providers
  dataset.json                     Ground-truth claim-citation pairs
Benchmarking_data_Citations.csv  Source ground-truth data
docs/                            Reference docs and design plans (see docs/README.md)

Architecture

• Single class: WikipediaSourceVerifier wrapped in an IIFE — no build step, no modules
• Pure ES6+ browser JS: runs directly in the Wikipedia page context (no Node.js APIs)
• Provider abstraction: each provider has an entry in this.providers and a callXxxAPI() method; callProviderAPI() routes between them
• CORS proxy: sources fetched via publicai-proxy.alaexis.workers.dev
• UI: OOUI (OOjs UI) buttons and dialogs, lazy-loaded through MediaWiki; CSS injected at runtime via createStyles()
• State: class instance variables plus user preferences in localStorage

See CLAUDE.md for a table of key methods and more detail.

Benchmark Suite

The benchmark suite evaluates provider accuracy against a curated dataset of 76 Wikipedia citations with human-labeled verdicts.

cd benchmark
npm install

npm run extract           # Extract dataset from Wikipedia
npm run extract:dry       # Dry-run extraction
npm run benchmark         # Run benchmarks on all providers
npm run benchmark:publicai  # Run a specific provider
npm run analyze           # Calculate metrics
npm run report            # Generate a markdown report

Required environment variables:

Variable	Provider
ANTHROPIC_API_KEY	Claude
OPENAI_API_KEY	OpenAI
GEMINI_API_KEY	Gemini
PUBLICAI_API_KEY	PublicAI models

See benchmark/README.md for the full workflow, including manual-review and resume instructions.

Development

• No CI/CD and no linter; core/*.js has a node:test suite (npm test), and end-to-end validation is via the benchmark suite
• Edit main.js directly; test by loading it on Wikipedia (via the user-script page or a browser-console importScript call)
• For testing changes before release, use User:Alaexis/AI_Source_Verification_test.js, which tracks the dev branch
• Feature branches off main, merged via pull requests
• To add a new provider: add an entry to this.providers in the constructor, implement callXxxAPI(), and add routing in callProviderAPI()

Constraints to Keep in Mind

• main.js runs in the browser — no Node.js APIs, no ES modules, no npm packages
• All external fetches must go through the CORS proxy
• OOUI components must be loaded via mw.loader.using() before use
• API keys live in localStorage, never in source
• The system prompt contains 9 tuned few-shot examples; edits affect benchmark accuracy
• Claim extraction uses "between adjacent citations" logic by design (not full sentences) for precision

core/ and the sync script

Pure-logic functions (prompt building, verdict parsing, URL extraction, claim extraction, provider dispatch, worker proxy calls) live in core/*.js as ESM modules and are tested with node:test:

npm install
npm test

main.js is a Wikipedia userscript with no module system, so core/ is also spliced into it by scripts/sync-main.js:

npm run build            # regenerate main.js from core/
npm run build -- --check # fail if main.js is stale (for CI)

The injected region in main.js is framed by // <core-injected> and // </core-injected> markers — do not edit between them by hand; edit the file in core/ and rerun npm run build.

Class methods on WikipediaSourceVerifier that correspond to core/ functions are thin wrappers; the bodies live in core/. The rest of main.js — UI, event handlers, MediaWiki integration — is hand-maintained as before.