Version 2.0 | Last updated: 2026-03-18
- What This Tool Does
- Prerequisites
- Installation
- Getting Started — Web UI
- Step-by-Step Workflow
- Extract Options Explained
- Testing Before a Full Run
- Overnight Queue
- Monitoring Progress
- Understanding the Output CSV
- CLI Reference (Advanced)
- Troubleshooting
- FAQ
This tool extracts voter data from Tamil Nadu electoral roll PDFs into structured CSV files. It processes English and Tamil PDF pairs for each Assembly Constituency (AC), using local OCR (Optical Character Recognition) — no internet or API costs required.
What you get: A CSV file per constituency with columns for Serial No, EPIC ID, Name (English & Tamil), Relation Name (English & Tamil), Relation Type, House No, Age, and Gender.
What you need: The electoral roll PDFs downloaded from voters.eci.gov.in in both English and Tamil.
| Requirement | Details |
|---|---|
| Operating System | Windows 10/11 (primary), Linux/macOS (supported) |
| Python | 3.10 or higher |
| Tesseract OCR | Version 5.4+ with Tamil language data |
| Disk Space | ~50 KB per page pair output; ~500 MB for Tesseract + dependencies |
| RAM | Minimum 2 GB free (4+ GB recommended for multi-worker) |
The easiest way — the Web UI checks everything for you and installs missing components with one click.
- Double-click
start.bat(Windows) or runbash start.sh(Linux/macOS) - A browser window opens at
http://localhost:7000 - Click the gear icon (⚙ Setup) in the sidebar
- The Setup panel shows three dependency checks:
- Python OCR Packages — PyMuPDF, OpenCV, pytesseract, NumPy, Pillow, pypdf
- Tesseract Binary — The OCR engine itself
- Tamil Tessdata — Tamil language model for Tesseract
- Click Install next to any item showing ❌
- When all three show ✅, you're ready
setup.batThis script checks and installs Python, Tesseract, Tamil language data, and Python packages automatically. Follow the on-screen prompts.
Download from python.org. Important: Check "Add Python to PATH" during installation.
winget install UB-Mannheim.TesseractOCRDuring installation, check "Additional language data" and "Additional script data" to include Tamil support.
Verify:
"C:\Program Files\Tesseract-OCR\tesseract.exe" --versionDownload tam.traineddata and copy to:
C:\Program Files\Tesseract-OCR\tessdata\tam.traineddata
pip install -r requirements.txtHere are the most common setup failures and how to fix them:
| Problem | Solution |
|---|---|
pip not found |
Python wasn't added to PATH. Reinstall Python and check "Add Python to PATH" |
| Permission denied | Open Command Prompt as Administrator, then run pip install -r requirements.txt |
No module named 'fitz' |
Run pip install pymupdf directly |
| Package version conflict | Run pip install --upgrade -r requirements.txt to force upgrade |
| Build errors (C++ compiler) | Install Visual Studio Build Tools from visualstudio.microsoft.com |
| Problem | Solution |
|---|---|
winget not available |
Your Windows version may not have it. Download Tesseract manually from UB-Mannheim GitHub. Install the .exe to C:\Program Files\Tesseract-OCR\ |
| Tesseract installed but not detected | Make sure C:\Program Files\Tesseract-OCR\tesseract.exe exists. If installed elsewhere, add its folder to your system PATH |
| Antivirus blocks install | Temporarily disable real-time protection, install, then re-enable |
| Problem | Solution |
|---|---|
| Permission denied on tessdata folder | The Web UI will automatically fall back to a project-local tessdata/ folder. Alternatively, open Command Prompt as Administrator and copy the file manually |
| Download fails (no internet) | Download tam.traineddata on another machine from this link, then copy it to C:\Program Files\Tesseract-OCR\tessdata\ via USB |
Tamil not showing in --list-langs |
The file may be corrupted. Re-download (should be ~40 MB) |
| Problem | Solution |
|---|---|
fastapi not found |
Run pip install fastapi uvicorn aiofiles psutil |
| Port 7000 in use | The launcher auto-tries ports 7000-7009. If all are in use, close other applications or check with netstat -ano | findstr :7000 |
| Unicode error on startup | This has been fixed in v2.0. If you see it, ensure you have the latest code |
| Browser doesn't open | Navigate manually to http://localhost:7000 |
General tip: If all else fails, try the Web UI Setup panel — it provides the most user-friendly installation experience with real-time progress streaming.
Windows:
start.batLinux/macOS:
bash start.shThe browser opens automatically at http://localhost:7000. This is a local-only address — nothing is exposed to the network.
When you're done, stop the server:
- Find the terminal/command prompt window that opened when you ran
start.bat - Press
Ctrl+Cin that window, or simply close the window - Close the browser tab
All your data, checkpoints, and queue state are saved to disk automatically. Next time you run start.bat, everything picks up where you left off.
The UI has a sidebar on the left (desktop) or a bottom bar (mobile/small screens):
| Section | Icon/Label | What it does |
|---|---|---|
| Setup | ⚙ (gear) | Collapsible panel to check and install dependencies. Hidden by default — click to expand. |
| Workflow | ▶ Workflow | Main workspace. Select an AC, configure options, run the pipeline. |
| Live Logs | 📋 Live Logs | Watch real-time output from running jobs. Colour-coded log lines, ETA, kill button. |
| Data | 📁 Data | Overview of all ACs — progress, record counts, CSV download. |
| History | 🕐 History | Browse past log files and run summaries. |
- Go to voters.eci.gov.in
- Select your state (Tamil Nadu), district, and Assembly Constituency
- Download the electoral roll PDFs in both English and Tamil
- Place them in the correct folder structure:
Input/ER_Downloads/AC-188/
english/ <-- English PDFs (e.g., 2026-EROLLGEN-...-ENG-1-WI.pdf)
tamil/ <-- Tamil PDFs (e.g., 2026-EROLLGEN-...-TAM-1-WI.pdf)
Important: The number of English and Tamil PDFs should match. If they don't, the UI will show a yellow warning with a link to re-download. If a folder is empty, the UI will show the exact file path where PDFs should be saved.
If the AC already has PDFs:
- Open the Workflow tab
- Select your AC from the dropdown (e.g.,
AC-188 (0%))
If you need to create a new AC:
- Click the green
+button next to the AC dropdown - Enter the AC number in
AC-xxxformat (e.g.,AC-188) - The tool creates the directory structure automatically
- Copy your PDF files into the newly created
english/andtamil/folders - Click the ↺ refresh button to update the dropdown
What this does: Splits each multi-page PDF into individual page files. All pages are split — non-data pages (metadata, summary, maps) are automatically skipped during extraction later.
- In the Pipeline section, find 1. Split PDFs
- (Optional) Check
--forceif you want to overwrite existing split files - Click ▶ Run
- The job opens in Live Logs — wait for completion
Tip: Click the ℹ button next to the step name to see a detailed explanation.
What this does: Runs Tesseract OCR on each page pair (English + Tamil) and extracts voter records into CSV files.
- Find 2. Extract (OCR) in the Pipeline
- Set your desired options:
- Workers slider (left panel): Controls parallel processing. The slider shows green/yellow/red based on your system resources.
- --part: Leave empty for all parts, or enter a range (e.g.,
1-50) - --page: Leave empty for all pages, or enter specific pages (e.g.,
1-10) - --limit: Leave as 0 for all, or set a number to process only that many pairs
- --cross-check: Enable to add validation columns to the CSV
- Click ▶ Run
- Monitor progress in Live Logs
For your first run, we recommend testing a few pages first — see Testing Before a Full Run.
What this does: Combines individual page CSVs into part-level and AC-level files.
- Find 3. Merge Outputs
- (Optional) Check
--forceif you're re-merging after additional extractions - Click ▶ Run
- Go to the Data tab
- Find your AC in the list
- Click ⬇ Download CSV to get the merged AC-level CSV file
- Open in Excel, Google Sheets, or any spreadsheet application
Or use Full Pipeline: Instead of running steps individually, click ⚡ Run Full Pipeline (Split → Extract → Merge) to run all three steps automatically.
| Option | Format | Example | What it does |
|---|---|---|---|
| --part | Number or range | 101, 50-100, 1,5,10-20 |
Process only specific parts. Leave empty for all. |
| --page | Number or range | 4, 1-10, 1,5,10-20 |
Process only specific pages within a part. Requires --part. |
| --limit | Number | 10 |
Stop after processing this many pairs. 0 = no limit. |
| --cross-check | Checkbox | — | Cross-validates English vs Tamil data. Adds two extra columns: Cross_Check (OK/REVIEW) and Cross_Check_Notes. |
| --reset | Checkbox (yellow) | — | Clears the checkpoint for the specified part, allowing re-processing. Use with care — a confirmation dialog will appear. |
| Button | What it does |
|---|---|
| Dry Run | Shows what file pairs would be processed, without actually processing them. Useful to verify before committing. |
| Validate Page | Tests extraction on specific pages and shows detailed output in Live Logs. Respects the --part and --page settings. |
| Test 1 Page (bottom) | Quick one-page test with inline preview table. Great for checking OCR quality. |
Before committing to a full AC extraction (which can take hours), it's wise to test:
- Select your AC
- Click 🔍 Test 1 Page at the bottom of the Pipeline section
- A preview table appears showing extracted records — verify names, EPIC IDs, ages look correct
- Set --part to a part number (e.g.,
1) - Set --page to a page range (e.g.,
1-5) - Click Validate Page
- Check the Live Logs for detailed output
- Click Dry Run
- Check Live Logs — it lists all pending file pairs without processing them
- Useful to verify the right files are queued
For processing multiple ACs unattended:
- Select an AC and configure workers/options
- Click Add Selected AC in the Overnight Queue section
- Repeat for other ACs
- Click Start Queue
- The tool runs Split → Extract → Merge for each AC sequentially
- A browser notification fires when each AC completes
Queue features:
- Queue state is saved to disk — if the server restarts, the queue resumes
- Click Stop After Current to gracefully pause (finishes the current AC)
- Remove waiting items with the X button
- Live Logs tab: Real-time colour-coded output. Red = errors, yellow = warnings, green = success.
- ETA indicator: Shows percentage complete and page count (e.g., "45% done (130/290 pages)")
- Kill button: Red button to stop a running job immediately
- File path summary: When a job completes, errors, or is killed, the log terminal shows the input and output file paths for that step
- Workflow tab: AC status card shows extraction progress bar, record count, and step completion (Downloads ✓, Split ✓, Merged ✓)
- Data tab: Overview of all ACs with progress bars and record counts
The merged CSV has 14 columns by default:
| Column | Description | Example |
|---|---|---|
| AC No | Assembly Constituency number | 188 |
| Part No | Part number within the AC | 33 |
| Serial No | Voter serial number | 211 |
| EPIC ID | Voter ID card number | RVJ1612993 |
| Name (English) | Voter name in English | Kavitha |
| Name (Tamil) | Voter name in Tamil | கவிதா |
| Relation Name (English) | Father/Husband name in English | Murugesan |
| Relation Name (Tamil) | Father/Husband name in Tamil | முருகேசன் |
| Relation Type | Relationship | Father |
| House No | House/door number | 1-192 |
| Age | Voter age | 30 |
| Gender | Male or Female | Female |
| DOB | Date of birth (always blank) | |
| ContactNo | Contact number (always blank) |
With --cross-check enabled, two extra columns are added:
| Column | Values | Meaning |
|---|---|---|
| Cross_Check | OK or REVIEW |
REVIEW = at least one field disagreed between English and Tamil |
| Cross_Check_Notes | Text | Details of mismatches (e.g., EPIC mismatch EN=WXJ1234567 TA=WXJ1234568) |
File details:
- Encoding: UTF-8 with BOM (opens correctly in Excel)
- House numbers prefixed with
'to prevent Excel auto-formatting - Records sorted by Serial No ascending
All operations available in the Web UI can also be run from the command line:
python split_pdfs.py --ac AC-188
python split_pdfs.py --ac AC-188 --force # Overwrite existingpython extract_ocr.py AC-188 --workers 4 # Full AC
python extract_ocr.py AC-188 --part 101 # Specific part
python extract_ocr.py AC-188 --part 50-100 # Part range
python extract_ocr.py AC-188 --part 3 --page 1-10 # Page range within part
python extract_ocr.py AC-188 --part 3 --page 1,5,10-20 # Specific pages
python extract_ocr.py AC-188 --limit 10 --workers 4 # First 10 pairs only
python extract_ocr.py AC-188 --validate # Test 1 pair
python extract_ocr.py AC-188 --part 3 --page 4 --validate # Test specific page
python extract_ocr.py AC-188 --dry-run # Show pending pairs
python extract_ocr.py AC-188 --reset --part 101 # Reset checkpoint for part
python extract_ocr.py AC-188 --cross-check --workers 4 # With cross-validation
python extract_ocr.py --all --workers 4 # All ACspython merge_outputs.py --ac AC-188
python merge_outputs.py --ac AC-188 --force # Re-merge all partspython analyze_quality.py --ac AC-188| Problem | Solution |
|---|---|
| Setup: Tesseract not found | Verify C:\Program Files\Tesseract-OCR\tesseract.exe exists. If installed elsewhere, add to PATH. |
| Setup: Tamil not available | Download tam.traineddata and copy to Tesseract's tessdata folder. See Installation. |
| Setup: Python packages fail | Try pip install --upgrade -r requirements.txt. If permission errors, run as Administrator. |
| Web UI won't start | Run pip install fastapi uvicorn aiofiles psutil. Check if port 7000 is free. |
| "Method Not Allowed" error | The server is running old code. Restart: close the server terminal, run start.bat again. |
| Count mismatch warning | English and Tamil PDF counts differ. Check your downloads at voters.eci.gov.in. |
| Tamil names are empty | Verify tam in tesseract --list-langs. Re-download tam.traineddata if needed. |
| Processing is slow | Reduce --workers if RAM-limited. Each worker uses ~200 MB RAM. |
| Merge missing pages | Use --force when merging: the merge does a full rewrite, not append. |
| Job shows error immediately | Check Live Logs — the subprocess exit message shows the cause. |
| Server changes not taking effect | Restart the web server. Python code changes require a server restart. |
| Grid detection failed | Automatic fallback handles this. If data quality is poor, try --validate on the specific page. |
Q: Does this tool need internet access? A: No. Once installed, everything runs locally. The only time internet is needed is for initial setup (downloading Tesseract and Python packages).
Q: How long does processing take? A: About 60 seconds per page pair. A full AC with ~720 pairs takes 3-4 hours with 4 workers. Multiple ACs can be queued for overnight processing.
Q: Is my data sent anywhere? A: No. All processing happens on your machine. No data leaves your computer.
Q: Can I stop and resume later? A: Yes. The tool saves a checkpoint after each page pair. When you run extraction again, it picks up where it left off automatically.
Q: What if I need to re-process some pages?
A: Use --reset with --part to clear the checkpoint for specific parts. Or use --page with --validate to re-inspect individual pages without affecting the checkpoint.
Q: What accuracy can I expect? A: 99.87% cell-level accuracy based on validation of 1,118 records. Occasional EPIC ID letter/digit confusion may occur due to source PDF resolution (115 DPI).
Q: Can multiple people work on different ACs simultaneously? A: Yes. Each AC has its own checkpoint. Different users can process different ACs at the same time (on different machines or terminals).
Q: What's the --cross-check option for?
A: It re-reads Tamil cells to independently verify EPIC IDs, house numbers, and serial numbers against the English data. Records with disagreements are flagged as REVIEW — useful for quality auditing.
Q: How do I update to a newer version? A: Pull the latest code (if using git) or download the updated files. Restart the web server. Your existing data and checkpoints are preserved.
This guide covers the Electoral Roll OCR Tool v2.0. For technical details, see README.md. For developer reference, see CLAUDE.md.