Narrative Consistency Verification Over Long-Form Text
Kharagpur Data Science Hackathon 2026 - Track A
This system automatically verifies whether character backstories align with their portrayal in classic literature. Using state-of-the-art Large Language Models (Google Gemini), BM25 information retrieval, and intelligent multi-stage reasoning, we achieve ~80% accuracy on this challenging literary analysis task—a significant improvement over the 33% baseline.
Key Innovation: Intelligent handling of retrieval uncertainty. When the system can't find evidence to verify claims but also finds no contradictions, it gives the benefit of the doubt to the backstory. This insight improved accuracy from 25% → 80%.
Given a character from classic literature and a hypothetical backstory description, determine if the backstory is:
- Consistent (1): The backstory aligns with the character's portrayal in the original book
- Contradictory (0): The backstory contains facts that contradict the original book
Source Literature:
- "In Search of the Castaways" by Jules Verne
- "The Count of Monte Cristo" by Alexandre Dumas
Dataset Size:
- Training Set: 81 labeled examples
- Test Set: 60 unlabeled examples
- Backstory Length: 50-200 words typically
Example:
character,book,backstory,label
"Thalcave","In Search of the Castaways","Thalcave's people faded as colonists advanced...",1
"Edmond Dantès","The Count of Monte Cristo","Edmond grew up wealthy in Marseille...",0- Long-Context Reasoning: Novels exceed typical model context limits (>100,000 words)
- Dispersed Evidence: Relevant details scattered across multiple chapters
- Temporal & Causal Reasoning: Detecting timeline inconsistencies and logical conflicts
- Narrative Noise: Distinguishing relevant from irrelevant text
- Subtle Contradictions: Detecting implicit conflicts, not just explicit ones
- 80% Accuracy (vs 33% baseline) on literary fact verification
- 15-20 seconds per example processing time
- Google Gemini LLM for advanced literary understanding
- BM25 Sparse Retrieval for efficient evidence gathering
- Cross-Platform Support (Windows/Linux/Mac/Google Colab)
- Cost-Effective (uses Gemini tier 1 - 7.4 cents per 100 requests)
- Modular Architecture with clear separation of concerns
- Explainable Results with reasoning for each decision
Input: Character Backstory
│
▼
┌────────────────────────────────────────────┐
│ 1. CLAIM EXTRACTION (claims.py) │
│ Extract atomic factual claims │
│ Technology: Regex-based sentence splitting │
└────────────┬───────────────────────────────┘
│
▼
┌────────────────────────────────────────────┐
│ 2. EVIDENCE RETRIEVAL (retrieval.py) │
│ Technology: BM25 + heuristic reranking │
│ Search: 2740 pre-chunked book passages │
│ Output: Top-8 relevant passages per claim │
└────────────┬───────────────────────────────┘
│
▼
┌────────────────────────────────────────────┐
│ 3. CLAIM VERIFICATION (llm.py) │
│ Technology: Google Gemini Flash │
│ Batch verify all claims in single API call │
│ Output: verdict + confidence + reasoning │
└────────────┬───────────────────────────────┘
│
▼
┌────────────────────────────────────────────┐
│ 4. SCORE CONVERSION (pipeline.py) │
│ Convert verdicts to probability scores │
└────────────┬───────────────────────────────┘
│
▼
┌────────────────────────────────────────────┐
│ 5. VERDICT CLASSIFICATION (decision.py) │
│ Apply calibrated thresholds │
│ Classify: Strong/Weak/Supported/Neutral │
└────────────┬───────────────────────────────┘
│
▼
┌────────────────────────────────────────────┐
│ 6. AGGREGATION (decision.py) │
│ Intelligent multi-claim aggregation │
│ Key: All neutral + no contradictions → 1 │
└────────────┬───────────────────────────────┘
│
▼
Output: Final prediction (0 or 1)
| Component | Technology | Purpose |
|---|---|---|
| Claim Extraction | Regex sentence splitting | Break backstory into atomic claims |
| Evidence Retrieval | BM25 (rank_bm25) | Find relevant book passages |
| Verification | Google Gemini 2.0/3.0 Flash | Verify claims against evidence |
| Score Conversion | Probability mapping | Convert verdicts to scores |
| Classification | Threshold-based | Categorize claim verdicts |
| Aggregation | Rule-based logic | Make final decision |
- Python 3.10 or higher
- Google Gemini API key (Get one free)
- Internet connection (for API calls)
- Clone the repository
git clone <repository-url>
cd KDSH-pub- Create virtual environment
# Windows
python -m venv venv
venv\Scripts\activate
# Linux/Mac
python -m venv venv
source venv/bin/activate- Install dependencies
pip install -r requirements-cpu.txt- Set up API key
Create a .env file or set environment variable:
# Option 1: Environment variable
export GEMINI_API_KEY="your-api-key-here"
# Option 2: .env file
echo "GEMINI_API_KEY=your-api-key-here" > .env- Run the system
python src/run.py --input dataset/train.csv --output predictions.csvWindows:
- Supported ✅
- BM25 retrieval works natively
- Pathway vector store not available (optional feature)
Linux/Mac:
- Fully supported ✅
- All features available including Pathway
Google Colab:
# Install dependencies
!pip install -r requirements-colab.txt
# Set API key
import os
os.environ['GEMINI_API_KEY'] = 'your-key-here'
# Run notebook
# See run.ipynb for examples# Process training set
python src/run.py --input dataset/train.csv --output train_predictions.csv
# Process test set
python src/run.py --input dataset/test.csv --output test_predictions.csv
# Enable verbose logging
python src/run.py --input dataset/train.csv --output predictions.csv --verbosefrom kds_reasoning.pipeline import verify_backstory
from kds_reasoning.config import SystemConfig
# Load configuration
config = SystemConfig.load_from_json('config.json')
# Verify a single backstory
result = verify_backstory(
character="Thalcave",
book="In Search of the Castaways",
backstory="Thalcave's people faded as colonists advanced...",
config=config
)
print(f"Prediction: {result['prediction']}")
print(f"Confidence: {result['confidence']}")
print(f"Reasoning: {result['reasoning']}")Edit config.json to customize system behavior:
{
"chunking": {
"chunk_words": 280,
"overlap_words": 40
},
"retrieval": {
"top_k": 8,
"prefetch_k": 20
},
"nli": {
"strong_contradiction_prob": 0.80,
"weak_contradiction_prob": 0.60,
"supported_prob": 0.65
},
"use_gemini": true,
"gemini_model": "gemini-3-flash-preview"
}| Metric | Value |
|---|---|
| Accuracy | ~80% |
| Baseline | 33% (random/all-0) |
| Improvement | +47 percentage points |
| Speed | 15-20s per example |
| API Cost | 100 request takes 7.4 cents |
| Approach | Accuracy | Speed | Notes |
|---|---|---|---|
| Gemini-Only (Ours) | 80% | 15-20s | Production system |
| DeBERTa NLI | 50-60% | 240s | Slow on CPU |
| Heuristic Rules | 35-40% | <1s | Baseline |
| Random Guess | 50% | - | Binary classification |
- Attribute Constraints: Handled reliably (explicit in text)
- Temporal Constraints: Strong performance (clear timeline markers)
- Relational Constraints: Moderate difficulty (gradual development)
- Causal Constraints: Most challenging (often implicit)
Conservative aggregation reduces false positives significantly
Batch processing provides 8x speedup over sequential verification
Gemini LLM shows 30-50% accuracy gain over traditional NLI models
BM25 retrieval sufficient for this task (dense retrieval minimal improvement)
The Critical Insight:
# Problem: Poor retrieval → all neutral → wrong prediction (25% accuracy)
if all_neutral:
return 0 # WRONG!
# Solution: No contradictions + all neutral → give benefit of doubt
if all_neutral and no_contradictions:
return 1 # CORRECT! (80% accuracy)This single change improved accuracy by +55 percentage points.
# Before: N API calls (slow)
for claim in claims:
verify_claim(claim) # 15s × 8 = 120s
# After: 1 API call (fast)
verify_claims_batch(claims) # 15s total
# Result: 8x faster# Primary: google-generativeai (stable)
import google.generativeai as genai
# Fallback: google-genai (new SDK)
from google import genai
# Auto-detection ensures compatibility across environments- Truncated response recovery
- Malformed JSON parsing with fallback
- Automatic retry on API failures
- Graceful degradation to neutral verdicts
- Comprehensive logging for debugging
KDSH-pub/
├── src/
│ ├── run.py # Main entry point
│ ├── calibrate.py # Threshold calibration
│ ├── serve_index.py # Server utilities
│ └── kds_reasoning/
│ ├── __init__.py
│ ├── claims.py # Claim extraction
│ ├── retrieval.py # BM25 evidence retrieval
│ ├── llm.py # Gemini LLM integration
│ ├── nli.py # Fallback NLI models
│ ├── decision.py # Verdict logic
│ ├── pipeline.py # Orchestration
│ ├── config.py # Configuration dataclasses
│ ├── config_io.py # Config I/O
│ ├── data.py # Data structures
│ ├── chunking.py # Text chunking
│ ├── embeddings.py # Embedding utilities
│ ├── evidence.py # Evidence handling
│ ├── pathway_server.py # Vector store (optional)
│ └── pathway_support.py # Platform detection
├── dataset/
│ ├── train.csv # Training data
│ ├── test.csv # Test data
│ └── Books/ # Source novels
│ ├── In search of the castaways.txt
│ └── The Count of Monte Cristo.txt
├── Cache/ # BM25 index cache (auto-generated)
├── vena/ # Virtual environment
├── requirements.txt # Production dependencies
├── requirements-cpu.txt # CPU-only dependencies
├── requirements-colab.txt # Google Colab dependencies
├── config.json # System configuration
├── run.ipynb # Jupyter notebook demo
├── evaluate.py # Evaluation script
├── test_gemini_direct.py # API testing
├── test_minimal.py # Minimal test
├── new_DOCUMENTATION.md # Complete technical docs
└── README.md # This file
# Test first 5 examples
./test_improvements.sh
# Output:
# Processing 5 examples...
# Accuracy: 80% (4/5)
# Average time: 18.2s per example# Evaluate on training set
python evaluate.py --input dataset/train.csv --predictions train_predictions.csv
# Output:
# Accuracy: 0.80
# Precision: 0.82
# Recall: 0.78
# F1-Score: 0.80# Run all tests (requires pytest)
pytest tests/
# Run specific test
pytest tests/test_claims.py
pytest tests/test_retrieval.py1. API Key Not Set
Error: Please set GEMINI_API_KEY environment variable
Solution: Export the API key or create .env file
2. Slow Performance
Warning: Processing taking >60s per example
Solution: Check internet connection, verify Gemini API quota
3. Import Errors
ModuleNotFoundError: No module named 'google.generativeai'
Solution: pip install -r requirements.txt
4. Pathway Not Available (Windows)
Warning: Pathway not available on Windows
Solution: Expected behavior. BM25 retrieval still works (recommended)
# Enable verbose logging
python src/run.py --input dataset/train.csv --output predictions.csv --verbose
# Check logs for detailed execution trace- Accuracy First: Use best-in-class AI for literary analysis
- Speed Matters: Enable fast iteration (15-20s per example)
- Cost Consciousness: Leverage cheap tiers (Gemini)
- Robustness: Multiple fallbacks for reliability
- Cross-Platform: Work everywhere (Windows/Linux/Mac/Colab)
- Explainability: Every decision backed by evidence
Why Gemini over traditional NLI?
- 30-50% accuracy improvement
- Better literary understanding
- Faster (15s vs 240s)
- Free tier available
Why BM25 over dense retrieval?
- Works on all platforms
- No GPU required
- 100x faster indexing
- Sufficient accuracy for this task
Why conservative aggregation?
- Prioritizes precision over recall
- Reduces false positives
- Aligns with task requirements
- Handles retrieval uncertainty
- Better Temporal Reasoning: Explicit timeline extraction and validation
- Causal Graph Construction: Model cause-effect relationships
- Character State Tracking: Monitor character development arcs
- Cross-Document Reasoning: Compare across multiple books
- Active Learning: Query user for ambiguous cases
- Confidence Calibration: Better uncertainty quantification
- Multi-Language Support: Extend to non-English literature
- Real-Time Processing: Streaming verification as user types
- Visual Explanations: Timeline and evidence visualization
- Fine-Tuned Models: Domain-specific LLM adaptation
- Narrative Understanding: Deeper plot and theme analysis
- Implicit Reasoning: Better handling of unstated information
- World Knowledge Integration: Combine text with common sense
- Multi-Modal Analysis: Include character dialogues, descriptions
- Scalability: Handle entire book series (e.g., Harry Potter)
This is a hackathon project!
- Kharagpur Data Science Hackathon 2026 organizers
- Google Gemini for providing free API access
- Pathway
For questions, issues, or collaboration:
- Open an issue on GitHub
- Contact team members through KIIT
Made with ❤️ by Team Mittens