Add tournament-style pairwise ranking aggregation

[bledden]

Summary

Adds calculate_tournament_rankings() as an alternative ranking method alongside the existing mean-based aggregation.

Motivation

The current calculate_aggregate_rankings() averages position numbers, which has limitations:

• Vulnerable to outlier rankings (one model ranking Response E first significantly affects E's score)
• Position 1→2 treated same as position 4→5
• Not theoretically principled

Tournament-style pairwise comparison is more robust:

• Converts rankings to head-to-head matchups
• Majority vote determines each matchup winner
• Final score = win percentage across all matchups
• Based on Condorcet voting theory

Algorithm

For rankings like:

Ranker 1: A > B > C
Ranker 2: A > C > B
Ranker 3: B > A > C

1. Extract pairwise preferences from each ranking
2. For each pair (A vs B), count votes: A wins 2, B wins 1 → A wins matchup
3. Calculate: A: 2 wins (100%), B: 1 win (50%), C: 0 wins (0%)

Changes

• Add calculate_tournament_rankings() function in backend/council.py
• Update run_full_council() to include tournament_rankings in metadata
• Both methods now available: aggregate_rankings (mean) and tournament_rankings (pairwise)

Validation

Tested with 7 unit test scenarios:

• ✅ Unanimous rankings
• ✅ Split decisions (2:1 votes)
• ✅ Tie handling (0.5 points each)
• ✅ Single ranker edge case
• ✅ Empty rankings edge case
• ✅ Cyclic preferences (A>B, B>C, C>A)
• ✅ Outlier robustness comparison

End-to-end test with 5 models ranking 5 responses confirms tournament ranking is more robust to outliers.

Test plan

• Verify tournament_rankings appears in metadata
• Verify ranking order matches expected pairwise winners
• Verify ties are handled correctly (0.5 points each)

🤖 Generated with Claude Code