|
| 1 | +<!DOCTYPE html> |
| 2 | +<html lang="en"> |
| 3 | +<head> |
| 4 | + <meta charset="UTF-8"> |
| 5 | + <meta name="viewport" content="width=device-width, initial-scale=1.0"> |
| 6 | + <title>Module 1 Deep Dive: Scraping ESPN's Scoreboard Endpoints for Every NBA Game in a Season — CodeFix Solution</title> |
| 7 | + <meta name="description" content="Scrape every NBA game in a full season from ESPN's undocumented JSON scoreboard endpoints. Working Python code with date iteration, rate limiting, retry, deduplication, and parquet output for downstream ML."> |
| 8 | + <link rel="canonical" href="https://codefixsolution.com/module-1-scraping-espn-scoreboard-nba/"> |
| 9 | + <meta property="og:title" content="Module 1: Scraping ESPN's Scoreboard for Every NBA Game in a Season"> |
| 10 | + <meta property="og:description" content="Complete Python tutorial: ESPN scoreboard endpoints, full-season iteration, rate limiting, retry, parquet output. The starting module of the Polymarket Bot Course."> |
| 11 | + <meta property="og:url" content="https://codefixsolution.com/module-1-scraping-espn-scoreboard-nba/"> |
| 12 | + <meta property="og:type" content="article"> |
| 13 | + <style> |
| 14 | + :root { --bg: #0a0a0a; --card: #111; --border: #222; --text: #e5e5e5; --muted: #888; --accent: #22c55e; --code-bg: #0d1117; } |
| 15 | + * { margin: 0; padding: 0; box-sizing: border-box; } |
| 16 | + body { font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif; background: var(--bg); color: var(--text); line-height: 1.8; } |
| 17 | + .container { max-width: 740px; margin: 0 auto; padding: 0 20px; } |
| 18 | + header { border-bottom: 1px solid var(--border); padding: 20px 0; } |
| 19 | + header a { color: var(--text); text-decoration: none; font-weight: 700; font-size: 18px; } |
| 20 | + article { padding: 48px 0 60px; } |
| 21 | + h1 { font-size: 34px; font-weight: 800; line-height: 1.2; margin-bottom: 16px; } |
| 22 | + .meta { color: var(--muted); font-size: 14px; margin-bottom: 32px; } |
| 23 | + h2 { font-size: 24px; font-weight: 700; margin: 40px 0 16px; padding-top: 16px; border-top: 1px solid var(--border); } |
| 24 | + h3 { font-size: 18px; font-weight: 600; margin: 28px 0 12px; } |
| 25 | + p { margin-bottom: 16px; color: #ccc; } |
| 26 | + ul, ol { margin: 0 0 16px 24px; color: #ccc; } |
| 27 | + li { margin-bottom: 6px; } |
| 28 | + a { color: var(--accent); } |
| 29 | + code { background: var(--code-bg); padding: 2px 6px; border-radius: 4px; font-size: 14px; font-family: 'SF Mono', Monaco, monospace; } |
| 30 | + pre { background: var(--code-bg); border: 1px solid var(--border); border-radius: 8px; padding: 20px; overflow-x: auto; margin: 16px 0 24px; } |
| 31 | + pre code { background: none; padding: 0; font-size: 13px; line-height: 1.6; } |
| 32 | + strong { color: var(--text); } |
| 33 | + blockquote { border-left: 3px solid var(--accent); padding: 8px 20px; margin: 24px 0; color: #bbb; background: var(--card); border-radius: 0 8px 8px 0; } |
| 34 | + .cta { background: var(--card); border: 1px solid var(--accent); border-radius: 12px; padding: 24px; margin: 32px 0; text-align: center; } |
| 35 | + .cta h3 { margin: 0 0 8px; border: none; padding: 0; } |
| 36 | + .cta p { margin-bottom: 16px; } |
| 37 | + .cta a { display: inline-block; background: var(--accent); color: #000; font-weight: 700; padding: 12px 28px; border-radius: 8px; text-decoration: none; } |
| 38 | + footer { border-top: 1px solid var(--border); padding: 30px 0; text-align: center; color: #555; font-size: 13px; } |
| 39 | + </style> |
| 40 | +</head> |
| 41 | +<body> |
| 42 | + <header><div class="container"><a href="/">CodeFix Solution</a></div></header> |
| 43 | + <div class="container"> |
| 44 | + <article> |
| 45 | + <h1>Module 1 Deep Dive: Scraping ESPN's Scoreboard Endpoints for Every NBA Game in a Season</h1> |
| 46 | + <div class="meta">May 11, 2026 · 12 min read · Python, ESPN, Web Scraping, Data Pipeline</div> |
| 47 | + |
| 48 | + <p>Building a sports prediction model starts with one question: where does the data come from? For NBA games specifically, the answer is ESPN's scoreboard JSON endpoints — undocumented, free, no auth required, and remarkably stable across years. This is the deep dive on Module 1 of our Polymarket Bot Course: scraping every NBA game in a full season into a clean parquet file ready for downstream ML.</p> |
| 49 | + |
| 50 | + <h2>What you will end up with</h2> |
| 51 | + <p>By the end of this module, you will have a parquet file containing every NBA game from the 2024-25 season — 1,230 regular-season games plus the playoffs — with columns:</p> |
| 52 | + <ul> |
| 53 | + <li><code>game_id</code> — ESPN's stable identifier</li> |
| 54 | + <li><code>game_date</code> — ISO date</li> |
| 55 | + <li><code>home_team, away_team</code> — abbreviation</li> |
| 56 | + <li><code>home_score, away_score</code> — final</li> |
| 57 | + <li><code>status</code> — "STATUS_FINAL" for completed games</li> |
| 58 | + <li><code>venue, attendance</code> — metadata</li> |
| 59 | + </ul> |
| 60 | + <p>This dataset is the input to the rest of the course: feature engineering in Module 2, model training in Module 3, backtesting in Module 4, live deployment in Module 5.</p> |
| 61 | + |
| 62 | + <h2>The endpoint</h2> |
| 63 | + <p>ESPN's NBA scoreboard endpoint takes a date parameter:</p> |
| 64 | + <pre><code>https://site.api.espn.com/apis/site/v2/sports/basketball/nba/scoreboard?dates=20250215</code></pre> |
| 65 | + <p>The date format is YYYYMMDD with no separators. The response is JSON containing all games on that date. To scrape an entire season, you iterate every day from October (season start) to June (Finals end).</p> |
| 66 | + |
| 67 | + <h2>The naive scraper</h2> |
| 68 | + <p>The simplest version that works:</p> |
| 69 | + <pre><code>import requests |
| 70 | +from datetime import date, timedelta |
| 71 | + |
| 72 | +def fetch_day(d: date) -> dict: |
| 73 | + url = f"https://site.api.espn.com/apis/site/v2/sports/basketball/nba/scoreboard?dates={d.strftime('%Y%m%d')}" |
| 74 | + return requests.get(url, timeout=10).json() |
| 75 | + |
| 76 | +start = date(2024, 10, 22) |
| 77 | +end = date(2025, 6, 30) |
| 78 | +all_games = [] |
| 79 | +d = start |
| 80 | +while d <= end: |
| 81 | + data = fetch_day(d) |
| 82 | + for ev in data.get("events", []): |
| 83 | + all_games.append(ev) |
| 84 | + d += timedelta(days=1)</code></pre> |
| 85 | + <p>This runs in about 5 minutes for a full season. It also has every problem a production scraper needs to handle: no rate limiting, no retry, no deduplication, no error handling. Let's fix each one.</p> |
| 86 | + |
| 87 | + <h2>Defensive scraping</h2> |
| 88 | + <pre><code>import requests, time, logging |
| 89 | +from datetime import date, timedelta |
| 90 | + |
| 91 | +logger = logging.getLogger(__name__) |
| 92 | +SESSION = requests.Session() |
| 93 | +SESSION.headers.update({"User-Agent": "MyBot/1.0 (educational)"}) |
| 94 | + |
| 95 | +def fetch_day_safe(d: date, retries: int = 3) -> dict: |
| 96 | + url = f"https://site.api.espn.com/apis/site/v2/sports/basketball/nba/scoreboard?dates={d.strftime('%Y%m%d')}" |
| 97 | + for attempt in range(retries): |
| 98 | + try: |
| 99 | + r = SESSION.get(url, timeout=10) |
| 100 | + if r.status_code == 429: |
| 101 | + wait = int(r.headers.get("Retry-After", 2 ** attempt)) |
| 102 | + logger.warning(f"Rate limited on {d}, waiting {wait}s") |
| 103 | + time.sleep(wait) |
| 104 | + continue |
| 105 | + r.raise_for_status() |
| 106 | + return r.json() |
| 107 | + except (requests.RequestException, ValueError) as e: |
| 108 | + logger.warning(f"Fetch failed {d} attempt {attempt}: {e}") |
| 109 | + time.sleep(2 ** attempt) |
| 110 | + return {}</code></pre> |
| 111 | + <p>Three things this version does right: respects the <code>Retry-After</code> header on 429 responses, exponentially backs off on transient errors, and returns an empty dict rather than crashing on total failure. The next day continues to scrape.</p> |
| 112 | + |
| 113 | + <h2>Parsing one event into a row</h2> |
| 114 | + <pre><code>def parse_event(ev: dict) -> dict | None: |
| 115 | + try: |
| 116 | + comp = ev["competitions"][0] |
| 117 | + teams = comp["competitors"] |
| 118 | + home = next(t for t in teams if t["homeAway"] == "home") |
| 119 | + away = next(t for t in teams if t["homeAway"] == "away") |
| 120 | + return { |
| 121 | + "game_id": ev["id"], |
| 122 | + "game_date": ev["date"][:10], |
| 123 | + "home_team": home["team"]["abbreviation"], |
| 124 | + "away_team": away["team"]["abbreviation"], |
| 125 | + "home_score": int(home.get("score", 0)), |
| 126 | + "away_score": int(away.get("score", 0)), |
| 127 | + "status": ev["status"]["type"]["name"], |
| 128 | + "venue": comp.get("venue", {}).get("fullName", ""), |
| 129 | + "attendance": comp.get("attendance", 0), |
| 130 | + } |
| 131 | + except (KeyError, IndexError, StopIteration) as e: |
| 132 | + logger.warning(f"Parse failed for event {ev.get('id')}: {e}") |
| 133 | + return None</code></pre> |
| 134 | + <p>The try/except is not optional. ESPN's response shape varies slightly between regular season, playoffs, postponed games, and forfeit games. A single malformed event should not crash the entire scrape.</p> |
| 135 | + |
| 136 | + <h2>Deduplication</h2> |
| 137 | + <p>The same game can appear in two days' scoreboards if it spans midnight UTC. Postponed games can have two game_ids. Always deduplicate by game_id before persisting:</p> |
| 138 | + <pre><code>seen = set() |
| 139 | +unique_games = [] |
| 140 | +for ev in all_games: |
| 141 | + parsed = parse_event(ev) |
| 142 | + if parsed is None: |
| 143 | + continue |
| 144 | + if parsed["game_id"] in seen: |
| 145 | + continue |
| 146 | + seen.add(parsed["game_id"]) |
| 147 | + unique_games.append(parsed)</code></pre> |
| 148 | + |
| 149 | + <h2>Politeness: rate-limit yourself</h2> |
| 150 | + <p>ESPN does not publish a rate limit, but their endpoints have throttled us at sustained rates above ~2 requests per second. Add a small sleep between requests:</p> |
| 151 | + <pre><code>import time |
| 152 | + |
| 153 | +DELAY_S = 0.5 # 2 req/sec |
| 154 | + |
| 155 | +while d <= end: |
| 156 | + data = fetch_day_safe(d) |
| 157 | + for ev in data.get("events", []): |
| 158 | + all_games.append(ev) |
| 159 | + d += timedelta(days=1) |
| 160 | + time.sleep(DELAY_S)</code></pre> |
| 161 | + <p>For a full season (about 250 days October to June), this adds about 2 minutes of total scrape time. Worth it to never get rate-limited.</p> |
| 162 | + |
| 163 | + <h2>Persisting to parquet</h2> |
| 164 | + <pre><code>import pandas as pd |
| 165 | + |
| 166 | +df = pd.DataFrame(unique_games) |
| 167 | +df = df[df["status"] == "STATUS_FINAL"] # Drop in-progress and postponed |
| 168 | +df = df.sort_values(["game_date", "game_id"]).reset_index(drop=True) |
| 169 | +df.to_parquet("nba_games_2024_25.parquet") |
| 170 | +print(f"Saved {len(df)} completed games")</code></pre> |
| 171 | + <p>Parquet is the right format for this dataset because it is columnar (fast to read specific columns) and compressed (small on disk). A full NBA season is around 1.5 MB on disk.</p> |
| 172 | + |
| 173 | + <h2>Going beyond scoreboard: the summary endpoint</h2> |
| 174 | + <p>The scoreboard gives you final scores. For modeling, you typically want play-by-play. ESPN's summary endpoint returns it:</p> |
| 175 | + <pre><code>def fetch_summary(game_id: str) -> dict: |
| 176 | + url = f"https://site.api.espn.com/apis/site/v2/sports/basketball/nba/summary?event={game_id}" |
| 177 | + return requests.get(url, timeout=15).json() |
| 178 | + |
| 179 | +summary = fetch_summary("401705412") |
| 180 | +plays = summary.get("plays", []) |
| 181 | +print(f"{len(plays)} plays in this game")</code></pre> |
| 182 | + <p>Each play has period, clock, score after the play, type, and (sometimes) coordinates. This is the input to in-play win-probability models. We cover building those in Module 3.</p> |
| 183 | + |
| 184 | + <h2>What can go wrong</h2> |
| 185 | + <ul> |
| 186 | + <li><strong>Time zones.</strong> ESPN's <code>date</code> field is UTC; some games on the West Coast cross midnight UTC and appear on two consecutive scoreboard days. Dedup catches this.</li> |
| 187 | + <li><strong>Postponements.</strong> Postponed games sometimes get a new game_id when rescheduled. Sometimes they keep the original. Both happen. Dedup catches one but not the other — manual review is required for season-end completeness.</li> |
| 188 | + <li><strong>Cancellations.</strong> Some games are cancelled outright. Their <code>status</code> will be <code>STATUS_CANCELLED</code>. Filter by status before modeling.</li> |
| 189 | + <li><strong>Schema changes.</strong> ESPN occasionally changes a field's name or type. We have not seen a breaking change in years, but assume it can happen and write parsers defensively.</li> |
| 190 | + </ul> |
| 191 | + |
| 192 | + <h2>The bottom line</h2> |
| 193 | + <p>A clean NBA season scrape is a foundational dataset for any prediction model. ESPN's free JSON endpoints make it easy. The 100 lines of Python in this post turn into a parquet file you can build the rest of your pipeline on. Module 2 (feature engineering) and Module 3 (model training) both consume this file directly.</p> |
| 194 | + |
| 195 | + <div class="cta"> |
| 196 | + <h3>The full Polymarket Bot Course</h3> |
| 197 | + <p>Six Jupyter modules: ESPN scraping, Elo, win-probability models, backtesting, live bot, deployment. $49 standalone or included with every ZenHodl API plan.</p> |
| 198 | + <a href="https://zenhodl.net/products" target="_blank" rel="noopener">Get the course</a> |
| 199 | + </div> |
| 200 | + </article> |
| 201 | + </div> |
| 202 | + <footer><div class="container">CodeFix Solution · Developer tutorials and code fixes · <a href="https://zenhodl.net">Powered by ZenHodl</a></div></footer> |
| 203 | +</body> |
| 204 | +</html> |
0 commit comments