A YachtWorld sailboat listing scraper with a Next.js web frontend for browsing, filtering, and annotating results.
Scrapes used sailboat listings from YachtWorld using Playwright with stealth plugins to bypass Cloudflare detection. Results are stored in a local SQLite database and presented in a filterable, sortable table with annotation tools for evaluating boats. Includes a one-click research feature that pulls sailboatdata.com specs, professional reviews, and owners forum posts for each boat model.
- Web-based scraper control -- Configure search parameters (price range, length, condition) and launch scrapes from the browser
- Filterable results table -- Sort by any column, resize columns, expand rows for details
- Tri-state filters -- Include or exclude specific manufacturers and states; range sliders for price, length, and build year
- Annotations -- Thumbs up/down, favorites, free-text notes, and property checkboxes (double-ender, tiller steering, furling main, etc.)
- Search -- Full-text search across listing name, seller, location, manufacturer, state, and notes
- Indicator icons -- At-a-glance icons on each row for favorites, thumbs, notes, and flagged properties
- Light/dark mode -- System-aware theme toggle with manual override
- Boat research -- One-click research pulls sailboatdata.com specs, professional reviews, and owners forum posts for each boat model with human-in-the-loop selection of the best matches
- Cloudflare bypass -- Persistent browser profile retains session cookies across scrapes
- Node.js >= 18
- npm
- A display environment (the scraper runs a visible browser window by default)
# Clone the repository
git clone https://github.com/your-username/ClaudeYacht.git
cd ClaudeYacht
# Install dependencies
npm install
# Install Playwright browsers
npx playwright install chromium
# Initialize the database
npx prisma db push
# Start the dev server
npm run devThe app will be available at http://localhost:3000.
No environment variables are required. The app uses sensible defaults:
| Setting | Default | Location |
|---|---|---|
| Database | data/yacht.db (SQLite) |
prisma.config.ts |
| Scrape output | data/*.jsonl |
lib/data-dir.ts |
| Browser profile | .browser-profile/ |
scripts/scrape_yachtworld.js |
| Dev server port | 3000 | Next.js default |
The data/ directory is created automatically on first use and is gitignored.
- Navigate to Scrape in the top nav
- Set price range, length range, and condition
- Toggle exclusions (ketch/yawl, multihull) as desired
- Click Start Scrape
- Monitor progress in the status panel -- results auto-ingest on completion
# With defaults (used sailboats, $10k-$100k, 37-42ft)
node scripts/scrape_yachtworld.js
# With custom parameters
node scripts/scrape_yachtworld.js \
--priceMin 20000 \
--priceMax 80000 \
--lengthMinFt 35 \
--lengthMaxFt 45 \
--condition used \
--excludeKetchYawl true \
--excludeMultihull true \
--outputFile data/custom_scrape.jsonlAfter a CLI scrape, ingest the results via the Scrape page in the web UI.
Navigate to Results to view ingested listings. Use the filter icon to open the filter rail, where you can:
- Include/exclude manufacturers or states (click to cycle: neutral -> include -> exclude)
- Adjust range sliders for price, length, and build year
- Toggle favorites-only or hide thumbs-down listings
- Search by any text field
Click a row to expand it and access annotations: thumbs up/down, favorite star, property checkboxes, and a notes field.
- Expand a listing row and click Research
- The system searches sailboatdata.com for the boat model -- if multiple matches are found, select the correct one
- Professional reviews are found via DuckDuckGo -- select the most relevant results
- Owners forum posts are found the same way -- select the most relevant discussions
- View the consolidated research panel with specs, reviews, and forum links
Research is cached per boat model (manufacturer + class + year range), so multiple listings of the same model share research data automatically.
- Clear all listings: Use the "Clear All Listings" button on the Scrape page
- Browse raw data: Run
npx prisma studioto open the database GUI - Schema changes: Edit
prisma/schema.prisma, then runnpx prisma db push
npm test # Fast tests — pure functions, API routes, components (~2s, no browser)
npm run test:watch # Watch mode for TDD
npm run test:e2e # E2E browser tests (launches Chromium)
npm run test:all # Both tiers
npm run test:coverage # Fast tests + V8 coverage reportTests are co-located with source files (*.test.ts / *.test.tsx). The build script runs fast tests before building (vitest run && next build).
This project follows test-driven development. When adding a new feature or fixing a bug:
- Write a failing test first (
npm run test:watchfor instant feedback) - Implement the minimum code to pass the test
- Refactor while keeping tests green
Test files live alongside source files (e.g., lib/foo.ts → lib/foo.test.ts). Use the factory functions in test/fixtures.ts for test data. See lib/filters.test.ts for a comprehensive example.
The scraper uses playwright-extra with puppeteer-extra-plugin-stealth to avoid bot detection. It opens a persistent browser context (stored in .browser-profile/) to retain Cloudflare cookies across sessions, then paginates through search results extracting listing data from data-ssr-meta attributes.
JSONL files are parsed line by line. Each listing's build year is extracted from the listing name (e.g., "2006 Hunter 38" -> year: 2006, name: "Hunter 38"). Listings are deduplicated by their unique YachtWorld URL. The ingestion runs automatically after each successful scrape.
All filtering and sorting happens in the browser using pure functions in lib/filters.ts. The dataset is small enough that server-side filtering isn't needed. Filter facets are cross-filtered: each dimension's available options are computed by applying all other filters except its own, so active filters don't hide their own options.
Notes, thumbs up/down, favorites, and property checkboxes are saved to the database via optimistic updates. The UI updates immediately and reverts if the server request fails.
Clicking "Research" on a listing triggers an orchestrated pipeline in lib/research.ts:
- Sailboatdata.com specs -- Playwright navigates to sailboatdata.com, searches for the boat model, and scrapes specifications (dimensions, displacement, sail area, etc.). If multiple models match, the user selects the correct one via the research panel.
- Professional reviews -- DuckDuckGo is searched for professional reviews of the boat model. The user selects the most relevant results from the candidates.
- Owners forum posts -- Another DuckDuckGo search finds owners forum discussions. The user again selects the best matches.
Research progress is streamed to the browser via Server-Sent Events, so the UI updates in real-time as each step completes. Results are cached in a ModelResearch table keyed by manufacturer, class, and year range -- so if you research one "Hunter 38", all other Hunter 38 listings share the same data. A SailboatDataMapping table remembers which sailboatdata.com model corresponds to each boat name, eliminating repeat selections.
An AsyncMutex ensures only one Playwright browser session runs at a time, preventing resource exhaustion when multiple research requests arrive concurrently.
app/ # Next.js App Router (pages + API routes)
api/ # REST endpoints for scrape, ingest, listings, research
results/ # Results page with filters, search, table
scrape/ # Scrape configuration page
components/ # React components
ui/ # Radix-based primitives (button, slider, switch, etc.)
research-panel.tsx # Research UI with SSE status and human-in-the-loop selection
lib/ # Business logic (types, filters, ingest, scraper, research, utils)
research.ts # Research orchestration (Playwright, DuckDuckGo, SSE)
scripts/ # Standalone scraper (CommonJS, runs as child process)
prisma/ # Database schema (Listing, ListingResearch, ModelResearch, SailboatDataMapping)
test/ # Test infrastructure (setup, fixtures, custom render)
e2e/ # Playwright E2E browser tests
data/ # Gitignored: JSONL files + SQLite database
- Next.js 15 + React 19
- Prisma 6 (SQLite)
- Playwright + stealth plugins
- TanStack Table
- Tailwind CSS v4 + Radix UI
- Lucide Icons
- Cheerio (HTML parsing for research scraping)
- Vitest + React Testing Library
- Playwright Test (E2E)
ISC