Add Hackrawl: hackathon discovery, Q&A, and registration agent - #175
Draft
fetchady wants to merge 35 commits into
Draft
Add Hackrawl: hackathon discovery, Q&A, and registration agent#175fetchady wants to merge 35 commits into
fetchady wants to merge 35 commits into
Conversation
Crawls cerebralvalley.ai hackathon listings using crawl4ai (Docker), parses Next.js RSC payloads to extract structured event data, enriches with ASI:One LLM extraction, and upserts to Supabase with deduplication on slug. Includes crawl logging and live progress monitoring. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…crawl When a Cerebral Valley event links out to a known third-party platform, the crawler now detects the URL, fetches that page via crawl4ai, and uses ASI:One LLM to extract additional fields (prizes, tracks, sponsors, schedule, deadlines). Results stored in external_url, external_source, external_data columns added to the events table. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Discovered the public REST API (api.cerebralvalley.ai/v1/public/event/pull) which exposes the full event catalogue with pagination. Replaced HTML crawl for listings with direct API calls; pipeline now ingests all 2,700+ events (including past events back to 2024) in ~9 minutes with zero browser overhead. Added venue, status, cv_event, featured_start/end_time columns to events table. Phase 2 detail crawl (--detail flag) still runs for platform hackathons. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
crawl_deep.py paginates through events with external_url but no external_data, crawls each page via crawl4ai + ASI:One LLM, and patches external_data in-place. Fully resumable on kill/restart. Includes per-platform delay tuning, ETA display, and graceful SIGINT. Also adds fetch_uncrawled_events, count_uncrawled_events, and update_external_data helpers to db.py. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Sources added: Devpost (REST API, 13k+ hackathons), MLH, HackerEarth (HTML scraping), Devfolio, ETHGlobal, DoraHacks, Unstop (browser automation). - crawl_devpost.py: paginated API ingestion - crawl_html.py: LLM extraction from server-rendered HTML (MLH, HackerEarth) - crawl_browser.py: JS-rendered pages with scroll simulation + LLM extraction - crawl_runner.py: dispatches to the right crawler per source via --source flag - config.py: unified source registry with crawler type, URL, and notes Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… corrections - db.py: coerce empty string to NULL for timestamp columns (fixes Devfolio errors) - crawl_browser.py: extract events from links.internal first before LLM fallback; cap LLM content at 6000 chars; add _fetch_result returning full crawl4ai result - crawl_html.py: same links-first strategy; fix MLH URL to www.mlh.com; add _fetch_result - crawl_devpost.py: fix total_pages calculation from total_count / per_page Results: ETHGlobal 84, DoraHacks 24, Devfolio 4, HackerEarth 13, MLH 2 stored. Devpost ingestion running (8,850/13,472 at commit time). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
crawl_mlh.py parses event name, dates, location, format (in-person/digital), and tags (diversity, high-school) directly from link text on the MLH season listing pages. Also crawls events.mlh.io detail pages for descriptions. Covers both 2025 and 2026 seasons (250 + 162 unique events). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Allows targeting deep crawl at events from a specific crawler (e.g. --event-source mlh) rather than just by external platform. Also adds event_source_filter param to fetch_uncrawled_events and count_uncrawled_events in db.py. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
recommend/filter.py — hard SQL filters (date, location, keywords, type, registration) recommend/score.py — weighted soft scoring (prize, upcoming, keyword match, data quality, featured) recommend/rerank.py — ASI:One reranking of top 30 with per-event explanations recommend/engine.py — orchestrator with recommend() and format_results() entry points Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
agent/parse.py — ASI:One parses natural language → structured intent + prefs agent/lookup.py — direct event lookup by name/slug with word-overlap scoring agent/stats.py — pre-defined aggregate queries narrated by ASI:One agent/qa.py — orchestrator + CLI (python scripts/agent/qa.py) Also fixes in recommend/filter.py: - Combined all OR conditions into one .or_() call (supabase-py ANDs multiple) - Registration open filter moved post-fetch to correctly handle NULL values - Short keyword filter threshold changed >= 2 (was > 2, excluded "AI") - Online filter now included in the single combined OR block Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The link text format for in-person events is "City, StateEvent Name", causing city to be prepended to the title. Fix strips the city prefix when it matches the parsed city field. Also applied a one-time SQL migration to fix the 384 already-stored events with mangled titles. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
agent/profile.py — UserProfile dataclass with all registration fields + auth agent/answer_gen.py — ASI:One generates answers to open-ended questions from profile agent/platforms/cerebralvalley.py — Playwright handler for CV platform events (login + form fill) agent/platforms/generic.py — browser-use handler for all other platforms (Luma, Eventbrite, etc.) agent/register.py — orchestrator: DB lookup → answer gen → platform routing → browser agent/create_profile.py — interactive CLI to create ~/.hackathon_profile.json Usage: python scripts/agent/register.py --event aiewf-hackathon-2026 --dry-run python scripts/agent/register.py --url https://lu.ma/some-hackathon Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
agent.py — uAgent with chat protocol: search/lookup/register/stat intents,
routes to recommend engine, Q&A agent, and registration agent
ui_cards.json — editable UI card templates (welcome, event_list, event_item,
event_detail, registration_confirm, badge variants)
scripts/agent/cards.py — fills ui_cards.json templates with real event data
agent_README.md — Agentverse listing page with usage examples and platform stats
.env.example created separately — add AGENT_MAILBOX_KEY + ASI_ONE_API_KEY to .env to run
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Rename everywhere: HackathonAgent → Hackrawl - Use register_chat_agent() on startup (matches deploy-agent-on-av pattern) - Remove unsupported handle_messages_concurrently arg (uagents 0.22.10) - Agent registered on Agentverse and mailbox confirmed active - Address: agent1qg9rkx0qccss92ung34fsrvyygngsf6ganusmt808vsugusk4kvku84s9r4 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
parser: explicitly instruct LLM never to put event-type words (hackathon/hackathons/event/find/upcoming) into keywords — these should set hackathon_only=true instead. Blank keywords now means "no topic filter" not "search for the word hackathons". filter: strip stop words from keywords list before building the DB OR query so generic words can't silently kill all results. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
All config now lives in a single .env file. .llm.env is deleted. Fixed SUPABASE_KEY which was truncated/wrong in .env (was missing role/iat/exp fields causing 401 on every DB query). Updated agent.py, config.py, db.py to load from .env only. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Use correct metadata keys (card_protocol_version, card_kind, card_payload) instead of custom 'card' key - Remove text fallback from card messages to avoid duplicate content - Send welcome card on first text message since StartSessionContent is not always fired - Strip _comment fields and badge_variants from ui_cards.json to prevent validation errors Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Wrap all Supabase operations with exponential backoff retry (5 attempts, 5s base delay doubling up to 120s). On network/DNS errors the client is reset so a fresh connection is made on the next attempt. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Parse @agent_address prefix from card button selections sent by ASI:One - Add _parse_selection() to route button clicks to correct action handlers - Fix _fill() bug where multiple placeholders in one field only applied the last - Remove welcome card from first-message flow; only show on greetings or StartSession - Remove Back and Open Website buttons (broken via ASI:One); add URL as text link - Add more event detail fields: organizer, team size, registrations count, venue - Remove action arrow from event list items - Fix BrowserConfig -> BrowserProfile import for updated browser_use package - Remove _comment fields and badge_variants from ui_cards.json Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Devpost hackathon detail pages were mostly returning fetch_failed from a prior crawl4ai outage; those rows have been reset and are being re-crawled separately (not part of this commit).
Drops the langchain_openai dependency, which was only used here — browser-use ships its own OpenAI-compatible client under browser_use.llm.
…rotected pages - Disable use_vision: screenshots were being serialized as literal text tokens against asi1-mini, blowing past its 262k context window before the agent could take any action (reproduced on MLH's event page). - Cap iframe/DOM payload size as a defensive limit for other bloated pages. - Add a bot-check watchdog that detects CAPTCHA/Cloudflare Turnstile pages and stops the agent after a few repeated attempts instead of burning all 25 steps in a blind retry loop when sign-in is blocked by anti-bot checks. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…e overreach - Reframe task instructions: RSVP/Join submission + confirmation is the actual success condition. Post-submit sign-in/OTP prompts (e.g. Luma's "verify your email to manage your registration") are optional and must not be chased or treated as failure, since the agent has no way to read a real OTP from the user's inbox and was previously guessing at codes. - Add explicit "already registered"/"already RSVP'd" confirmation phrases and a check-before-reclicking rule, since the agent was blindly re-clicking a submit button 12+ times even after a clear confirmation was already visible. - Flip _check_result priority to trust the agent's own is_successful() verdict before falling back to the judge's is_validated(). The judge scores against literal instruction-following (every optional field filled) rather than whether registration actually went through, and was observed failing runs that had already succeeded and were independently confirmed via email. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Per-user profiles now live in the hackathon_profiles table (RLS-locked to service_role), keyed by uAgents sender address, so the deployed chat bot can serve multiple concurrent users without profile data leaking between them. Local JSON storage is kept as a fallback for solo CLI usage. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Slices results by offset/limit so "View all" can page through the full result set as the same card UI, and hides the button once there's nothing left to page to. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Fields like T-shirt size or dietary restrictions were previously either guessed silently by the browser agent or caused it to loop indefinitely retrying the same failed click. The agent now fills what it can, batches every field it can't answer from the profile into a single stop-and-ask report instead of guessing, and persists the human's answers to the profile so future registrations never ask again. - platforms/generic.py: stricter leading-sentinel matching (a prior substring match falsely fired on unrelated failures), a batched FIELD_INPUT_REQUIRED protocol covering the whole form in one pass, a resumed-step-budget fix (agent.run(max_steps=N) treats step count as cumulative, so a resumed run needs a cap above steps already taken, not the same N), a stuck-select watchdog that detects an action repeating with zero DOM change and forces a switch to keyboard navigation, "already registered" detection, and human-readable result messages instead of a raw AgentHistoryList repr. - register.py / agent.py: thread an interactive/non-interactive mode through so the CLI can block-and-prompt while the chat UI instead pauses the live browser session (kept alive) and asks via a real ASI:One form card covering every missing field at once, resuming the same session once the human answers. - requirements.txt: pin uagents-core>=0.4.6 for the official FormCardPayload form-card support used above. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
New chat users had no way to populate their Supabase profile short of the CLI, so registration silently proceeded with an empty profile. Registration now checks for a usable profile (name + email) before continuing and shows an onboarding form instead, resuming the original action once it's saved. Also adds an "edit profile" text shortcut to review/update it anytime. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
… a source of truth looking_for_job/needs_visa defaulted to false in both the dataclass and the DB schema, making "never asked" indistinguishable from an explicit "no" -- registration answers were silently submitting "no" for users who were never asked. Both fields are now Optional[bool] (None = unknown), the DB columns allow NULL with no default, and answer_gen.py only auto-answers from these when a real value is set, otherwise leaving the question for the browser agent's DO NOT GUESS/ask-the-human path instead of assuming "no". Also collapses local-JSON-vs-Supabase into a single source of truth: load_profile()/save_profile() now always hit Supabase, keyed by the given agent_address or a fixed LOCAL_AGENT_ADDRESS for solo CLI usage, instead of silently falling back to an independent (and easily stale) local file. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…aise step budget
Live dry runs turned up the browser-use agent fabricating values for
unmatched fields (company guessed from email domain, invented project
descriptions, a picked "harmless" dropdown option) and, separately,
attempting to click submit through JS `evaluate` even after stating it
would report FIELD_INPUT_REQUIRED instead. The task prompt's NEVER GUESS
rule had a loophole ("...unless you can reasonably infer it from
role/skills/bio") and step 6 actively told the agent to improvise answers
for anything left over -- both removed. The rule is now a strict per-field
checklist (exact Standard-field or Custom-answer match, or leave blank and
report), the submit button is off-limits by any method once any field is
unresolved, and a stuck-field cap ("3 total attempts on the same element,
by any method, then stop") replaces the previous soft guidance that wasn't
being followed.
Also adds detection for closed/no-longer-accepting-registrations events:
previously the agent would wander for 30+ steps looking for a workaround
and the user just got a generic "wasn't able to finish" message with no
real reason. It's now checked immediately after page load via a
REGISTRATION_CLOSED sentinel, short-circuiting straight to a clear message.
Raised the default step budget from 25 to 40 -- a single unreliable custom
widget (e.g. a toggle checkbox the agent can't visually confirm) could
burn the whole budget before ever reaching FIELD_INPUT_REQUIRED, so the
user never got asked about the real missing fields at all.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Prompt-only "don't submit" instructions to the browser-use agent proved unreliable during testing -- it circumvented them at least twice (once via a JS evaluate() click, once with a direct click contradicting its own stated plan) against a real, live registration page. This installs a CDP init script that blocks submission at the DOM-API/network level (native form submit, fetch, XMLHttpRequest, sendBeacon) regardless of how it's triggered, so dry runs against real event pages can no longer submit no matter what the agent decides to do. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…facts Previously every unmatched registration field got surfaced to the human, even open-ended, forward-looking questions about the event itself (e.g. "what would you build/break/prove?", "which topic interests you most?") that don't require any real fact about the person. Added classify_and_generate_field_answers() to split unmatched fields into "opinion" (safe for the AI to answer authentically from role/skills/bio, same idea as the existing pre-crawled open-ended question generation) vs "personal_fact" (company, role, tools used, dietary needs, etc. -- always left for the human). Feeding the generated answers into the SAME already-running browser session via a mid-task "resume" message proved unreliable in testing (the agent sometimes ignored values it was just given and re-reported them as unknown). Instead, register_for_event() now accepts extra_answers merged in before the browser ever starts, and agent.py closes the stuck session and starts one fresh attempt with them included from the first prompt -- the same reliable path already used for pre-crawled question answers. Also fixes two related bugs surfaced while testing this: - _parse_field_requests() failed outright (silently dropping every field) when the agent appended trailing prose after its JSON array instead of stopping at the array as instructed; now extracts just the bracket-matched array regardless of what follows it. - _apply_field_answers_and_resume()'s prompt had no branch for "some fields got answers, others are still genuinely unknown," so it could tell the agent to submit when it shouldn't; now it re-applies the DO NOT GUESS protocol and re-reports remaining fields with their real details instead of empty placeholders. - missing_fields_form() truncated each field's question to an 80-character placeholder (which also vanishes once the user starts typing) -- now uses the full question text as the persistent field label instead. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Merges origin/main to pick up recent contributor changes, applies ruff format/lint fixes across the browser-use agent's Python files, resolves a few real mypy findings (missing annotations, Optional handling, Callable typing), and adds an empty scripts/__init__.py so mypy stops treating scripts/agent/ and the root agent.py as the same module.
CI installs the latest ruff, whose newer default rule set (BLE001, S110, B005, SIM102, etc.) is much broader than the classic E4/E7/E9/F defaults this repo's other ~200 files with `except Exception` rely on. Scope this directory's ruff config to those defaults instead of rewriting exception handling repo-wide. Also re-sorts imports and restores noqa: E402 markers dropped by the previous auto-fix pass.
CI's mypy (no deps installed besides mypy itself) was resolving `import supabase` in scripts/db.py to our own Browser-based-agents/browser-use/supabase/ migrations directory as an implicit namespace package, then failing with "has no attribute create_client". Root mypy.ini turns namespace package resolution off repo-wide, which is the safer default anyway.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Test plan