Skip to content

operations.md

Latest commit

 

History

History
201 lines (166 loc) · 10.2 KB

operations.md

File metadata and controls

201 lines (166 loc) · 10.2 KB
Raw

Operations runbook

The short list of things the operator actually has to do, in one place. Everything here assumes a Vercel deployment + Neon Postgres as described in the README — and there are now two of those (mainnet and testnet, each with its own database, secrets and knobs; see docs/deployments.md). Each section below applies to both unless it says otherwise; anything involving minting or the faucet is testnet-only.

Database migrations

Schema changes ship as idempotent SQL in scripts/migrate.mjs. After any deploy that touches lib/db/schema.ts, run the migration immediately — the app tolerates missing columns on most read paths (falls back to defaults), but full-row selects can error until the migration lands.

As the superadmin (ADMIN_EMAIL account), in a signed-in browser tab:

fetch('/api/admin/migrate', { method: 'POST' }).then(r => r.json()).then(console.log)
// → { ok: true }

Or locally with the production connection string: DATABASE_URL=postgres://… pnpm db:migrate (mind WHICH database that URL points at — the admin endpoint exists precisely because a local run once targeted the wrong Neon branch).

Desktop app releases

Tag-based, built by .github/workflows/desktop-release.yml on real Windows/macOS runners:

  1. Bump the version in desktop/src-tauri/tauri.conf.json and desktop/src-tauri/Cargo.toml (keep them equal).
  2. GitHub → Releases → "Draft a new release" → type a new desktop-vX.Y.Z tag targeting main → Publish. The tag push builds and attaches the installers to that release automatically (published directly, no draft).
  3. Manual workflow runs (Actions tab) are for TEST builds — they produce a draft, and re-running against an existing published tag re-uploads assets without updating the release date. Prefer tags for anything users will see.

Builds are unsigned: Windows SmartScreen and macOS Gatekeeper will warn. desktop/README.md documents the exact user workaround — say it up front when sharing links.

Watching production

  • Vercel → project → Logs, or filter level=error. Known-noise: the pg SSL mode deprecation warning on every cold start (harmless).
  • Settlement is self-healing: if an approve/refund dies mid-flight (RPC 429s), the stuck-settlement sweep re-drives it the next time anyone loads the Jobs or Delegate page. JOB_AUTO_APPROVE_INCOMPLETE / JOB_REPOST_FAILED platform-feed events are the two cases that DO need a human (funds moved but bookkeeping failed — backfill manually).
  • On-chain reads are batched (Multicall3) and cached ~4s per warm lambda. If Alchemy 429s return, check for a new unbatched read path before paying for a bigger RPC plan.

Spending & abuse knobs (env)

See .env.example for the full commented list. The ones that gate money and abuse: WALLET_MAX_TX_USD, WALLET_DAILY_CAP_USD (platform defaults; users override per-account in Worker Console), WALLET_CAP_HARD_MAX_USD, AUTO_APPROVE_MAX_BOUNTY_USD, REGISTER_HOURLY_MAX_USERS, MAX_AGENTS_PER_ACCOUNT. Never set a cap to an empty string — unset means default, empty once meant "$0, block everything" before the parser was hardened.

Office sessions (docs/office-sessions.md) add one real-money flag and three clocks. OFFICE_SESSION_ALLOW_REAL_MONEY=true lets a policy decision release an escrow on a real-money deployment; unset, such a task waits for the owner's click (the LINEAGE / REVIEW_STAKE shape). The clocks — OFFICE_SESSION_HEARTBEAT_TIMEOUT_MS, OFFICE_SESSION_PICKUP_TIMEOUT_MS, OFFICE_SESSION_RUN_TIMEOUT_MS — are bounded (30 s to an hour, 30 s to a day, a minute to a day) and exist so an end-to-end run can prove crash recovery in a minute rather than five; production leaves them unset. The officeSessions ops step runs on the cron only, after delegations.

Settlement heartbeat (CRON_SECRET)

Settlement is three-layered: grading + payout at submission time (the callback), opportunistic sweeps on page reads, and the background heartbeat — GET /api/cron/settle, called by .github/workflows/settle-heartbeat.yml on a schedule. The heartbeat is what re-drives payouts that failed transiently (RPC 429) while nobody has a tab open.

CRON_SECRET must hold the SAME value in two places: a Vercel Production env var (remember: env changes only apply on the NEXT deployment) and a GitHub Actions repository secret. With it missing, the workflow skips silently (by design — no red X spam) and the endpoint answers 503; set it before onboarding real users. Verify with: curl -H "Authorization: Bearer $CRON_SECRET" .../api/cron/settle → {"ok":true,...}. The secret only authorizes triggering settlement work, never moving funds anywhere new.

Job faucet (OPT-IN)

Off by default. The board's standing demand now comes from the real backlog (i18n / documentation jobs below); auto-posting synthetic practice exercises next to real work made the board read as clutter. Set FAUCET_ENABLED=true to bring it back (e.g. a workshop/demo where guaranteed instant work matters more than the board's story) — testnet only: on a real-money deployment FAUCET_ENABLED trips the faucet-enabled blocker in the mainnet guard, which refuses every money path, not just the faucet — and use Admin → Access Control → Clear practice jobs to cancel any still-open exercises (escrow refunds on-chain).

When enabled: a house agent ("Job Faucet", owned by the password-less faucet@handsel.internal account) keeps FAUCET_TARGET_OPEN (default 3) small Python-test jobs open, bounded by FAUCET_MAX_PER_DAY (default 15). Grading is mechanical (no LLM dependency), escrow is self-funded via the testnet mint when the wallet drops under $20 (testnet only — on mainnet minting is blocked and the house wallet takes a real USDC transfer), and ticks ride the settlement heartbeat + the jobs-page read path (10-min in-memory throttle). Hard kill switch on top: FAUCET_DISABLED=true. Every template's reference solution is executed against its own asserts in the test suite — a faucet job with broken tests would poison worker credit scores, so the catalog is proven solvable in CI.

Translation is done directly, not bought

npm run i18n:translate fills every missing key in lib/i18n-dict.ts from the same model the market would have hired, in one command, for the price of an API call. --check reports the gaps without a key; --add ja --label 日本語 adds a whole locale. The runtime twin is /api/admin/i18n (BYOK), which writes to the i18nString overrides instead of the file.

This used to be dogfood demand and no longer is. i18n jobs and documentation-translation jobs escrowed real bounties for output the operator could already produce inline, and the grader was an LLM reading a translation — the weakest verification in the system (graderWeight: llm-review 0.6) applied to the one class of work that needed no market at all. On a testnet with mintable USDC that was a harmless way to keep a board populated. With real money it is the house paying itself to look busy.

What went with them: lib/i18n-jobs.ts, lib/docs-jobs.ts, the two admin cards, and restockBoard — whose only supply was the renewable i18n backlog. The board no longer refills itself, and that is the intended behaviour. An empty board is now a true statement about demand instead of a gap the house papers over.

Test-suite jobs (Admin → Access Control → Board curation) are the dogfood source that survived, graded by MUTATION TESTING — fully mechanical, no LLM: the worker submits Python asserts for a published function contract; grading runs them against a hidden correct reference (must pass) and several hidden buggy variants (must fail every one), all on the same platform-runtime /grade sandbox as code jobs. Every reference and mutant in the catalog (lib/test-suite-jobs.ts) is verified against real Python in the unit suite. A winning suite is a verified test battery for a future auto-graded job template — harvest it from the job card. Each contract posts at most once ever. Fail-safe: an unavailable sandbox run yields passed:null (manual review), and a suite that fails the reference is failed with an explicit "your tests contradict the contract" message.

Weekly contest (CONTEST_PRIZE_USD)

OFF by default. Setting CONTEST_PRIZE_USD (e.g. 20) makes /live show a "Weekly contest" panel: live standings for the current Mon→Mon UTC window, computed from the same JOB_COMPLETED events as every earnings figure. The prize is a real-money contest paid manually by the operator (gift card / PayPal — a contest, not a payment rail): at week's end, read the panel, pay the top earner, announce it. Do not set the env var unless you intend to pay — an advertised prize nobody funds is worse than no contest. Unset the var to turn the panel off instantly.

Solana devnet loop (operator-only)

POST /api/admin/solana-loop, authorized with CRON_SECRET, drives the whole devnet cycle on the deployed program: post → accept → submit → approve → withdraw. It spends devnet SOL and mints devnet test USDC; there is no mainnet path.

stop_after=<step> truncates the loop, which is how the public board is left with an Open job for a screenshot or a recording — the default run finishes every job and correctly leaves an empty board. The plan is computed before the first spend, so stopping at post never funds a worker that will never accept, and an unrecognised step name is rejected rather than treated as "run everything" (lib/solana-loop-plan.ts). The response echoes stopped_after and a note naming what was deliberately left unfinished, so a half-run loop is never mistaken for a stuck one.

curl -X POST -H "Authorization: Bearer $CRON_SECRET" ".../api/admin/solana-loop?stop_after=post"

Addresses and the on-chain verification are in solana-port.md.

Tests

pnpm test (vitest) runs the unit/regression suite — money-adjacent pure logic: cap parsing, planner guardrails, TaskSpec normalization, settlement retry classification. CI (.github/workflows/ci.yml) runs typecheck + tests on every push/PR to main. Every production incident that gets fixed should land with a test that pins it — the suite IS the incident log.