db-task-set.md

DB Task Set

The DB task set is the central provider-neutral database benchmark catalog. Tasks live under category directories, and each task has its own prompt, verifier, optional attacker, and cleanup script.

Categories

• access-control: who can read, write, update, or delete which rows or relations.
• integrity: database invariants, lifecycle rules, constraints, triggers, state transitions, and server-maintained state.
• query: correct reporting, aggregation, search, pagination, and query-facing relations. Query tasks may report query performance metrics in verifier details.
• vector: embedding, similarity, hybrid retrieval, metadata filtering, and ranking tasks.

The category is the task directory. meta.json keeps capability equal to the category so existing --capability filtering can run one category at a time.

Current Tasks

Access Control

The access-control category is the RLS core suite. It intentionally keeps one basic ownership baseline and removes narrower or mixed lifecycle scenarios from the core set.

Task	Difficulty	Goal
document_sharing_acl	hard	Document owners can share documents through an ACL without leaking private documents.
field_level_update_mask	hard	Role-scoped users can update allowed fields but cannot alter protected fields.
hierarchical_org_tree	hard	Organization access follows a hierarchy without leaking sibling or unrelated org data.
jsonb_profile_visibility	medium	Public profile projection hides private JSONB fields while preserving owner access.
org_role_write_matrix	hard	Organization roles enforce a write matrix for project changes.
owner_notes	simple	Users can create and read only their own notes.
public_read_owner_write	simple	Public rows are readable, private rows remain private, and only owners can write.
rls_recursive_membership_guard	hard	Membership-backed policies isolate memberships and documents without recursive RLS failures.
team_membership	medium	Team membership controls note visibility and revocation behavior.
temporal_embargo_access	medium	Time-based visibility controls protect embargoed content until release.

Integrity

Task	Difficulty	Goal
approval_workflow_status	hard	State transitions follow an approval workflow and reject invalid jumps.
booking_exclusion_constraint	medium	Booking windows cannot overlap for the same resource.
comment_counter_consistency	medium	Server-maintained counters stay correct as comments change.
invite_token_claim	hard	Invite tokens can be claimed once and create the expected membership.
ledger_balance_trigger	hard	Ledger-derived balances remain server-maintained and tamper resistant.
moderation_queue_claim	hard	Queue items can be leased or claimed without cross-team interference.
project_archive_freeze	hard	Archived projects freeze child writes while preserving owner reads.
protected_child_delete_guard	hard	Protected active or locked child rows block parent deletion while safe child states can be cleaned up.
soft_delete_unique_active_slug	medium	Soft-deleted rows release active slugs without breaking historical rows.
tenant_quota_reference_guard	hard	Tenant project quotas, member-scoped project visibility, and project identity guards are enforced together.
versioned_document_history	hard	Document version history and latest pointers are server-maintained.

Query

Query tasks measure query-facing database work only: reporting, aggregation, search, pagination, joins, ranking, and query performance. They do not require auth users, JWT claims, RLS policies, or access-control attacks.

Task	Difficulty	Goal
cursor_paginated_activity_feed	medium	Activity feeds paginate stably with actor joins, visibility filters, optional type filters, and timestamp ties.
dashboard_metric_rollups	hard	Dashboard metrics aggregate day+channel revenue, refunds, AOV, new customers, and repeat customers.
denormalized_order_list	hard	Join-heavy order list queries return search-filtered orders with item totals, refunds, latest payment, and latest shipment.
faceted_catalog_search	hard	Catalog search combines optional filters, stock filtering, ranking, price bounds, and pre-limit facet counts.
latest_status_timeline	hard	Ticket timelines expose latest status, response timestamps, reopen counts, and response-needed state.
query_performance_indexes	medium	Multiple high-volume support-ticket lookup workloads have practical reusable indexes.
time_bucket_analytics_report	hard	Analytics reports return workspace-scoped zero-filled buckets with revenue, churn, conversion, and retention metrics.
top_n_leaderboard	hard	Leaderboards aggregate season scores, apply eligibility filters, and return top-N rows per team with deterministic tie breaks.

Vector

Vector tasks use deterministic fixed vectors and do not call external embedding, AI, or LLM APIs.

Task	Difficulty	Goal
filtered_multilingual_kb_search	medium	Search published knowledge-base documents with product, language, and collection filters.
thresholded_ticket_deduplication	medium	Find similar support tickets above a threshold within a category.
related_product_recommendations	medium	Recommend related available products using item-to-item vector similarity.
rag_parent_document_collapse	hard	Search chunk embeddings while returning parent document context and one best chunk per document.
hybrid_support_rank_fusion	hard	Blend weighted keyword relevance and vector similarity for support-center search.
multi_vector_profile_search	hard	Search profiles by combining multiple embedding columns with caller-provided weights.
vector_index_performance	hard	Create an ANN vector index and expose filtered indexed nearest-neighbor search.
embedding_version_selection	hard	Search only active embeddings for the requested embedding model while ignoring stale versions.
diverse_result_rerank	hard	Rerank nearest neighbors with a per-topic diversity cap.

Running

Run the whole DB task set:

npm run benchmark:all -- --module db

Run one category:

npm run benchmark:all -- --module db --capability access-control
npm run benchmark:all -- --module db --capability integrity
npm run benchmark:all -- --module db --capability query
npm run benchmark:all -- --module db --capability vector

Run one task:

npm run benchmark:cli-skills -- --module db --task owner_notes

When --module db is supplied and no --suite is provided, the CLI defaults to --suite db. Other modules keep the pilot default.

Task Authoring

Each DB task must live directly under a category directory:

tasks/db/<category>/<task_id>/

Required files:

meta.json
prompt.md
verify.ts

Optional files:

prepare.ts
attack.ts
cleanup.ts
run.ts
state.ts

Use run.ts only when one file exports the task lifecycle functions. Do not add compatibility wrappers for older task paths or suites.