feat(admin/contagens): persistence layer — Drizzle + Zod, SQLite for dev - #163
Open
iacapuca wants to merge 6 commits into
Open
feat(admin/contagens): persistence layer — Drizzle + Zod, SQLite for dev#163iacapuca wants to merge 6 commits into
iacapuca wants to merge 6 commits into
Conversation
Stands up the data layer for contagens. Local dev runs against SQLite via Cloudflare's local D1 binding; production will swap to Postgres + Hyperdrive near the end of this PR (sqliteTable -> pgTable, drizzle-orm/d1 -> drizzle-orm/postgres-js, wrangler d1 binding -> hyperdrive). All Zod schemas, the form, and the server function are dialect-independent. Schema (`contagem`) - Real columns: id, local_name, started_at (ISO 8601 wall-clock), timezone (default America/Recife), bucket_minutes, bucket_count, latitude, longitude, topology, notes, schema (variant tag), data (jsonb-as-text), and timestamps. - Generated columns derived from `data.totals.*` for cheap list-view filters: total_cyclists, peak_bucket_count. - UNIQUE (local_name, date(started_at)) — one count per place per day, per the user spec; cross-year repeats are fine. JSON shape (Zod, app/admin/contagens/schema/contagem-data.ts) - Discriminated union on `schema`. v1: `ameciclo.v1` with approaches (snapshot of labels), per-bucket movements & characteristics, per-session outros, bucket-scoped notes, and totals. - Canonical characteristic taxonomy with parent_key for rollups (caronas/cargueiras/servicos/contramaos derive from leaves). Form (TanStack Form + Zod) - Migrated from useState to @tanstack/react-form so the rest of the TanStack stack is consistent. - Removed the topology-sync useEffect: approaches/movements are reset inline in the topology-change handler (event-driven, no post-render side effect). - Field-level validation via Zod resolver; submit calls the createContagem server fn. Server fn - app/admin/contagens/server/createContagem.ts — Zod-validated input, computes totals, inserts via Drizzle. End-to-end tested locally: form submission produces a row with derived totals correctly populated from the JSON path. Local-dev wiring - wrangler.jsonc: D1 binding (DB) for local-only — no database_id, no remote attachment. The on-disk file lives under .wrangler/state/v3/d1/. - pnpm db:generate → drizzle-kit migration generation - pnpm db:migrate → wrangler d1 migrations apply --local - pnpm db:studio → drizzle-kit studio - .gitignore: .wrangler/ and *.xlsx (reference spreadsheets stay local) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Following the TanStack Form validation guide:
- Submit button now disables on !canSubmit || isSubmitting (was just
isSubmitting). Users can't fire a request through an invalid form.
- Per-field Zod validators on the cheap-to-check required fields so
errors surface on blur instead of only at submit time:
- existingLocationId (onChange — picker has no blur to wait for)
- locationName, date, startTime, endTime (onBlur)
- Cross-field rules (end > start, mode→id, total > 0) stay in the
form-level superRefine on NovaFormSchema. Per the docs:
field-level validators take precedence on the same path, so the
field-level checks above intentionally only handle presence/null;
the form-level schema owns the cross-field bits.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The form was carrying an invented list of characteristic keys
(ride/cargo/wrong_way/...) that didn't match the canonical taxonomy
in CHARACTERISTICS — and didn't match what the actual xlsx files
carry. This swaps the form's characteristic fields to be driven by
that taxonomy directly.
- nova-form.ts: CHARACTERISTIC_KEYS now derived from CHARACTERISTICS
(women, juveniles, helmet, sidewalk, mascara, carona_*, cargueira_*,
servico*, contramao*, modal/program leaves). Added CHARACTERISTIC_GROUPS
for display ordering and characteristicsInGroup() helper.
- Added an `outros` field (array of {label, count}) to the form schema
with cross-validation: a row needs both label and count, or neither.
- NovaContagemForm: characteristic fields render by iterating
CHARACTERISTIC_GROUPS. Each group shows a derived rollup badge
(Caronas / Cargueiras / Serviços / Contramãos) computed from its
child fields — read-only, mirrors the xlsx "padrões" totals.
- New Outros card with a TanStack Form array field — Adicionar to push
a new row, Trash button to remove. Empty state copy when there are
no rows.
- Submit now passes the cleaned outros to data.outros (filtering rows
with neither label nor count > 0).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Selecting an Ameciclo "Dados da Contagem" .xlsx pre-fills the form so
the user can review and save instead of retyping. SheetJS is
dynamic-imported so it only loads when the user actually clicks
"Importar de planilha" — keeps the route's first-load weight unchanged.
Parser (app/admin/contagens/parser/xlsx-import.ts):
- Reads the Resumo sheet for cruzamento name and date.
- Walks the Dados sheet: locates the movement table by ORIGEM/DESTINO
headers, derives approach order from the ORIGEM column (with any
destino-only labels appended), and infers topology from the count
(2=point, 3=t_junction, 4=crossroad).
- Sums hourly columns into single per-movement totals (form is
single-bucket today; will switch to arrays when hourly inputs land).
- Parses both characteristic tables (qualitativo padrões + observações)
with these rules:
- Plural rollup labels (Caronas, Cargueiras, Serviços, Contramãos)
are skipped — derived from leaves.
- "Serviço" / "Contramão" (singular) are leaves only when their
qualified sibling (Serviço APP / Contramão para conversão) is in
the same section — handles both spreadsheet variants we have on
file (2024 puts rollups in padrões, 2026 puts them in observações).
- "Outros - <descrição>" rows become CustomObservations carrying
the free-form description.
- Surfaces non-fatal issues as toast warnings (coordinates detected
but form lacks lat/lng inputs; unmapped characteristic labels).
Verified against both xlsx files I have on disk (2024.06.06,
2026.03.25): name, date, approaches, totals, characteristic counts,
and the Caronas/Cargueiras/Serviços/Contramãos rollups all match the
Resumo sheet's authoritative numbers.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…eristics The schema (contagem-data.ts) has always modeled movements and characteristics as per-bucket arrays — what was collapsed was the form layer. This swaps both to per-bucket arrays end-to-end, with a "Por hora" view toggle on the cards that own them. Form schema (nova-form.ts) - movements: Record<string, string[]> — one entry per bucket - characteristics: Record<string, string[]> — same - outros[].counts: string[] — same - bucketMinutes: enum 15/30/60/120, default 60 - bucketCount derives from startTime/endTime/bucketMinutes via deriveBucketCount(); helpers (sumStringArray, resizeBucketArray, emptyBucketArray, readCell, timeToMinutes) live alongside the schema. Form (NovaContagemForm.tsx) - New Sessão field: "Granularidade" select for bucketMinutes. - Resultados card and Características card each grow a "Totais / Por hora" toggle (UI state, useState). - Totals view: same UI as before — the diagram, MovementMatrix and the characteristic input grid all read sums of the bucket arrays. Editing in totals view writes [value, "", "", ...] of length bucketCount, putting the entered value in bucket 0. - Hourly view: new MovementHourlyTable and CharacteristicsHourlyTable components render a wide grid — one row per movement / characteristic, one column per bucket, with HH:MM headers. Each cell maps to a single bucket index. - Outros: edit count in totals mode (writes [value, "", ...]); the row also persists per-bucket counts when imported. - Topology change still resets movements; bucketCount changes don't — arrays grow/shrink lazily on read. - Submit sends the actual arrays (no more wrap-as-length-1) sized to bucketCount via resizeBucketArray. - Import handler explicitly setFieldValue's each key (form.reset doesn't reliably rebroadcast every Subscribe in this setup) and auto-flips both view toggles to "Por hora" when imported data is per-bucket. Parser (xlsx-import.ts) - Detects bucket geometry from the Dados header row's hourly columns: bucket count, bucket minutes (from the diff between consecutive headers — 15/30/60/120 with a fallback warning), startTime/endTime. - Each movement / characteristic / outros row produces a bucket array the same width as the detected geometry, no longer summing. End-to-end verified on 2024.06.06 file: - Form switches to "Novo ponto" + Hourly views automatically on import. - DB row: bucket_minutes=60, bucket_count=14, total_cyclists=1708, peak_bucket_count=218 (matches xlsx Máximo em uma hora). - data.movements."0-1" = [1,1,1,1,2,4,1,2,1,2,0,3,5,3] (Centro→Boa Viagem hourly column from the xlsx). - data.characteristics.women = [3,31,13,5,6,20,9,38,8,15,20,23,9,11]. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The first column in the hourly tables was sticky-left, but its background was bg-muted/40 (40% opacity). When you scrolled horizontally, the hour column headers slid behind it but bled through the translucent fill, making the labels appear "on top of" the De → Para / Característica sticky cell. - Replace bg-muted/40 with the solid bg-muted on every header cell. - Bump the sticky-left header cell to z-20 so it sits above the rest of the thead row during horizontal scroll. - Same fix applied to MovementHourlyTable and CharacteristicsHourlyTable. - Body sticky cells already used bg-background (solid), so they were already fine. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stands up the data layer for contagens. Local dev runs against SQLite via Cloudflare's local D1 binding (a temporary crutch — never wired to a remote D1). Near the end of this PR I'll swap to Postgres + Hyperdrive; the JSON Zod schemas, the form, and the server function are dialect-independent and stay put.
What's new
Schema (
contagemtable)id,local_name,started_at(ISO 8601 wall-clock),timezone,bucket_minutes,bucket_count,latitude,longitude,topology,notes,schema(variant tag),data(text/jsonb),created_at,updated_at.data.totals.*:total_cyclists,peak_bucket_count. Cheap reads, can never disagree with the JSON.UNIQUE (local_name, date(started_at))enforces one count per place per day.JSON shape (Zod)
app/admin/contagens/schema/contagem-data.ts— discriminated union onschema. v1:ameciclo.v1carriesapproaches(snapshot), per-bucketmovements&characteristics, per-sessionoutros,bucketNotes, andtotals.parent_keyfor rollups (caronas/cargueiras/servicos/contramaosderive from leaves).Form (TanStack Form + Zod)
useStateto@tanstack/react-formso the rest of the TanStack stack stays consistent.useEffect— approaches/movements reset inline in the topology-change handler.createContagemserver function that validates with Zod, computes totals, and inserts via Drizzle.Local-dev wiring
wrangler.jsoncD1 binding (DB) for local only. Nodatabase_idset in any remote sense; on-disk SQLite under.wrangler/state/v3/d1/.pnpm db:generate— generate migrationpnpm db:migrate— apply to local D1pnpm db:studio— drizzle-kit studio.gitignore:.wrangler/and*.xlsx(reference spreadsheets stay local)End-to-end verified
Still TODO before merge
sqliteTable→pgTable,drizzle-orm/d1→drizzle-orm/postgres-js,text json→jsonb,json_extract→data->. Schema definitions stay otherwise unchanged.wrangler.jsonc; remove the D1 block.Test plan
pnpm buildandpnpm install --frozen-lockfilegreenpnpm db:migrateapplies the migration🤖 Generated with Claude Code