feat: consolidate schema, seeds, migrations, and APIs across multiple ap - #23
Open
dvalenca wants to merge 24 commits into
Open
feat: consolidate schema, seeds, migrations, and APIs across multiple ap#23dvalenca wants to merge 24 commits into
dvalenca wants to merge 24 commits into
Conversation
…tegorias normalizadas - Troca Client + ensureConnection() por Pool (conexão automática) - Remove ~25 chamadas de ensureConnection() de todos os handlers - Corrige RMR: 14 cidades (remove Goiana, adiciona as 5 faltantes) - Adiciona display_name com capitalização e acentos corretos - validos agora é objeto com breakdown (atendimento_concluido, removido_particulares, removido_bombeiros, obito_local) - por_sexo: keys lowercase + nao_informado - por_faixa_etaria: 0_17_anos, 18_29_anos, ..., nao_informado (split 50+ em 50-64 e 65+) - por_categoria: 9 buckets fixos (sinistro/atropelamento x 4 + outro) - Adiciona campo total = validos.total + invalidos - Remove módulo unsafe-streets v1 (handlers, routes, index) - Limpa dados mock de v2 unsafe-streets - Remove extensao/km mock de streets endpoints - Corrige tipos (string→number) em summary - Atualiza specs para refletir mudanças
…fix null gender
- GET /v1/calls/profiles now returns por_sexo, por_faixa_etaria,
por_categoria no mesmo formato normalizado do /v1/cities
- Reordena registro de rotas em app.ts (summaryRoutes antes de
callsRoutes) para evitar conflito /calls/profiles vs /calls/{id}
- Corrige captura de gênero null tanto em profiles quanto em cities
…colisão, decoder DATASUS e nome RMR - Corrige /v1/summary: query real com ocorrência × residência filtrado RMR - Corrige /v1/cities-by-year: retorna 14 cidades RMR com dados reais, ordenado por total - Corrige /v1/matrix: substitui cidades×anos por matriz de colisão vítima×contraparte - Corrige /v1/filtros: filtros reais, decoder DATASUS idade, porFaixaEtaria no resumo - Corrige /v1/causas-secundarias: adiciona filtros cidade/ano - Corrige /v2/deaths/by-city: filtro RMR - Cria lib/rmr.ts: constantes RMR compartilhadas (6 e 7 dígitos) - Cria lib/idade.ts: decoder DATASUS SIM + classificador faixa etária
…es, capture border ways - Remove ciclomapa_infra table (migration 0008), all GeoJSON files, and related API routes - Add city_boundaries table (migration 0009) from pe_limites_municipais.geojson (IBGE 2025) - Fix city_ids in pdc_ways.json via point-in-polygon (fix_pdc_city_ids.py) - Capture border ways missed by area(a) using around.border_ways:20 (fetch_border_ways.py) - Update seed script: load city boundaries via ST_GeomFromGeoJSON - Update API queries: replace ciclomapa_infra with pdc_relation_ways
…des RMR faltantes Igarassu (2606804): 37.63 km PDC Moreno (2609402): 15.62 km PDC Ilha de Itamaracá (2607604): 6.77 km PDC Itapissuma (2607752): 6.27 km PDC Araçoiaba (2601052): 3.43 km PDC PDC ways nestas cidades foram atribuídos via fix_pdc_city_ids.py (point-in-polygon sobre limites municipais do IBGE). Overpass fetch das 5 cidades retornou 0 elementos (sem ciclovias em OSM). Essas cidades têm apenas infraestrutura PDC (state-funded).
…h geometry/extension data and fix frontend compatibility - citySummary: fix total_streets as number, add extensaoTotalKm from pcr_streets - cityConcentration: add street_extension_km per entry via pcr_streets JOIN - cityGeoJSON: replace empty coordinates with real PostGIS geometry (ST_AsGeoJSON/ST_Collect), add extension_km in properties - streetSummary: add street_extension_km - streetGeoJSON: add real geometry + extension_km - /v1/streets/top: add 'nome' field in response items - /v1/streets/history: accept ?via= as alias for ?nome= - /v1/streets/search: accept ?street= as alias for ?nome= - import pcrStreets schema in db/schema.ts - fuzzyMatchStreet helper for matching emergency addresses against pcr_streets.nlogra_conc
…nd add period
- Add pcr_street_id FK column to emergency_calls linking to pcr_streets
- Add indexes on emergency_calls(pcr_street_id) and pcr_streets(nlogra_conc)
- Backfill exact matches via pcr_address = nlogra_conc
- Replace slow correlated EXISTS+LIKE query (>60s timeout) with fast
IN subquery using the new index (~30ms)
- Add period { start_year, end_year } to city summary response
- Cast COUNT(*)::int to prevent bigint being returned as string
…ead of raw address - Group by pcr_address (normalized) instead of address (raw) in concentration, geojson, and most_dangerous_street queries - Filter out generic PCR addresses (NAO IDENTIFICADO, #N/A, OUTRO MUNICIPIO) to exclude non-specific street records from rankings - Move genericPcrFilter to module scope for reuse across handlers - This fixes top rankings showing 'BR', 'AV', empty strings as most dangerous streets
…matching and match by pcr_address only - Replace fuzzyMatchStreet() O(N*M) loop with FK-based pcr_address→nlogra_conc mapping in cityConcentration and cityGeoJSON handlers - Use pcr_street_id FK for extensaoTotalKm query in citySummary (via FK→name→all segments) - Use pcr_street_id FK for extension/geometry queries in streetSummary and streetGeoJSON - Change street-level endpoints to match by pcr_address ILIKE only instead of address ILIKE (avoids false positives and ensures canonical name matching) - Add start_year/end_year query params to citySummary, cityConcentration, cityGeoJSON
…et_id FK with normalized categories, age groups, and hourly breakdown
…gories, filters, points, OD, distribution and downloads - Refactor app.ts to standard createApp()/.route() pattern (10 other apps) - Consolidate health route into single file with DB check - Add GET /v1/dictionary — 35 field definitions with labels, types, sources - Add GET /v1/categories — distinct values from JSONB fields - Add GET /v1/filters — available years, areas, points, categories - Add POST /v1/cross — dynamic cross-tabulation engine with 19 fields, 24 metrics, whitelist, min_n suppression - Add GET /v1/points + /points.geojson — aggregated survey points with spatial query - Add POST /v1/od/matrix + /od/flows.geojson — origin-destination matrix and desire lines - Add POST /v1/distribution — histograms for age, distance_time, days_total - Add POST /v1/downloads/aggregate.csv — CSV export from cross results - Refactor summary endpoint to clean cards format with methodology metadata - Remove inline handlers and raw Hono app, align with monorepo standards
…no), add 2020 as default startYear, ensure int casts
…start_year=2020 on all endpoints - Create shared src/lib/categories.ts (CATEGORY_MAP + normalizeCategories) - Add accidents_by_category (9 buckets) to city GeoJSON response - Add default start_year=2020 to v1/summary, v1/cities, v1/streets, v1/filters, calls, summary endpoints - Use EXTRACT(YEAR FROM date) >= start_year filter on v1/summary handler - Use EXTRACT(YEAR FROM date) >= start_year filter on v1/cities handler - Fix missing AppRouteHandler/SummaryRoute imports in summary handler
… points, survey-locations, filters endpoints - GET /v1/points, /v1/points.geojson: ?min_interviews=30 default - GET /v1/cyclist-profiles/survey-locations: ?min_interviews=30 default - GET /v1/filters: ?min_interviews=30 default for count_points - Replaces hardcoded HAVING count(*) >= 3 with configurable threshold
- Add 6 dashboard endpoints (overview, top-violations, top-streets, temporal, agent-analysis, violation-codes) - All dashboard endpoints accept violation_codes filter for frontend-driven category classification - Include law_code (base legal CTB) in all violation responses - Remove mock data: by_hour (now real EXTRACT HOUR query), violations_per_km (now from pcr_streets.db2gse_sde) - Remove /violations/hotspots endpoint (coordinates were NULL) - Fix /violations/geojson to JOIN pcr_streets for street geometry via ST_AsGeoJSON - Change db/index.ts from pg.Client to pg.Pool for auto-connection - Fix agent categorization: 7,8=manual, 9=eletronico - Add pcrStreets export to db/schema.ts - Add location_street_matches schema, relations, zod schemas and types - Add street matching pipeline, normalizer, matcher and validation routes
…assificacao CTB - Cria tabela violation_categories (violation_code, law_code, description_keyword, category) - Seed script que importa CSV com 243 artigos do CTB classificados em 8 categorias: Segurança viária, Pedestres, Ciclistas, Transporte coletivo, Fluidez, Estacionamento, Administrativas, Outras - Normaliza law_code do banco para matching com CSV (acentos, cross-references, formatação) - Keyword-based sub-classificacao para codigos 5452, 5819, 5622, 6122, 6017 (ex: mesmo artigo cobre pedestres E ciclistas) - Adiciona parametro 'category' aos endpoints dashboard (top-violations, top-streets, temporal, agent-analysis) - Novo endpoint GET /dashboard/categories lista categorias com totais - 211 codigos classificados, 0 unmatched
…lation to top-streets
…es, gitignore - Extract shared query helpers (AGENT_INFO, buildConditions, resolveCategoryCodes) - Refactor dashboard handlers to use shared helpers - Add seed pipeline (seed-all, seed-violation-categories, seed-description-corrections) - Add ETL normalize script (Python) for processing raw CSV data - Add reference CSVs (CTB classification, description corrections, equipment) - Add migration for traffic_locations and traffic_equipment tables - Add .gitignore for large raw data files - Refactor match pipeline, street matcher/normalizer - Update route definitions and validation schemas
…log, and API ETL Pipeline: - Rewrite etl-normalize.py for 19 files (2007-2025), 2 formats (TSV/CSV), 7 columns - Fix 2015/2024 date parsing (time from separate column, not embedded) Location Dictionary (build-location-dict.py): - Extract equipment (semaphores + posts -> lat/lon), street names, directions, address numbers - Inverted-index token-based street matching (97.4% coverage) - Street type normalization (AV/R/C/EST/PCA/REF abbreviations) - Address number extraction (EM FRENTE, NUMERO, AONUMERO, DEFRONTE patterns) - Landmark fallback (aeroporto -> Praca Ministro Salgado Filho) - Street renames (NORTE -> Norte Miguel Arraes de Alencar, SUL -> Sul Cid Sampaio) Infraction Catalog (build-infraction-catalog.py + classify-catalog.py): - 500 canonical descriptions from 985 raw variants - 489 classified into 8 categories via CTB table + keyword rules - Encoding correction auto-detection (trigram similarity) + truncation fixes - descricao_mapping.csv: raw -> canonical mapping for known_variants[] Database: - Drop: violation_categories, description_corrections, traffic_equipment, face_quadra, pcr_streets_new, location_street_matches - Add: infraction_catalog (618 rows with known_variants TEXT[]) - street_codes: table -> VIEW over pcr_streets - traffic_locations: merge extracted fields + matching results - traffic_violations.street_code: populated from traffic_locations (98.8%) API: - top-violations: add total/percentage/violations_count, fix category+code filter - Category resolution via infraction_catalog (description = ANY(known_variants)) - Remove matching/validation endpoints (obsolete location_street_matches) - Fix buildConditions: apply both category AND violation_codes when present
- database: replace official_streets with street_codes (migration 0013) - database: add slim_cyclops, clean_duplicates, execution_columns migrations - database: add traffic_violations_catalog, description_corrections, traffic_violations_locations, traffic_equipment tables - database: import-google-sheets and seed-cyclist-counts-imported for Google Sheets integration - traffic-violations: rebuild ETL pipeline, infraction catalog, and API - traffic-violations: violation_code -> cttu_code, remove violation_type_id, remove coordinates, description-based category matching - traffic-violations: new app migrations for catalog rebuild and backfill - cyclist-profile: add min_interviews param, cross-tabulation API - emergency-calls: pcr_street_id FK, accidents_by_category, hourly breakdown - traffic-deaths: transportMode filter for causas-secundarias - cycling-infra: city_id via point-in-polygon for 5 RMR cities - docs: ARCHITECTURE_ANALYSIS and deploy analysis for traffic-violations - specs: updated OpenAPI v1/v0 specs for all changed services
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resumo
Este branch consolida ~227 commits de trabalho abrangendo múltiplos apps do monorepo Atlas. As principais áreas de mudança são: banco de dados compartilhado (schema + migrations + seeds), API de traffic-violations (reforma completa), cyclist-profile (cross-tabulation), emergency-calls (FK de ruas + endpoints v2), traffic-deaths (filtros), cycling-infra (city_id por point-in-polygon).
Banco de Dados Compartilhado (packages/database)
Schema: