Skip to content

AGENTS.md

Latest commit

 

History

History
202 lines (147 loc) · 16.8 KB

AGENTS.md

File metadata and controls

202 lines (147 loc) · 16.8 KB
Raw

Repository Guidelines

This file is the canonical engine convention reference, shared across all AI coding agents (Claude Code, Codex, etc.). It is the single source of truth for tech stack, architecture, testing, quality gates, CI, security, and commands.

CLAUDE.md holds only Claude Code-specific context (working-memory header, VBW commands, and the .claude/ agent/skill catalogs) and references this file for conventions. When a convention changes, edit it here, not in CLAUDE.md.

Use rbenv for all ruby and bundler/gem commands, not the system ruby.

Tech Stack

Layer Technology
Ruby 4.0+
Rails 8.x
Testing Minitest (no fixtures -- uses factory helpers + WebMock/VCR)
Authorization Host app responsibility (mountable engine); fail-closed by default (#129)
Jobs Solid Queue
Frontend Hotwire (Turbo + Stimulus) + Tailwind CSS
Linting RuboCop (omakase) + Brakeman
Database PostgreSQL only

Project Structure & Module Organization

The engine follows the Rails mountable layout generated by rails plugin new source_monitor --mountable. Runtime code lives under app/ (controllers, jobs, views) and is namespaced as SourceMonitor. Long-lived services, adapters, and instrumentation helpers belong in lib/source_monitor/. Generator templates and install scripts reside in lib/generators/source_monitor/. Tests sit in test/, with fixtures under test/fixtures/feeds/, and the dummy host app in test/dummy/ for integration coverage. Keep shared UI assets under app/assets/ and engine configuration in config/initializers/.

Architecture Conventions

Models First

  • Business logic lives in models. Use concerns for horizontal sharing.
  • Service objects ONLY for operations spanning 3+ models or external integrations.
  • Query objects for complex queries that don't fit a single scope.
  • Presenters (SimpleDelegator) for view-specific formatting.

Everything-is-CRUD Routing

  • Prefer creating a new resource over adding custom actions.
  • POST /posts/:id/publications over POST /posts/:id/publish.
  • RESTful routes only; no member or collection blocks with custom verbs.

State as Records

  • Track business state transitions as separate records (who/when/why).
  • Boolean columns ONLY for technical flags (e.g., email_verified, open_access).

Jobs

  • Shallow jobs: call _later or _now methods on models/services.
  • Jobs contain only deserialization + delegation. No business logic.
  • Use Solid Queue recurring jobs for scheduled work.

Frontend

  • Turbo Frames for partial page updates.
  • Turbo Streams for real-time broadcasts.
  • Stimulus controllers: small, focused, one behavior each.
  • Tailwind CSS utility classes; extract components for repeated patterns.

Coding Style & Naming Conventions

Use two-space indentation and Ruby 4.0+ syntax. Keep engine classes under the SourceMonitor:: namespace; new modules should mirror their directory, e.g., lib/source_monitor/fetching/pipeline.rb. Favor service objects ending in Service, jobs ending in Job, and background channels ending in Channel. Rails defaults handle formatting, but run bin/rubocop (configured via .rubocop.yml) before opening a PR. For views, stick with ERB and Tailwind utility classes.

Sub-module extraction pattern: create module/submodule.rb with require_relative, lazy accessors, and forwarding methods for backward compatibility. The project uses Ruby autoload for lib/ modules (not Zeitwerk).

Clean Coding Principles

  • SRP: Classes and methods should have a single responsibility.
  • DRY: Avoid duplication; changes should only need one edit.
  • Depend on behaviour, not data: Wrap instance variables in methods (attr_reader); use Struct for data structures.
  • Minimise dependencies: Use dependency injection, encapsulate external messages, prefer hash arguments.
  • Depend on things that change less often than you do.

Configuration DSL

  • SourceMonitor.configure exposes structured namespaces:
    • config.authentication gates the engine. With no authenticate_with/authorize_with handler configured, the engine is fail-closed and returns 403. Set config.authentication.open_access = true (default false) only for local demos/sandboxes — never production.
    • config.http for Faraday timeouts, retry policy, proxy, and default headers. Per the Faraday retry docs, middleware options map 1:1 to the settings we surface (max retries, interval, backoff, statuses).
    • config.scrapers registers/overrides adapters by name; adapters must inherit from SourceMonitor::Scrapers::Base and are discovered before constant lookup.
    • config.retention supplies global defaults for items_retention_days, max_items, and the pruning strategy (:destroy or :soft_delete). Runtimes treat blank source fields as “inherit from config”.
    • config.models lets host apps adjust table name prefixes, mix in concerns, and register custom validations per engine model. Use it to bolt on associations or STI-specific rules without monkey patches.
    • config.realtime selects the Action Cable backend (:solid_cable by default). Solid Cable keeps Turbo streams in the primary database; set config.realtime.adapter = :redis and optionally config.realtime.redis_url when hosts prefer Redis.
  • Install generator and dummy initializer list all knobs with comments—update those when slicing future roadmap items.

Retention Defaults

  • Per-source retention settings live on SourceMonitor::Source (items_retention_days and max_items). Negative values are rejected; blank means unlimited.
  • SourceMonitor::Items::RetentionPruner runs after every fetch via SourceMonitor::Fetching::FetchRunner, pruning stale items and their associated content/logs while keeping counter caches in sync.
  • Age-based rules prune items when their published timestamp (or created_at fallback) is older than the configured window. Count-based rules keep the newest N items.
  • SourceMonitor::ItemCleanupJob batches retention pruning across sources and can soft delete records (rake source_monitor:cleanup:items honours SOFT_DELETE=true, SOURCE_IDS=1,2). SourceMonitor::LogCleanupJob prunes old fetch/scrape logs (rake source_monitor:cleanup:logs, override FETCH_LOG_DAYS / SCRAPE_LOG_DAYS).
  • Nightly recurring entries in config/recurring.yml enqueue both cleanup jobs by default; adjust schedules or disable via Solid Queue overrides as needed.

Background Job Defaults

  • SourceMonitor ensures Solid Queue is the default adapter when the host app is still using the async adapter, but respects any explicit ActiveJob configuration already in place. Override queues/concurrency via SourceMonitor.configure.
  • Queue names are namespaced (source_monitor_fetch/source_monitor_scrape by default) and automatically honor host queue_name_prefix. Use SourceMonitor.queue_name(:fetch) helpers inside jobs.
  • Dashboard queue metrics read directly from Solid Queue tables via SourceMonitor::Jobs::SolidQueueMetrics. Host apps must install the Solid Queue migrations (reuse the engine's 20251009140000_create_solid_queue_tables.rb or run rails solid_queue:install) for the card to surface ready/scheduled/failed counts; otherwise the UI falls back to an availability warning. Mission Control remains optional for deeper drill-downs.
  • The dummy host keeps Solid Queue tables in the primary database via 20251009140000_create_solid_queue_tables.rb. Real apps can either reuse that migration or run rails solid_queue:install to manage a dedicated queue database—Mission Control expects one of those setups before it can surface data.
  • Recurring schedules live in config/recurring.yml, scheduling SourceMonitor::ScheduleFetchesJob each minute plus the scraping scheduler every two minutes. Override the schedule path with bin/jobs --recurring_schedule_file=... (or SOLID_QUEUE_RECURRING_SCHEDULE_FILE) and disable recurring runners with SOLID_QUEUE_SKIP_RECURRING=true or bin/jobs --skip-recurring.
  • Hosts that need to wrap Solid Queue command execution can set config.recurring_command_job_class in the generated initializer to point at their custom job class.

Build, Test, and Development Commands

Run bin/setup to install gems, prepare the dummy database, and compile Tailwind. During feature work, bin/dev starts the dummy app with Solid Queue workers and Tailwind watcher.

bin/dev                     # Start dev server
bin/rails test              # Run the MiniTest suite (NOT a substitute for CI -- see below)
bin/rubocop                 # Check style
bin/rubocop -a              # Auto-fix style
bin/brakeman --no-pager     # Security scan
bin/rails db:migrate        # Run migrations

The dummy host app runs on port 3002 (cd test/dummy && bin/rails server -p 3002).

Testing

  • Framework: Minitest. NEVER use RSpec or FactoryBot. Name files with _test.rb and wrap suites in module SourceMonitor. Tests live in test/models, test/controllers, test/system, test/lib, test/integration.
  • Helpers: create_source! factory, with_inline_jobs, with_queue_adapter.
  • HTTP: WebMock disables external HTTP; VCR for recorded cassettes under test/vcr_cassettes/ (record new fixtures with descriptive names like source_fetch_success.yml).
  • Config isolation: test/test_helper.rb calls SourceMonitor.reset_configuration! in every test's setup (rebuilding a fresh Configuration), so any config a test relies on must be set in setup, not the dummy initializer.
  • Coverage: Target >90% for new services; cover every model validation, scope, public method, and controller action, plus regression tests for bug fixes.
  • Parallelism: Coverage runs need COVERAGE=1 PARALLEL_WORKERS=1 with threads (not forks) to avoid a PG segfault and SimpleCov data loss. Scope queries to a specific source/item to prevent cross-test contamination in parallel runs.
  • Automate what you can: Anything verifiable programmatically (config defaults, job enqueue behavior, controller responses) should be a test, not a manual checkpoint.

Quality Gates

  • bin/rubocop — zero offenses before commit (omakase: only ~45/775 cops enabled, all Metrics cops disabled — no file-size enforcement).
  • bin/brakeman --no-pager — zero warnings before merge.
  • bin/rails test — all tests pass.
  • yarn build — rebuild JS assets if any .js files changed (ESLint runs in CI).
  • No N+1 queries (use includes/preload).
  • No hardcoded credentials (use Rails credentials or ENV).

Pre-Push CI Checklist (run ALL before pushing to GitHub)

Before pushing any branch (especially release branches), run the full CI equivalent locally:

  1. bin/rubocop — catches Ruby lint issues.
  2. bin/test-coverage — this is what CI's test job actually runs. Do NOT rely on bin/rails test to predict CI: bin/rails test uses a different harness/seed and does NOT run the diff-coverage gate, so it can be green (e.g. 1741/0) while CI fails. bin/test-coverage runs the real seed, the second health_suite pass, and the host-app-template test that changes the process CWD mid-suite (see gemspec note below).
  3. bundle exec ruby bin/check-diff-coverage — run AFTER bin/test-coverage (it reads coverage/.resultset.json). Reproduces the CI diff-coverage gate locally (threshold 90% on changed app//lib/ lines vs origin/main). This is the only way to know the gate passes before pushing.
  4. bin/brakeman --no-pager — catches security issues.
  5. yarn build — rebuilds JS and catches ESLint issues (CI runs ESLint separately).

If legitimate coverage gaps remain after new code paths, refresh the baseline: bin/test-coverage then bin/update-coverage-baseline, and commit the regenerated config/coverage_baseline.json.

Why: CI failures cost ~5 min per round-trip. Hard-won lessons:

  • CI runs bin/test-coverage, not bin/rails test — replicate the gate locally with steps 2 + 3. (v0.14.0: a green local bin/rails test masked a CI bin/test-coverage failure.)
  • Diff coverage covers EVERY changed app//lib/ line, including defensive/edge branches with no natural caller. If a branch can't be reached through a normal engine route (e.g. the #130 turbo_stream flash-append, which only fires when a turbo_stream request carries a Rails flash), add a test-only probe controller in the dummy app (pattern: test/dummy/app/controllers/test_support_controller.rb + a route in test/dummy/config/routes.rb); subclass SourceMonitor::ApplicationController if the branch lives in the engine's filter chain.
  • Every rescue/fallback/error path in new source code needs test coverage.
  • Gemspec packaging must be CWD-independent. Do NOT use a bare Dir[...] glob in source_monitor.gemspec for file selection — it resolves against the process CWD, and during bin/test-coverage a sibling test chdir's into a generated host app, so the glob returns nothing and files silently drop from the package (v0.14.0 #131). Drive packaging from git ls-files inside the existing Dir.chdir(File.expand_path(__dir__)) block.
  • JS files need /* global */ declarations for browser APIs (MutationObserver, requestAnimationFrame, etc.); ESLint no-undef rejects them otherwise. Run yarn build after JS changes to sync sourcemaps.

Contribution Workflow

  • Treat the engine like any external contributor would: no direct commits to main.
  • Before writing code, branch off the latest origin/main (use a descriptive feature/ or bugfix/ prefix) and open a draft PR on github.com/dchuk/source_monitor.
  • Push early and often to that branch so history stays visible; keep commits scoped and rebases local to your branch only.
  • Move the PR out of draft once tests pass and the slice is ready for review; request at least one review and wait for all CI jobs (lint, security, test + diff coverage, release_verification) to succeed before merging.
  • Merge via the PR UI (squash or rebase as agreed) after approval; avoid rewriting shared history post-push.
  • Tag releases only after the release PR merges, then follow the checklist in CHANGELOG.md. There are TWO version files — lib/source_monitor/version.rb AND the top-level VERSION — and both must match; run bundle install after a bump so Gemfile.lock stays in sync (CI runs --frozen).

Commit & Pull Request Guidelines

Use Conventional Commit subjects in the format type(scope): description, e.g., fix(fetch): resolve advisory lock contention. Group unrelated work into separate commits. PRs should describe context, summarise the slice delivered, and list validation steps (bin/test-coverage, manual fetch run). Include screenshots or console output when altering UI or background jobs. Request at least one review and ensure CI completes before merge.

Security

Protected files (NEVER read or output)

  • .env, .env.*
  • config/master.key, config/credentials.yml.enc
  • .kamal/secrets
  • Any *.pem / *.key files

Forbidden operations

  • git push --force to main/master/production
  • git reset --hard without explicit user confirmation
  • rm -rf on root, home, or parent directories
  • chmod 777

General

Store secrets (API keys, webhook tokens) in config/credentials/ and never commit plain-text values. When adding HTTP endpoints or webhooks, default to Solid Queue middleware for retries and respect the allowlist in config/source_monitor.yml. Document new environment variables in config/application.yml.sample and call out any migrations that impact host apps.

Maintenance: Skills & Docs Alignment

Whenever engine code changes (models, configuration, pipeline, jobs, migrations, scrapers, events, health rules, or dashboard), the corresponding sm-* skill and its reference/ files MUST be updated in the same PR so skills always reflect current engine behavior. Releases must also audit README.md, docs/ (especially docs/upgrade.md), and the install initializer template against the source code.

Library Documentation

  • Use Context7 MCP constantly to look up fresh documentation for any task, especially tasks that rely on libraries or gems.

Project Dependencies & context7 links

Claude Code Skills

SourceMonitor ships 15 engine-specific Claude Code skills (sm-* prefix) covering the domain model, configuration DSL, pipeline stages, testing conventions, and more. Skills are distributed with the gem and installed into .claude/skills/ via rake tasks:

bin/rails source_monitor:skills:install        # Consumer skills (host app integration)
bin/rails source_monitor:skills:contributor     # Contributor skills (engine development)
bin/rails source_monitor:skills:all            # All skills
bin/rails source_monitor:skills:remove         # Remove all sm-* skills

See CLAUDE.md for the full skills catalog (consumer vs. contributor) and the .claude/ agent catalog.