Skip to content

INTERVIEW_GUIDE.md

Latest commit

 

History

History
415 lines (287 loc) · 16.8 KB

INTERVIEW_GUIDE.md

File metadata and controls

415 lines (287 loc) · 16.8 KB
Raw

Distributed Job Queue System — Interview Guide

Use this document to explain the project confidently in technical interviews. It is written in first person, as if you are walking an interviewer through your work.

1. Elevator Pitch (30 seconds)

"I built a distributed job queue platform — similar in spirit to Sidekiq or Celery, but implemented in Java 21 and Spring Boot 3. Clients submit background jobs through a REST API; workers pull jobs from Redis, execute them, and persist all metadata in PostgreSQL. The system supports priority queues, delayed scheduling, retries with exponential backoff, dead-letter queues, idempotency, JWT auth, rate limiting, and crash recovery. I also built a React dashboard for monitoring, plus Prometheus and Grafana for observability. Everything runs in Docker Compose with horizontally scalable workers."

2. "Tell Me About the Project" (2–3 minutes)

The Problem

Many applications need to offload slow or unreliable work — sending emails, generating reports, calling webhooks — without blocking the HTTP request. You need:

  • A durable queue so jobs aren't lost
  • Multiple workers for throughput
  • Retries when things fail
  • Visibility into what's running, stuck, or dead

My Solution

I designed a two-service architecture:

Service Responsibility
API Service (jobqueue-api, port 8080) Accepts job submissions, handles auth, writes metadata to Postgres, enqueues to Redis. Never executes jobs.
Worker Service (jobqueue-worker, port 8081) Polls Redis, executes job handlers, updates status, handles retries and DLQ

Both services share PostgreSQL (source of truth for job state) and Redis (fast queue layer). They do not talk to each other over HTTP — this decoupling lets you scale workers independently.

Tech Stack

  • Backend: Java 21, Spring Boot 3.3.5, Spring Security, Spring Data JPA, WebSocket (STOMP)
  • Data: PostgreSQL 16, Redis 7 (Streams + Sorted Sets)
  • Frontend: React 18, TypeScript, Vite, Tailwind CSS
  • Ops: Docker Compose, Prometheus, Grafana, Micrometer metrics
  • Testing: JUnit 5, Mockito, Testcontainers

3. Architecture Deep Dive

┌─────────────┐     REST + JWT      ┌──────────────┐
│ React UI    │ ──────────────────► │  API Service │
│ (port 3002) │ ◄── WebSocket ───── │  (port 8080) │
└─────────────┘                     └──────┬───────┘
                                           │
                          ┌────────────────┼────────────────┐
                          ▼                ▼                ▼
                    PostgreSQL          Redis          WebSocket
                    (metadata)       (queue layer)    (/ws/jobs)
                          ▲                ▲
                          │                │
                    ┌─────┴────────────────┴─────┐
                    │   Worker Service (×N)      │
                    │   worker-1, worker-2, ...  │
                    └────────────────────────────┘

Why This Split?

  • API stays fast — no heavy processing on request threads
  • Workers scale horizontally — add more containers, they compete via Redis consumer groups
  • Failure isolation — a crashing worker doesn't take down the API

Module Structure (Maven Multi-Module)

jobqueue-parent/
├── jobqueue-common/    # Shared entities, DTOs, enums, RetryPolicy
├── jobqueue-api/       # REST API, JWT, enqueue, stats, DLQ admin
├── jobqueue-worker/    # Consumer, handlers, schedulers, heartbeats
└── jobqueue-frontend/  # React dashboard

jobqueue-common avoids duplicating domain models. Both API and worker depend on it.

4. Job Lifecycle

PENDING → QUEUED → RUNNING → COMPLETED
                    ↓
                  FAILED → RETRYING → RUNNING → ...
                    ↓
              DEAD_LETTERED

Step-by-step when a user creates a job:

  1. Client sends POST /api/jobs with JWT + optional Idempotency-Key
  2. JobService checks idempotency (unique DB constraint) — returns existing job if duplicate
  3. Job row saved in Postgres with status QUEUED
  4. RedisQueueService enqueues to Redis (stream + priority sorted set)
  5. JobEventPublisher pushes WebSocket event to /topic/jobs
  6. Worker JobQueueConsumer reads from Redis via XREADGROUP
  7. JobExecutionService sets status RUNNING, records a JobAttempt, dispatches to the right JobHandler
  8. On success → COMPLETED with execution time
  9. On failure → retry with exponential backoff, or DEAD_LETTERED if max retries exceeded

5. Redis Design (Common Interview Topic)

Redis is the queue engine. I use three data structures:

Structure Key Purpose
Stream job_queue Main FIFO queue with consumer group job_workers for at-least-once delivery
Sorted Set priority_jobs Priority fallback — score = priority (HIGH=1, MEDIUM=5, LOW=10)
Sorted Set scheduled_jobs Delayed jobs — score = Unix timestamp when job should run
Stream dead_letter_queue Mirror of DLQ for streaming consumers
String + Lua rate_limit:{type} Fixed-window rate limiter per job type

Enqueue Flow

// RedisQueueService.enqueue()
1. Serialize job to QueueMessage JSON
2. ZADD priority_jobs {score: priority, member: jobId}
3. XADD job_queue * jobId type priority payload

Consumer Flow

// JobQueueConsumer
1. XREADGROUP GROUP job_workers BLOCK 5000 COUNT 1 STREAMS job_queue >
2. If stream empty → ZPOPMIN priority_jobs (highest priority first)
3. Process job → XACK

Why Stream + Sorted Set?

  • Streams give consumer groups, blocking reads, and at-least-once semantics
  • Sorted sets give O(log N) priority ordering when the stream is idle
  • Scheduled jobs use ZSET with timestamp scores — a scheduler moves due jobs into the stream every 5 seconds

Trade-off I'd mention: This is a dual-write pattern (Postgres + Redis). In production I'd add a transactional outbox so DB and queue writes are atomic.

6. Job Handlers — Strategy Pattern

Each job type has a handler implementing:

public interface JobHandler {
    JobType supportedType();
    void execute(String payload) throws Exception;
}

Implementations: SendEmailHandler, GenerateReportHandler, ResizeImageHandler, WebhookDeliveryHandler, DataCleanupHandler.

JobHandlerFactory injects all handlers via Spring, builds an EnumMap<JobType, JobHandler>, and resolves at runtime:

JobHandler handler = handlerFactory.getHandler(job.getType());
handler.execute(job.getPayload());

Why Strategy? Adding a new job type = one new @Component class. Open/closed principle. No switch statements scattered across the codebase.

7. Retry, DLQ & Crash Recovery

Exponential Backoff (RetryPolicy)

delay = 5000ms × 2^min(attemptNumber, 10)
→ 5s, 10s, 20s, 40s, 80s ...

Default maxRetries = 3. Failed jobs go to scheduled_jobs ZSET with score = now + delay.

Dead Letter Queue

When retries are exhausted:

  • Job status → DEAD_LETTERED
  • Row inserted into dead_letter_jobs table
  • Entry added to Redis dead_letter_queue stream
  • Admins can replay via POST /api/dlq/{id}/replay — resets job to QUEUED and re-enqueues

Crash Recovery (CrashRecoveryScheduler)

Runs on startup and every 60 seconds:

  • Finds jobs stuck in RUNNING for > 10 minutes (worker died mid-execution)
  • Sets status RETRYING, increments retry count, re-enqueues to Redis

Worker Heartbeat (WorkerHeartbeatScheduler)

Every 5 seconds, each worker updates workers.last_heartbeat. Workers with no heartbeat for 30 seconds are marked inactive — visible on the admin Workers page.

8. Authentication & Authorization

  • JWT (JJWT 0.12.6) with 24-hour expiry
  • Claims: sub (userId), email, role
  • BCrypt password hashing
  • Roles: USER, ADMIN → Spring authorities ROLE_USER, ROLE_ADMIN
Endpoint Access
/api/auth/register, /api/auth/login Public
/api/jobs/** Authenticated (users see own jobs; admin sees all)
/api/stats, /api/workers/**, /api/dlq/** Admin only
/actuator/**, Swagger, /ws/** Public (dev setup)

JwtAuthenticationFilter extracts Bearer token on every request and populates SecurityContext.

Bootstrap admin: admin@jobqueue.com / admin123 created by DataInitializer on first startup.

9. Idempotency & Rate Limiting

Idempotency

Clients send Idempotency-Key header. DB has a unique constraint on jobs.idempotency_key. Duplicate submissions return the existing job — critical for webhook retries and network failures.

Rate Limiting

Lua script in Redis implements a fixed-window counter per job type:

rate-limits:
  send_email: 100      # per 60 seconds
  generate_report: 50
  webhook_delivery: 200

If limit exceeded, job is re-queued to priority_jobs without counting as a failure.

10. Database Schema

Table Purpose
jobs Core job record — type, payload, status, priority, retries, timestamps, idempotency key
job_attempts Audit trail — each execution attempt with worker ID, duration, error
dead_letter_jobs Permanently failed jobs awaiting admin replay
users Email, BCrypt hash, role
workers Worker name, hostname, status, last heartbeat, active job count

Indexes on user_id, status, and unique idempotency_key for fast lookups.

Schema managed via Hibernate ddl-auto: update (fine for demo; production would use Flyway/Liquibase).

11. Frontend Dashboard

Stack: React 18 + TypeScript + Vite + Tailwind + Recharts

Page Route Features
Login /login JWT stored in localStorage
Dashboard / Stats cards, pie chart by status, recent jobs
Jobs /jobs Paginated list, create job modal
Job Detail /jobs/:id Full history, cancel/retry actions
Workers /workers Admin — live worker status
DLQ /dlq Admin — failed jobs, replay button

Real-time: STOMP over SockJS subscribes to /topic/jobs. API publishes on create/cancel/retry. Worker execution updates use polling (5–10s) since workers don't push WebSocket events.

12. Observability

  • Micrometer metrics exposed at /actuator/prometheus
    • jobs_created_total, jobs_retry_total, queue size gauges, worker counters
  • Prometheus scrapes API + workers (port 9090)
  • Grafana dashboards provisioned automatically (port 3001, admin/admin)
  • Structured JSON logging via logstash-logback-encoder with MDC fields: jobId, workerId, status

13. Docker Compose

One command: docker-compose up -d

Container Port Role
postgres 5433 Job metadata
redis 6380 Queue layer
api 8080 REST API
worker-1, worker-2 internal Job executors
frontend 3002 React UI (nginx)
prometheus 9090 Metrics
grafana 3001 Dashboards

Multi-stage Dockerfiles: Maven 21 Alpine build → JRE 21 Alpine runtime.

14. Testing Strategy

Test Type What It Validates
RetryPolicyTest Unit Backoff math, shouldRetry logic
JobPriorityTest Unit Priority score ordering
JobServiceTest Unit (Mockito) Idempotency, enqueue behavior
JobQueueIntegrationTest Integration (Testcontainers) Full Spring context with real Postgres + Redis
JobHandlerFactoryTest Unit All 5 handlers registered
RateLimiterServiceTest Unit Lua script key/limit behavior

Run: mvn test (unit) / mvn verify -pl jobqueue-api (integration)

15. Design Patterns Used

Pattern Where
Strategy JobHandler + implementations
Factory JobHandlerFactory
Repository Spring Data JPA
Publisher/Subscriber WebSocket job events
Scheduled Tasks Retry mover, crash recovery, heartbeats
Filter Chain JWT authentication filter

16. Trade-offs & "What Would You Improve?"

Interviewers love this section. Be honest:

Current Choice Limitation Production Improvement
Dual-write (Postgres + Redis) Not atomic — job could exist in DB but not queue Transactional outbox pattern
Hibernate ddl-auto No versioned migrations Flyway/Liquibase
JWT in localStorage XSS risk HttpOnly cookies + refresh tokens
Crash recovery by time threshold Slow jobs > 10 min may be re-processed Heartbeat per job lease, not just worker
Simulated handlers No real integrations Plug in SendGrid, S3, etc.
Worker doesn't push WebSocket UI polls for execution updates Worker publishes to Redis pub/sub → API relays
Duplicate repository code in API/worker Maintenance overhead Keep repos in common module or use shared library

17. Sample Interview Q&A

Q: Why Redis Streams instead of RabbitMQ or Kafka?

"For this scale and use case, Redis Streams give me consumer groups, blocking reads, and persistence without operating a separate message broker. Redis was already needed for rate limiting and scheduled jobs (sorted sets), so using it for the main queue keeps the infrastructure simple. For very high throughput or event sourcing, I'd evaluate Kafka."

Q: How do you guarantee a job runs exactly once?

"At-least-once delivery, not exactly-once. Redis Streams with consumer groups can redeliver if a worker crashes before ACK. I mitigate duplicates with idempotency keys on submission and crash recovery that re-queues stuck RUNNING jobs. Handlers should be idempotent by design — e.g., 'send email' checks if already sent."

Q: How would you scale this to 10,000 jobs/second?

"1) Add more worker instances — they're stateless and compete via consumer groups. 2) Shard Redis Streams by job type or tenant. 3) Connection pool tuning for Postgres. 4) Batch enqueue API. 5) Separate priority streams instead of ZSET fallback. 6) Consider partitioning Postgres by date or tenant."

Q: Walk me through what happens when a worker crashes mid-job.

"The job stays RUNNING in Postgres with a started_at timestamp. CrashRecoveryScheduler runs every 60 seconds, finds RUNNING jobs older than 10 minutes, marks them RETRYING, increments retry count, and re-enqueues to Redis. Another worker picks it up. If retries are exhausted, it goes to the dead letter queue for manual inspection."

Q: Why separate API and worker modules?

"Separation of concerns and independent scaling. The API is I/O-bound (HTTP, DB writes); workers are CPU/I/O-bound (job execution). In Kubernetes you'd deploy them as different Deployments with different resource limits and replica counts."

18. Quick Demo Script (Live Interview)

If asked to demo:

# 1. Start everything
docker-compose up -d

# 2. Open UI
# http://localhost:3002 — login as admin@jobqueue.com / admin123

# 3. Create a job via curl
curl -X POST http://localhost:8080/api/jobs \
  -H "Authorization: Bearer <TOKEN>" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: demo-001" \
  -d '{"type":"SEND_EMAIL","priority":"HIGH","payload":{"to":"user@test.com","subject":"Hello"}}'

# 4. Show job transitioning PENDING → QUEUED → RUNNING → COMPLETED in UI

# 5. Show Grafana dashboards at http://localhost:3001

# 6. Show Swagger at http://localhost:8080/swagger-ui.html

19. Resume Bullet Mapping

Each README resume bullet maps to concrete code:

Resume Bullet Evidence in Codebase
Distributed job processing platform Multi-module Maven, API + Worker split
Priority scheduling, retries, DLQ, delayed jobs Redis ZSET + Streams, RetryPolicy, DeadLetterQueueService, ScheduledJobMover
Worker heartbeat, crash recovery, idempotency, JWT, rate limiting WorkerHeartbeatScheduler, CrashRecoveryScheduler, idempotency key constraint, SecurityConfig, RateLimiterService
Prometheus, Grafana, WebSocket, JSON logging docker-compose.yml, JobEventPublisher, logstash encoder
Scalable multi-worker architecture worker-1, worker-2 in compose, Redis consumer groups

Good luck in your interview. Know the job lifecycle, Redis data structures, and one trade-off you would improve — that's what separates a strong answer from a memorized one.