SkyStay Lab — Travel Data Platform Learning Environment

A hands-on lab for learning databases, streaming, OLAP, ML, AI/RAG, observability, and .NET CQRS — all through one fictional online travel agency (SkyStay) that sells flights and hotels.

Status: Educational sandbox. Not production code. Mock payments, fake inventory, local Docker only.

What you will practice

• Flight search, revalidate, book, pay, complete, refund
• Hotel availability, checkrate, book, pay, complete, refund
• Anonymous, authenticated, and bot traffic patterns
• Kafka event streaming, Spark analytics, StarRocks / ClickHouse OLAP
• RAG support policies, LangGraph agents, Temporal sagas
• OpenTelemetry → Jaeger + Prometheus → Grafana
• Capstone SkyStay Assistant chatbot using the full stack

Repository layout

Track	Folders	Purpose
A — Databases	00–05	Postgres, MongoDB, Redis, Neo4j, Elasticsearch, MinIO
B — App & pipeline	10–17	Kafka, Spark, OLAP, MLflow, Airflow, Travel API, simulator
C — AI	20–25	Ollama, Qdrant, LiteLLM, RAG, LangGraph, Temporal
D — Observability	27	OTel Collector, Jaeger, Prometheus, Grafana
E — Assistant	28	Unified chatbot capstone
Curriculum	_shared/curriculum/	152 steps with lessons, tasks, and Q&A
Domain	_shared/domain/	Entities, events, fixtures

See _shared/docs/tracks.md and _shared/curriculum/SYLLABUS.md.

Quick start

Prerequisites

• Windows 10/11 with WSL2 recommended
• Docker Desktop (16 GB RAM minimum; 24 GB for heavy steps)
• .NET 8 SDK
• Python 3.11+
• Git

Clone and configure

git clone https://github.com/abuzargashtasebi/SkyStayLab.git
cd SkyStayLab
copy .env.example .env

Edit .env for local passwords (never commit .env).

Start your first services (step 001)

All compose files use project name skystay (Docker Desktop groups containers under the skystay stack). Containers are named skystay-postgres, skystay-redis, etc.

cd 00-postgres-pgvector
docker compose up -d

Optional: set COMPOSE_PROJECT_NAME=skystay in root .env (see .env.example).

Follow the curriculum from step 001: _shared/curriculum/phase-00-orientation/step-001-lab-setup-windows/README.md.

Study workflow

1. Open the step folder under _shared/curriculum/.
2. Read README.md → complete every item in TASKS.md.
3. Compare results with EXPECTED.md.
4. Answer qa/questions.md before opening qa/answers.md.
5. Score ≥ 70% → copy progress.template.json to progress.json and mark complete.
6. Proceed to the next step in SYLLABUS.md.

Learning path (152 steps, 30 phases)

Phase	Steps	Topic
0–1	001–010	Setup, SkyStay domain, seed data
2–6	011–049	Postgres, Travel API, auth, bots
7–11	050–071	Redis, Mongo, Kafka, traffic simulator
12–21	072–108	Search, Spark, OLAP, MLflow, Airflow
22–26	109–128	Temporal, Ollama, RAG, LangGraph
27–28	129–138	Capstones, heavy data
29	139–147	Observability (OTel, Jaeger, Grafana)
30	148–152	SkyStay Assistant (final chatbot)

Recommended duration: 10–14 weeks self-paced.

Heavy data tiers

Generated locally — not in git. See _shared/docs/heavy-data-tiers.md.

Tier	Scale	Use when
T0	Seed fixtures	Steps 001–010
T1	~100K events	First pass per tool
T2	~5M events	Spark / OLAP
T3	~50M events	Kafka lag, indexing
T4	10GB+ Parquet	Optional step 138

cd _shared/data-generator
python generate_catalog.py --tier T2

SkyStay glossary

Term	Meaning
Revalidate	Re-check flight price/availability before ticket issue
Checkrate	Re-check hotel room rate before confirmation
Complete	Issue ticket (flight) or confirmation (hotel) after payment
Actor: anonymous	X-Session-Id only — limited refund/history
Actor: bot	High-volume search — rate-limited, excluded from conversion metrics

Full event list: _shared/docs/event-catalog.md.

Service ports (local)

Service	Port
Travel API	5080
SkyStay Assistant	5090
PostgreSQL	5434
Kafka	9092
Grafana	3000
Jaeger UI	16686
Prometheus	9090
MLflow	5000

Full list: _shared/docs/ports.md.

Which store for what?

Data	Store
Bookings, payments, refunds	PostgreSQL
Search sessions / clickstream	MongoDB
Sessions, price-hold cache	Redis
Routes, hotels-near-airport	Neo4j
Catalog text search	Elasticsearch
Files, Parquet archives	MinIO
Policy RAG vectors	Qdrant (+ optional pgvector / ES)
Analytics	StarRocks / ClickHouse

Capstone demos

Walkthrough	Folder	Curriculum steps
Booking journeys	26-booking-journey-walkthrough	129–133
Analytics pipeline	18-analytics-pipeline-walkthrough	134
Support RAG	19-support-agent-walkthrough	135
Full stack + traces	28-skystay-assistant	152

Observability

After step 139:

cd 27-observability
docker compose up -d

• Traces: http://localhost:16686 (Jaeger)
• Dashboards: http://localhost:3000 (Grafana)

Tech stack

Area	Technologies
API	ASP.NET Core 8, MediatR, FluentValidation, Npgsql
Stream	Apache Kafka
Batch	Apache Spark (PySpark)
OLAP	StarRocks, ClickHouse
ML	MLflow
Orchestration	Apache Airflow, Temporal
AI	Ollama, Qdrant, LangGraph, LiteLLM
Observability	OpenTelemetry, Jaeger, Prometheus, Grafana

GitHub hygiene

• Do not commit .env, progress.json, or data/generated/
• Do not commit large Parquet/event dumps
• Safe to commit: curriculum, fixtures (T0), docker configs, sample code

License

MIT — see LICENSE.

Disclaimer

Synthetic data and mock payment flows only. Do not point at real GDS/PSP systems without proper contracts and compliance review.