AI Engineering Journey

Hi, I'm Alperen. This repository documents my journey to become a production-oriented AI Engineer.

Goal

Build practical skills in:

• Python engineering
• Machine learning
• Deep learning
• RAG systems
• LLM applications
• AI agents
• MLOps
• Cloud deployment
• Evaluation and monitoring

Learning Format

Each day follows this structure:

1. Learn the concept
2. Build a small artifact
3. Commit the work
4. Document progress

Current Progress

• Pre-roadmap foundation: Python engineering, SQL, EDA, baseline ML, validation, and experiment logging
• Month 1/9 completed: mini RAG from scratch with chunking, TF-IDF, hybrid retrieval, no-answer detection, and answerability evaluation
• Month 2/9 in progress: dense embeddings, semantic retrieval, dense/lexical hybrid retrieval, threshold experiments, and cross-encoder reranking
• Week 2 product core: layered FastAPI service, durable PDF jobs, tenant/ACL-ready retrieval, evaluation artifacts, observability and Compose demo

Current planning documents:

• Revised 9-month AI Engineering roadmap
• Internship Week 1 program comparison and integration
• Internship Week 2 product engineering implementation plan

Week 2 Document Intelligence Service

The current local deployment is intentionally small but follows the Week 2 topology: Qdrant, API, a separate durable ingestion worker and a static demo UI. Ollama stays on the host so the 32 GB RAM machine does not run a second model container. API and worker use the same locally built image; SQLite keeps accepted jobs, idempotency identities and staged PDF bytes across restarts. Redis remains an optional later scale-out decision, not a hidden dependency.

docker compose up --build -d
docker compose ps
curl -i http://127.0.0.1:8010/v1/health/live
curl -i http://127.0.0.1:8010/v1/health/ready
open http://127.0.0.1:8501

The Compose Qdrant service is reachable from the host on 6335 by default, because the earlier local demo may already occupy 6333. Run the reproducible smoke with:

./toolbox/scripts/run_document_service_compose_smoke.sh

The API listens on container port 8000 and host port 8010 by default, because other local services occupy host ports 8000 and 8001. Override either host port explicitly with API_HOST_PORT=8000 or QDRANT_HOST_PORT=6333 only after confirming that port is free.

If readiness is 503, inspect the dependency checks; this is deliberately reported as not-ready instead of allowing the UI to claim that answer queries are available. On this workstation Ollama runs as the existing ai-journey-ollama container and is bound to the Compose network. The reproducible local check starts API, worker and UI, uploads the sample PDF, waits for the worker, and verifies Qdrant point persistence:

./toolbox/scripts/run_document_service_compose_smoke.sh

For a long-running local stack, docker compose up --build -d uses http://ai-journey-ollama:11434. On another machine where Ollama is exposed through the host gateway, override it with DIS_OLLAMA_URL=http://host.docker.internal:11434.

The UI exposes health, tenant/ACL context, document selection, upload idempotency, per-stage ingestion duration/decision, retry attempts, query mode, canonical evidence metadata, retrieval candidate scores, no-answer reason, latency breakdown and privacy-safe metrics. It renders returned text with textContent; model output is never treated as HTML.

Useful operational endpoints:

curl -sS http://127.0.0.1:8010/v1/metrics
curl -sS -H 'X-Tenant-ID: default' 'http://127.0.0.1:8010/v1/documents?limit=100'

Use POST /v1/queries for the current query contract. /v1/query remains as a compatibility alias. POST /v1/search returns evidence without calling Ollama. When the answerability gate rejects a query, the response contains no_answer.reason_code, model: null, latency.llm_ms: 0 and the trace records the decision. Query trace logs contain a question hash and stage metadata, not the raw question or evidence. Document lifecycle audit events contain only document/version/action/result identifiers and bounded metadata.

Detailed service setup, API examples, known limitations and the 28-page acceptance matrix are in:

• Document Intelligence service README
• API examples
• Architecture and query sequence
• 28-page acceptance matrix

Repository Philosophy

This is not a passive course repo. Every folder should contain proof of work: scripts, notebooks, tests, reports, demos, or deployment artifacts.

Local Git Hooks

This repository includes Git hooks under .githooks.

After cloning the repository, enable hooks with:

git config core.hooksPath .githooks

The current hook validates commit messages using the Conventional Commits format:

type(scope): short description

Example:

docs(repo): add contributing guide

Current Artifacts

• toolbox/: shell scripts and CLI notes
• labs/lin_alg/vec.py: pure Python vector operations and cosine similarity
• labs/lin_alg/text_similarity.py: word-count based text similarity demo
• labs/lin_alg/debug_examples.py: handled debugging examples

Weekly Summaries

• Week 1 — Foundations

Useful Commands

Run Week 1 checks:

./toolbox/scripts/run_week1_checks.sh

Type Checking

• pyproject.toml configures mypy in strict mode.
• examples/w2d2_type_check_demo.py demonstrates typed DailyLog usage.

Type Checking

• pyproject.toml configures mypy in strict mode.
• examples/w2d2_type_check_demo.py demonstrates typed DailyLog usage.

Testing

• tests/test_domain.py covers Artifact, Milestone, and DailyLog behavior.
• pytest is used for automated unit tests.

Matrix Multiplication

• labs/lin_alg/matrix.py implements pure Python matrix multiplication.
• labs/lin_alg/matrix_benchmark.py compares pure Python matrix multiplication with NumPy.

Matrix Multiplication

• labs/lin_alg/matrix.py implements pure Python matrix multiplication.
• labs/lin_alg/matrix_benchmark.py compares pure Python matrix multiplication with NumPy.

Pandas Basics

• data/raw/students.csv is a small toy dataset for EDA practice.
• labs/data/pandas_basics.py demonstrates loading CSV data, checking missing values, and simple groupby analysis.

Data Cleaning

• labs/data/clean_students.py fills missing student data and writes a cleaned CSV.
• data/processed/students_clean.csv is generated from the raw student dataset.

SQL Basics

• labs/data/sql_basics.py writes the cleaned student dataset to SQLite.
• SQL examples cover SELECT, WHERE, ORDER BY, LIMIT, and GROUP BY.

EDA Report

• labs/data/eda_report.py generates a Markdown EDA report from the cleaned student dataset.
• docs/w3_eda_report.md summarizes missing values, target distribution, group statistics, and modeling readiness.

Feature Dataset

• labs/data/build_features.py splits the cleaned student dataset into features and target.
• The generated files under data/processed/ are ready for a baseline classification model.

Baseline ML Classifier

• labs/ml/train_baseline_classifier.py trains a Logistic Regression baseline model.
• docs/w3_baseline_model_report.md summarizes accuracy, predictions, and model limitations.

Week 3 — Data and Baseline ML

• Week 3 Summary
• W3 EDA Report
• W3 Baseline Model Report

Run Week 3 checks:

./toolbox/scripts/run_week3_checks.sh