[DATA] Extract data preprocessing code and add DVC pipeline

[rojberr]

User Story:

As a data scientist,
I want to extract our monolithic code into discrete pipeline stages managed by DVC,
So that I can run steps independently, track dependencies clearly, and maintain a clean, reproducible workflow.

---

Acceptance Criteria:

1. Data download logic extracted to src/data_download.py
2. Training logic extracted to src/train.py
3. DVC pipeline with two stages in dvc.yaml
4. Pipeline supports independent execution of each stage
5. Clear input/output contracts between stages
6. Parameter management via config.yaml

---

Definition of Done:

• All acceptance criteria met
• Code reviewed and approved
• Both stages run successfully independently and sequentially f.e. using dvc repro <stage>
• Documentation updated in docs/dvc_workflow.md
• Git-tracked pipeline files

---

Pipeline Architecture:

graph LR
    A[data_download.py] -->|outputs| B[data/raw]
    B --> C[train.py]
    C -->|outputs| D[models/trained_model.pth]

Loading

[rojberr]

Example implementation:
Stage 1: Data Download (dvc.yaml):

yaml
stages:
download:
cmd: python src/data_download.py
deps:
- src/data_download.py
outs:
- data/raw
params:
- download:
source_url
max_retries
Stage 2: Training (dvc.yaml):

yaml
train:
cmd: python src/train.py
deps:
- src/train.py
- data/raw # Explicit dependency
outs:
- models/
params:
- training:
epochs
batch_size
learning_rate
File Structure:
src/
├── data_download.py # New: handles data acquisition
├── train.py # New: handles model training
params.yaml # Central configuration
dvc.yaml # Pipeline definition
data/
├── raw/ # Stage1 output
models/ # Stage2 output
Implementation Plan:

1. Data Download Script (src/data_download.py):

python
import yaml
from utils.downloader import fetch_data

if name == "main":
# Load parameters
with open("params.yaml") as f:
params = yaml.safe_load(f)["download"]

# Download data
fetch_data(
    url=params["source_url"],
    output_dir="data/raw",
    max_retries=params["max_retries"]
)

2. Training Script (src/train.py):

python
import yaml
from utils.preprocess import load_and_transform # Preprocessing inside train
from models.trainer import train_model

if name == "main":
# Load parameters
with open("params.yaml") as f:
params = yaml.safe_load(f)["training"]

# Load and preprocess data
dataset = load_and_transform("data/raw")

# Train model
train_model(
    dataset,
    epochs=params["epochs"],
    batch_size=params["batch_size"],
    lr=params["learning_rate"]
)

3. Parameters File (params.yaml):

yaml
download:
source_url: "s3://our-data-bucket/v1/"
max_retries: 5

training:
epochs: 100
batch_size: 32
learning_rate: 0.001
Execution Workflow:
bash

Run only download stage

dvc run download

Run only training stage (requires existing data)

dvc run train

Run full pipeline sequentially

dvc repro

Check status

dvc status
Dependency Management:
Data Download Outputs:

data/raw/ directory

Checksum file data/raw.dvc

Training Dependencies:

Requires existing data/raw

Monitors changes to:

Training script (src/train.py)

Parameter changes (params.yaml:training)

Reproducibility Features:

dvc.lock tracks all file hashes

dvc.yaml defines execution graph

params.yaml versions hyperparameters

Benefits:
Selective Execution:

Download new data without retraining

Retrain models with existing data

Clean Code Separation:

Single responsibility per script

Reusable components

Change Detection:

DVC skips unchanged stages

Automatic rebuild on parameter changes

Preprocessing Note:

Preprocessing remains inside training stage

Can be extracted later as separate stage

Migration Steps:
Refactor existing code into two files

Create params.yaml with necessary parameters

Initialize DVC pipeline:

bash
dvc init
dvc stage add -n download -d src/data_download.py -o data/raw -p download python src/data_download.py
dvc stage add -n train -d src/train.py,data/raw -o models -p training python src/train.py
Test pipeline variations:

bash
dvc repro download # Download only
dvc repro train # Training only (requires prior download)
Documentation Updates:
docs/dvc_workflow.md:

markdown

Two-Stage Workflow

Download Data

dvc repro download
Train Model
bash
dvc repro train
Modify Parameters
bash
# Change epochs
dvc params modify training.epochs 150

# Retrain with new params
dvc repro train
Check Dependencies
bash
dvc dag
# Expected: download → train