README.md

DVC Feature Store Documentation

Overview

A centralized feature store built on DVC for versioning, sharing, and validating ML features across teams.

Architecture

┌─────────────────────────────────────────────────────────────┐
│                      Feature Store                          │
├─────────────────────────────────────────────────────────────┤
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────────────┐  │
│  │   CLI       │  │  Registry   │  │  Schema Validator   │  │
│  │  Interface  │──│  Manager    │──│                     │  │
│  └─────────────┘  └─────────────┘  └─────────────────────┘  │
│         │                │                    │              │
│         ▼                ▼                    ▼              │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────────────┐  │
│  │    DVC      │  │  Storage    │  │   Pydantic Models   │  │
│  │ Versioning  │  │  Manager    │  │                     │  │
│  └─────────────┘  └─────────────┘  └─────────────────────┘  │
└─────────────────────────────────────────────────────────────┘
                          │
                          ▼
              ┌───────────────────────┐
              │    Remote Storage     │
              │  (S3 / GCS / Azure)   │
              └───────────────────────┘

Quick Start

Installation

pip install -e .

Initialize Feature Store

feature-store init --remote-url s3://your-bucket/features

Add a Feature

feature-store feature add customer demographics data/customer_demographics.parquet \
  --schema schemas/customer_demographics.yaml \
  --tag production

List Features

feature-store list

Validate Feature

feature-store feature validate customer/demographics

Push to Remote

feature-store push

Pull from Remote

feature-store pull

Feature Schema Format

name: customer_demographics
version: "1.0.0"
description: Customer demographic features
primary_keys:
  - customer_id
event_timestamp_column: updated_at
columns:
  - name: customer_id
    dtype: string
    nullable: false
    description: Unique customer identifier
  - name: age
    dtype: int64
    nullable: false
  - name: country
    dtype: string
    nullable: true
    default_value: "unknown"

Supported Data Types

Type	Description
int32	32-bit integer
int64	64-bit integer
float32	32-bit float
float64	64-bit float
string	Text data
boolean	True/False
datetime	Timestamp
category	Categorical

CI/CD Integration

The feature store includes GitHub Actions workflows for:

1. Code Quality - Linting with Ruff, type checking with MyPy
2. Unit Tests - pytest with coverage reporting
3. Feature Validation - Schema validation for changed features
4. DVC Integrity - Verify .dvc files and registry consistency

Best Practices

1. One schema per domain - Keep feature_schema.yaml in each domain directory
2. Version schemas - Use semantic versioning for schema changes
3. Tag features - Use tags for environment (production, staging) and purpose
4. Validate before commit - Run feature-store feature validate locally
5. Use pre-commit hooks - Install with pre-commit install