[Build] Add S3 dvc remote

[rojberr]

User Story:

As a machine learning engineer,
I want to configure an S3 bucket as a DVC remote for our ML project,
So that we can securely version large datasets/models, enable team collaboration, and reduce local storage requirements.

---

Acceptance Criteria:

0. Do a cost analysis
1. S3 bucket created with proper IAM permissions and bucket policies
2. DVC configured to use S3 as default remote (dvc remote add)
3. Authentication credentials securely stored (AWS CLI profile or env variables)
4. Existing data pushed to S3 remote (dvc push)
5. Cost monitoring alerts configured in AWS
6. Documentation for team members on S3 remote usage

---

Definition of Done:

• All acceptance criteria met
• Code reviewed and approved
• Validation tests for push/pull operations
• Documentation updated in /docs/data_management.md
• Team members can successfully pull data from S3

---

[rojberr]

EXAMPLE Implementation Plan:

S3 Configuration:

# Create S3 bucket (replace with actual name)
aws s3api create-bucket --bucket ml-project-dvc-store --region us-east-1

# Apply bucket policy (example)
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {"AWS": "arn:aws:iam::ACCOUNT_ID:user/DATA_SCIENTIST"},
      "Action": ["s3:PutObject", "s3:GetObject"],
      "Resource": "arn:aws:s3:::ml-project-dvc-store/*"
    }
  ]
}
DVC Setup:

bash
dvc remote add -d s3remote s3://ml-project-dvc-store
dvc remote modify s3remote profile my-aws-profile  # or use env vars
dvc push -r s3remote
Cost Planning & Optimization:
Storage Cost Projection (AWS us-east-1):

Component	Size	Cost/Month	Notes
Model weights	50GB	$1.15	Standard storage ($0.023/GB)
Training datasets	1TB	$23.00	Standard storage
Validation assets	200GB	$4.60	Standard storage
Data transfer OUT	100GB	$9.00	($0.09/GB after 1GB free)
Requests	10K PUT	$0.05	($0.005/1k requests)
Total Estimated	-	$37.80	-
Cost Optimization Strategies:

Lifecycle Rules:

Transition to Glacier after 90 days (70% savings)

Auto-delete failed experiment data after 30 days

Storage Classes:

Use Intelligent-Tiering for frequently accessed data

Archive older versions to S3 Glacier Deep Archive ($0.00099/GB)

Monitoring:

Configure CloudWatch billing alerts at $30 and $50 thresholds

Set up Cost Explorer reports

Access Controls:

Restrict upload permissions to CI service accounts

Use SCPs to prevent bucket deletion

Security Requirements:
🔒 IAM roles with least-privilege access (S3 only)

🔑 Credentials stored in GitHub Secrets (not in code)

🛡️ Bucket versioning enabled for data recovery

🚫 Public access blocked (Block Public Access enabled)

🔄 Server-side encryption (SSE-S3) enabled

Validation Steps:
Test push/pull operations:

bash
dvc push -r s3remote
dvc pull -r s3remote
Verify file integrity:

bash
dvc status --cloud
Check AWS Cost Explorer after 24 hours

Test team member access from clean environment

Documentation Updates:
Add to data_management.md:

markdown
## S3 Remote Usage

1. **Configure AWS credentials**:
   ```bash
   aws configure --profile ml-project
Pull data:

bash
dvc pull -r s3remote
Add new data:

bash
dvc add data/new-dataset
dvc push -r s3remote