Minimum Viable Training and Inference Pipelines

[rahibAmin84]

Setup up automated training and inference pipelines

[rahibAmin84]

includes ROI calculations and interactive features but lacks a structured workflow for training and inference. To improve reproducibility, scalability, and experimentation, we need to implement Minimum Viable Training and Inference Pipelines (MVP pipelines).

Objectives:
1. Training Pipeline:
• Load raw datasets (from cloud storage or local).
• Preprocess and clean the data (e.g., encode categorical variables, normalize numeric features).
• Train the Bayesian regression model (or other chosen predictive models).
• Save trained model artifacts (e.g., posterior samples, model coefficients) to cloud storage.
2. Inference Pipeline:
• Load trained model artifacts.
• Accept new input features from users.
• Generate predicted ROI with uncertainty estimates.
• Save inference results back to cloud storage.
• Optionally, provide visualizations (posterior distribution, confidence intervals).

Requirements:
• Use modular Python scripts or a lightweight workflow orchestrator (e.g., make, snakemake, or Python functions).
• Ensure pipelines are reproducible, well-documented, and easy to run for lab experiments.
• Integrate with cloud storage helper class to handle input/output datasets.
• Include basic logging for pipeline steps.
• Provide example scripts/notebooks to demonstrate usage.

Benefits:
• Enables systematic experimentation with different models and hyperparameters.
• Supports reproducibility and collaboration across the team.
• Prepares the project for future enhancements like automated retraining or batch inference.