AWS Reserved Instances Inventory Pipeline

[JGrubb]

Summary

Create a DLT pipeline to ingest AWS Reserved Instances inventory data across EC2, RDS, and other services to provide comprehensive commitment tracking.

Background

While AWS is transitioning to Savings Plans, Reserved Instances remain active and important for cost analysis. Many organizations have significant RI commitments that need tracking alongside billing data.

Scope

Data Sources: Reserved Instances inventory (current state)
Starting Point: EC2 Reserved Instances, expand to RDS/ElastiCache/etc.
Granularity: Daily pipeline runs (reservations change infrequently)
Schema: Separate schema from aws_billing (e.g., aws_commitments)
Relationships: Link to CUR data via account IDs and resource identifiers

AWS APIs to Integrate

EC2 Reserved Instances (boto3.client('ec2'))

• describe_reserved_instances() - Active reservations
• describe_reserved_instances_offerings() - Available offerings
• describe_reserved_instances_modifications() - Modification history

Future Expansion

• RDS: boto3.client('rds').describe_reserved_db_instances()
• ElastiCache: boto3.client('elasticache').describe_reserved_cache_nodes()
• Redshift: boto3.client('redshift').describe_reserved_nodes()

Implementation Plan

Phase 1: EC2 Reserved Instances

• Create aws_reserved_instances_source() DLT source
• Implement EC2 RI inventory collection using API response payloads
• Create tables in aws_commitments schema
• Add CLI command: ./finops aws import-reserved-instances

Phase 2: Multi-Service Expansion

• Add RDS Reserved DB Instances
• Add ElastiCache Reserved Cache Nodes
• Add Redshift Reserved Nodes
• Implement service-specific table organization

Phase 3: Data Enrichment

• Add offering details and pricing
• Implement state tracking (reservations rarely change)
• Create unified view with billing data

Success Criteria

• Daily pipeline ingests current Reserved Instances inventory
• Data structure follows AWS API response payloads
• CLI integration matches existing AWS pipeline UX
• Tables ready for joining with CUR data analysis
• Supports multiple AWS services (EC2, RDS, etc.)

Related Work

• Issue AWS Savings Plans Inventory Pipeline #38: AWS Savings Plans Inventory Pipeline
• Utilization Data Discovery (separate issue)
• Issue Does the open-finops-stack intends to show cost optimization recommendations? #10: Cost Optimization Recommendations (Phase 5)

Dependencies

• Existing AWS pipeline infrastructure
• DLT and DuckDB setup

Target Phase: Phase 5 (Production Features)