AWS Commitment Utilization Data Discovery

[JGrubb]

Summary

Research AWS APIs for Savings Plans and Reserved Instances utilization data to determine optimal pipeline structure and update frequency.

Background

While inventory data (Issues #38, #39) tracks active commitments, utilization data shows how effectively those commitments are being used. This is crucial for cost optimization analysis but requires understanding data granularity and API behavior.

Research Objectives

1. Data Granularity Analysis

Question: What granularity is available and practical?

• Hourly vs daily utilization rates
• Historical data availability and retention
• API response sizes and pagination

2. API Response Structure Investigation

APIs to Research:

Cost Explorer API (boto3.client('ce'))

• get_savings_plans_utilization() - Savings Plans usage
• get_reservation_utilization() - Reserved Instance usage
• get_reservation_coverage() - Coverage analysis

Savings Plans API (boto3.client('savingsplans'))

• describe_savings_plans_utilization_details() - Detailed utilization

3. Pipeline Architecture Questions

• Update frequency: How often does utilization data change?
• Data volume: How much data per API call?
• Cost implications: API request costs at various frequencies
• State tracking: Do we need deduplication for utilization data?

Research Tasks

Phase 1: API Response Analysis

• Call each utilization API with test account
• Document response structure and data volume
• Identify available time granularities (hourly/daily)
• Test historical data availability periods

Phase 2: Pipeline Requirements

• Determine optimal update frequency based on data freshness
• Estimate API costs for different polling schedules
• Design table structure based on actual response payloads
• Identify relationships with inventory data (Issues AWS Savings Plans Inventory Pipeline #38, AWS Reserved Instances Inventory Pipeline #39)

Phase 3: Integration Strategy

• Recommend schema organization (same aws_commitments or separate?)
• Design unified views combining inventory + utilization
• Plan CLI commands and user workflow

Deliverables

• Documentation of API response structures and granularities
• Recommended pipeline architecture and update frequency
• Cost analysis for different polling strategies
• Implementation roadmap for utilization pipelines

Success Criteria

• Clear understanding of utilization data characteristics
• Defined pipeline structure for both Savings Plans and RI utilization
• Cost-effective polling strategy recommendation
• Ready for implementation in Phase 5

Related Work

• Issue AWS Savings Plans Inventory Pipeline #38: AWS Savings Plans Inventory Pipeline
• Issue AWS Reserved Instances Inventory Pipeline #39: AWS Reserved Instances Inventory Pipeline
• Issue Does the open-finops-stack intends to show cost optimization recommendations? #10: Cost Optimization Recommendations (Phase 5)

Dependencies

• AWS account with active Savings Plans or Reserved Instances for testing
• Cost Explorer API access

Target Phase: Phase 5 (Production Features) - Research phase