Implement "accounting" service that tracks user egress and updates Stripe

[travis]

When requests come into the gateway, the last thing they do will be to send a UCAN invocation (? should it be a UCAN invocation?) to a new "egress accounting service" that will record important metadata about the request and record the usage in our Stripe account.

While this seems like it might make sense as an AWS-based lambda service, I think it's also worth seriously considering writing this as a Cloudflare worker, since the rest of the read pipeline is in Cloudflare. If we go this route, we should consider alternatives to the AWS UCAN stream and will need to figure out how to connect to Stripe from Cloudflare apps (this should not be hard).

The accounting service may also be expected to update caches or datastores based on the invocations it receives.

This service will record usage in Stripe, but should also keep enough information about accounting to support our billing and product development processes.

Tasks for Egress Event Tracking and Billing Integration

• DynamoDB Table for Egress Events [Accounting Service] #147
• SQS Queue for Egress Events [Accounting Service] #148
• AWS Lambda Handler for Event Processing [Accounting Service] #149
• Stripe Integration for Billing [Accounting Service] #150
• Documentation [Accounting Service] #153
• Monitoring and Logging [Accounting Service] #152
• Testing and Validation [Accounting Service] #151
• Update the Gateway to track egress [Accounting Service] #154

[fforbeck]

Hey @travis and @alanshaw,
Check out the plan for the egress traffic system. Let me know if you're good with it, and I'll start the implementation.

In order to implement the Accounting Service that tracks egress traffic, we will establish a system with the following components:

1. DynamoDB Table for Egress Events:

   • Purpose: Store egress events, where each event represents a request served to a customer.
   • Schema: Include attributes such as customerId, resourceId, and timestamp to uniquely identify and track each request.

2. SQS Queue for Egress Events:

   • Purpose: Serve as the entry point for egress events pushed by the Freeway Cloudflare worker.
   • Functionality: Acts as a buffer to handle incoming events, ensuring they are processed in a reliable and scalable manner.

3. AWS Lambda Handler for Event Processing:

   • Purpose: Process events from the SQS queue and store them in the DynamoDB table.
   • Processing Strategy: Utilize immediate processing to handle events as they arrive, ensuring real-time tracking of egress requests.
   • Implementation: Save the event in DynamoDB for each request, maintaining an accurate tally of requests per customer.

4. Stripe Integration for Billing:

   • Purpose: Stripe provides a way to track usage-based billing through its Usage Records API. This allows us to bill customers based on the number of units they consume, such as API requests. We need to integrate Stripe to track egress traffic for billing purposes.
   • Method: For each processed event, increment the usage quantity by 1 in Stripe for the corresponding customer, ensuring accurate billing based on the number of requests served.

5. Testing and Validation:

   • Unit Testing: Implement unit tests for the Lambda function to ensure correct processing logic and error handling.
   • Integration Testing: Deploy to a test environment and simulate egress events to verify end-to-end functionality, including DynamoDB updates and Stripe integration.
   • Mocking External Services: Use mocks for AWS services and Stripe API during testing to simulate interactions and validate behavior without incurring costs.
   • Monitoring and Logging: Set up logging and monitoring to track the processing of events and identify any issues during testing and production.

6. Architecture Diagram:

   • Visual Representation: The following Mermaid diagram illustrates the architecture of the egress traffic accounting system:

graph TD;
    A[Freeway Cloudflare Worker] -->|Push Egress Events| B[SQS Queue];
    B -->|Trigger| C[AWS Lambda Function];
    C -->|Store Event| D[DynamoDB Table];
    C -->|Record Usage| E[Stripe API];
    subgraph AWS
        B
        C
        D
    end
    subgraph External
        A
        E
    end
    style A fill:#f9f,stroke:#333,stroke-width:2px;
    style B fill:#bbf,stroke:#333,stroke-width:2px;
    style C fill:#bbf,stroke:#333,stroke-width:2px;
    style D fill:#bbf,stroke:#333,stroke-width:2px;
    style E fill:#f9f,stroke:#333,stroke-width:2px;

Loading

[travis]

This looks great to me! The one thing it's missing is a discussion of operational monitoring - I think we need, at minimum, monitoring and alerting on the SQS queue to make sure we know when the reader (a lambda I assume?) is falling behind. We definitely also want alerting on any errors coming out of this system - this is probably as simple as making sure Sentry is configured for each of the components, but worth noting.

Before you start implementation could you create issues (probably sub-issues of this one - should be as simple as creating a "todo" list with new issues - see https://dev.to/keracudmore/create-sub-issues-in-github-issues-409m for more detail) for each of these tasks? We'll make sure to get them into the sprint starting this week.

[fforbeck]

This looks great to me! The one thing it's missing is a discussion of operational monitoring - I think we need, at minimum, monitoring and alerting on the SQS queue to make sure we know when the reader (a lambda I assume?) is falling behind. We definitely also want alerting on any errors coming out of this system - this is probably as simple as making sure Sentry is configured for each of the components, but worth noting.

Before you start implementation could you create issues (probably sub-issues of this one - should be as simple as creating a "todo" list with new issues - see https://dev.to/keracudmore/create-sub-issues-in-github-issues-409m for more detail) for each of these tasks? We'll make sure to get them into the sprint starting this week.

Sounds great! Thanks for reviewing it. I just updated the ticket description with the tasks.

[fforbeck]

Final Architecture:

Egress Traffic Tracking and Stripe Billing Meters API Integration

An asynchronous solution for tracking egress traffic in the Freeway Gateway and reporting events to Stripe’s Billing Meters API.

RFC Reference

Storacha Network is implementing a scalable, automated mechanism for tracking egress traffic and updating Stripe’s API with relevant data to ensure accurate customer billing. This RFC outlines the proposed approaches and their trade-offs.

Implementation

The selected approach for this implementation is Alternative 3:

• Freeway Worker invoking usage/record, SQS, Lambda with Stripe Integration

Flow

1. To avoid blocking, the Freeway Gateway asynchronously invokes the space/content/server/egress/record capability using ctx.waitUntil.
2. A new handler added to w3infra/upload-api/stores/usage.js receives the events from the Kinesis stream and places the egress data into an Egress Traffic SQS queue.
3. A new Lambda function consumes the events from the SQS queue and publishes the egress data to the Stripe Billing Meters API.

graph TD
    CF[Freeway Gateway] --> w3infra[w3infra/upload-api]
    w3infra --> Kinesis[Kinesis Stream]
    Kinesis --> EgressRecord[Space Content Handler]
    EgressRecord --> SQS[Egress Traffic Queue]
    SQS --> Lambda[Lambda Function]
    Lambda --> DynamoDB[Egress Traffic DynamoDB Table]
    Lambda --> Stripe[Stripe Billing Meters API]
    EgressRecord --> Logs[S3 Bucket /Receipts]

Loading

Key Advantages

• Asynchronous execution: ctx.waitUntil ensures the Freeway Worker won’t block, allowing seamless egress traffic handling.
• Decoupled architecture: By using SQS and Lambda, the architecture is decoupled and includes a buffer before publishing events to Stripe.
• Signed receipts: Each egress event generates a signed receipt, providing auditability and traceability. The receipts are stored in the general bucket in S3 where we already store other receipts.

[fforbeck]

This is done and deployed to Staging.