Skip to content

T-22776 [CI] One-off probe: Firehose delivery errors without logs:CreateLogGroup (not for merging) - #20

Closed
PetrHeinz wants to merge 1 commit into
claude/t-22776-lambda-log-groupsfrom
claude/t-22776-firehose-error-probe
Closed

PetrHeinz wants to merge 1 commit into
claude/t-22776-lambda-log-groupsfrom
claude/t-22776-firehose-error-probe

Conversation

@PetrHeinz

@PetrHeinz PetrHeinz commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

Not for merging. One run that answers the question on #18: does Firehose still write its delivery errors into /aws/kinesisfirehose/better-stack-logs and /aws/kinesisfirehose/better-stack-metrics when better-stack-firehose-role has no logs:CreateLogGroup?

Result: yes. Run https://github.com/BetterStackHQ/aws/actions/runs/43446705641 passed:

  • better-stack-firehose-role had no managed policy and only logs:CreateLogStream and logs:PutLogEvents.
  • Both DestinationDelivery log streams received Firehose's error events, the first one in better-stack-logs 67 s after the first record went in:
{"deliveryStreamARN":"arn:aws:firehose:us-east-1:251917137055:deliverystream/better-stack-logs","destination":"https://eu-central-1a-aws-eu-central-1.betterstackdata.com/aws-firehose...","deliveryStreamVersionId":"1","message":"Delivery to the endpoint was unsuccessful. See Troubleshooting HTTP Endpoints in the Firehose documentation for more information. Response received with status code. 401: {\"timestamp\":1794582235000,\"requestId\":\"86f618cb-ea20-44b6-b1a8-ee326d900771\",\"errorMessage\":\"Unauthorized\"}","errorCode":"HttpEndpoint.DestinationException","processor":"arn:aws:lambda:us-east-1:251917137055:function:better-stack-logs:$LATEST"}
  • The rejected batches landed in the backup bucket under logs/errors/http-endpoint-failed/ and metrics/errors/http-endpoint-failed/.
  • CloudTrail saw eight CreateLogGroup calls during the run, all successful and all from the session that deployed the stack. Firehose made none, allowed or denied.

How the run forces the error: this branch sits on top of #18 and replaces .github/workflows/e2e.yml with a probe.

  • The stack is deployed from the checkout with a source token ingest does not know and FirehoseRetryDurationSeconds=0, so every Firehose batch is rejected with HTTP 401.
  • The metric stream feeds better-stack-metrics by itself. better-stack-logs gets hand-made CloudWatch Logs subscription records through put-record.
  • A step fails if the role can create log groups, so the run cannot pass for the wrong reason.

The usual cleanup deleted the stack and the source.

🤖 Generated with Claude Code

…merging)

Replaces the E2E on this branch with one run that deploys the stack with a
source token ingest does not know and retries off, then checks that Firehose
writes its delivery errors into both /aws/kinesisfirehose/better-stack-* log
groups while better-stack-firehose-role has no logs:CreateLogGroup.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant