problems with the generate-traffic.sh script on the EC2 terminator-test-test-instance

[AstroTom]

Hi Chetz, finally got back to playing :-)

As I've noted before, when running the traffic generation about 99% of the traffic goes to the internet, instead of to S3, DDBand ECR. Here is some output from version v0.7.1 today:

Collected Traffic Sample

Total: 160 records, 0.16 GB

Service	Data (GB)	Percentage
S3	0.00	0.8%
DynamoDB	0.00	0.1%
ECR	0.00	0.0%
Other	0.16	99.0%

Cost Estimate

Projected from 5-minute sample to monthly estimate

NAT Gateway Rate: $0.0450 per GB

Metric	Amount
NAT Gateway Data Processing Cost	$60.95/month
S3 Endpoint Savings	$0.49/month
DynamoDB Endpoint Savings	$0.08/month
ECR Traffic Cost over NAT (no free endpoint)	$0.02/month
Estimated ECR Interface Endpoint Cost (2 endpoint(s), 1 AZ)	$14.40/month
└ Fixed hourly component	$14.40/month
└ Data processing component (0.36 GB/month)	$0.00/month
Total Potential Savings	$0.57/month

When I try for 15 minuutes the results are about the same.

I started looking at the code for generate-traffic.sh and I saw some issues. The main one is that the curl to get the region does not work:

REGION=$(curl -s http://169.254.169.254/latest/meta-data/placement/region)

This is because EC2 no longer supports the old IMDSv1 (with no token) by default, so the curl fails. You can either use the new IMDSv2 or simply add the --http-tokens optional option when creating (or modifying) the instance.

After I did that, I got 97% of the traffic going to ECR. ( I also changed the sleep from 10 seconds to 1. Not sure why any sleep is needed.

Total: 42 records, 0.14 GB

Service	Data (GB)	Percentage
S3	0.00	0.0%
DynamoDB	0.00	0.5%
ECR	0.14	97.3%
Other	0.00	2.2%

Cost Estimate

Projected from 5-minute sample to monthly estimate

NAT Gateway Rate: $0.0450 per GB

Metric	Amount
NAT Gateway Data Processing Cost	$55.98/month
S3 Endpoint Savings	$0.00/month
DynamoDB Endpoint Savings	$0.26/month
ECR Traffic Cost over NAT (no free endpoint)	$54.47/month
Estimated ECR Interface Endpoint Cost (2 endpoint(s), 1 AZ)	$26.50/month
└ Fixed hourly component	$14.40/month
└ Data processing component (1210.40 GB/month)	$12.10/month
Total Potential Savings	$0.26/month

I am still not sure why S3 and Dynamo don't get more traffic. In the case of S3, I think it is because of the small size (1MB) of the file. I created a 500MB files and manually copied it and did get reasonable results.

[AstroTom]

Looking back at issue 23 I see why ECR worked for you and not for me. You tested in us-east-1 and I was testing in eu-west-1. Presumably, when the curl failed and the region was not set, it defaulted to us-east-1.
I suggest you run all your tests in another region to catch those kind of errors.

[eranchetz]

Hi @AstroTom — glad you're back, and great timing! This is fixed in v0.7.2 🎉

Root causes found and fixed:

1. FROM public.ecr.aws/lambda/python:3.12 was the main culprit — the first Docker build pulled a ~500MB base image from public ECR through NAT. Those IPs don't match the private ECR service ranges, so they were classified as "Other". Fixed by switching the test Dockerfile to FROM scratch — no base image pull at all, just the 20MB random layer pushed to your private ECR repository.

2. S3 loop was too slow — the nested aws s3 cp loop (50 iterations × 10 files = 500 sequential AWS CLI invocations with full boto3 startup overhead each) couldn't complete even one full pass in a 5-minute window. Replaced with aws s3 sync — a single process that downloads all 10 test files, ~10× faster.

3. DynamoDB traffic was negligible — --max-items 10 returned only ~10 tiny items. Removed the limit; now scans all 100 items × 1KB payload ≈ 100KB per scan call.

4. Sed placeholder injection was broken — a YAML block scalar newline bug caused sed to fail silently, leaving __BUCKET_PLACEHOLDER__ in the script. Removed the entire placeholder/sed approach; the script now just requires positional args (which continuous-traffic.sh already passes correctly).

Expected results after v0.7.2:

• S3 traffic: meaningful percentage (not 0%)
• DynamoDB traffic: meaningful percentage (not 0%)
• ECR traffic: meaningful percentage to private ECR IPs (not 0%)
• "Other": should be a small minority, not 99%

To test:

# Deploy fresh test infrastructure
./test/scripts/deploy-test-infra.sh

# Start traffic generation
./test/scripts/continuous-traffic.sh start

# Run scan
./terminat scan deep --region us-east-1 --duration 5 --auto-approve

# Cleanup
./test/scripts/continuous-traffic.sh stop
./test/scripts/cleanup.sh --yes

Please give it a run and let us know how the traffic classification looks. Close the issue if it checks out — thanks for all the detailed investigation!

[AstroTom]

Hi Chetz, great to be back!

I tried again on v0.7.2. The ECR test is not working and takes about 30% but 69% is still "other" with S3 and Dynamo are less than 1 %.
:-(

I started with a clean new ./test/scripts/deploy-test-infra.sh

Sample output:

Traffic Sample

• Duration: 5 minute(s)
• Total: 184 records, 0.63 GB
• S3: 0.00 GB (0.4%)
• DynamoDB: 0.00 GB (0.1%)
• ECR: 0.14 GB (22.6%)
• Other: 0.49 GB (76.9%)

Cost Estimate (projected from sample)

• NAT data processing rate: $0.0450 per GB
• NAT Gateway Data Processing Cost: $246.02/month
• S3 savings potential: $0.96/month
• DynamoDB savings potential: $0.32/month
• Total savings potential: $1.27/month ($15.29/year)

[AstroTom]

BTW, I tested in both Virginia and Ireland with the same results.