Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use asg warm pools for faster buildkite job starts #822

Open
nitrocode opened this issue Apr 8, 2021 · 6 comments · May be fixed by #838
Open

Use asg warm pools for faster buildkite job starts #822

nitrocode opened this issue Apr 8, 2021 · 6 comments · May be fixed by #838
Labels
agent lifecycle Agent boot, job lifecycle, agent shutdown asg-initiated-termination enhancement

Comments

@nitrocode
Copy link
Contributor

nitrocode commented Apr 8, 2021 •
edited
Loading

https://aws.amazon.com/about-aws/whats-new/2021/04/amazon-ec2-auto-scaling-introduces-warm-pools-accelerate-scale-out-while-saving-money/

https://aws.amazon.com/blogs/compute/scaling-your-applications-faster-with-ec2-auto-scaling-warm-pools/

If we could keep X instances warmed up, we could start jobs much faster without having to set the min count on the asg to something non-zero.

It should be added soon to cloudformation: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-as-group.html
@chloeruka
Copy link
Contributor

Woah! You're quick. Yeah this feature looks great; especially for Windows instances that take a while to boot. According to the announcement it's not yet available on Cloudformation, but when it is we'll take a look into it.

@nitrocode
Copy link
Contributor Author

nitrocode commented May 4, 2021 •
edited
Loading

The only param that would require an additional input to use the warm pool would be the min size of it. We use an asg min size of 0 and max size of 10 so a warm pool min size of 3 seems reasonable.

The cloudformation docs have been released.

We could add the following

Parameters:
  WarmPoolMinSize:
    Description: Minimum number of instances in warm pool
    Type: Number
    Default: 0

Conditions:
    UseWarmPool:
      !Not [ !Equals [ !Ref WarmPoolMinSize, 0 ] ]

Resources:
  WarmPool: 
    Type: AWS::AutoScaling::WarmPool
    Condition: UseWarmPool
    Properties:
      AutoScalingGroupName: !Ref AgentAutoScaleGroup
      MinSize: !Ref WarmPoolMinSize
      PoolState: Stopped

What do you folks think?

@nitrocode nitrocode linked a pull request May 4, 2021 that will close this issue
Open
@theonlysinjin
Copy link

This is fantastic.
I've been using the Cloudwatch metrics to scale ASG when there are no idle agents (ie all are busy now), to help beat the agent start time.
Though it seems our agents now start (with the update) in just under 2 minutes which is pretty good.

@keithduncan keithduncan mentioned this issue Jul 5, 2021
Closed
@keithduncan keithduncan added the agent lifecycle Agent boot, job lifecycle, agent shutdown label Sep 6, 2021
@dieend
Copy link

dieend commented Nov 9, 2021

Warm Pool is now available in CloudFormation https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-autoscaling-warmpool.html

Right now I can't manually update the created ASG from the current template because the template configuring MixedInstancesPolicy

elastic-ci-stack-for-aws/templates/aws-stack.yml

Lines 1038 to 1060 in 7b3d02c

MixedInstancesPolicy:
InstancesDistribution:
OnDemandPercentageAboveBaseCapacity: !If [ SpotPriceSet, 0, !Ref OnDemandPercentage ]
SpotAllocationStrategy: capacity-optimized
SpotMaxPrice: !If [SpotPriceSet, !Ref SpotPrice, !Ref "AWS::NoValue"]
LaunchTemplate:
LaunchTemplateSpecification:
LaunchTemplateId: !Ref AgentLaunchTemplate
Version: !GetAtt "AgentLaunchTemplate.LatestVersionNumber"
Overrides:
- InstanceType: !Select [ "0", !Split [ ",", !Join [ ",", [ !Ref InstanceType, "", "", "" ] ] ] ]
- !If
- UseInstanceType2
- InstanceType: !Select [ "1", !Split [ ",", !Join [ ",", [ !Ref InstanceType, "", "", "" ] ] ] ]
- !Ref "AWS::NoValue"
- !If
- UseInstanceType3
- InstanceType: !Select [ "2", !Split [ ",", !Join [ ",", [ !Ref InstanceType, "", "", "" ] ] ] ]
- !Ref "AWS::NoValue"
- !If
- UseInstanceType4
- InstanceType: !Select [ "3", !Split [ ",", !Join [ ",", [ !Ref InstanceType, "", "", "" ] ] ] ]
- !Ref "AWS::NoValue"

We have to remove them if we'd like to use WarmPool

@keithduncan keithduncan mentioned this issue Nov 22, 2021
Closed
7 tasks
@aiven-amartin
Copy link

Any progress/caveats on this issue? Would be really nice to have this option rather than trying to bake git checkouts/other slow workspace parts into the AMI itself.

@alex-shakouri-ai
Copy link

This would be helpful on our side as well! Would help out with ensuring we have agents on a warm start up to be able to start jobs faster!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
agent lifecycle Agent boot, job lifecycle, agent shutdown asg-initiated-termination enhancement
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add Warm Pool nitrocode/elastic-ci-stack-for-aws
7 participants
@keithduncan @dieend @theonlysinjin @nitrocode @chloeruka @alex-shakouri-ai @aiven-amartin