Skip to content

Commit 7c7d508

Browse files
shub-krisshub-kris
authored
Add packer support for building AWS AMI (#441)
* Add packer support for bulding AWS AMI * Update README * Add slack notification on failure * Add comment to explain workflow
1 parent 7e8a33b commit 7c7d508

File tree

9 files changed

+329
-0
lines changed

9 files changed

+329
-0
lines changed

‎.github/workflows/build-ami.yml

+82
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
# The workflow file for building the AWS Neuron AMI using Packer
2+
# It can be triggered by push and pull request to main when changes made to infrastructure/ami folder, manually and scheduler.
3+
name: Build AWS Neuron AMI
4+
on:
5+
push:
6+
branches:
7+
- main
8+
paths:
9+
- 'infrastructure/ami/**'
10+
pull_request:
11+
branches:
12+
- main
13+
paths:
14+
- 'infrastructure/ami/**'
15+
workflow_dispatch:
16+
inputs:
17+
tag:
18+
description: 'Tag to use for the AMI build'
19+
default: 'main'
20+
schedule:
21+
# Schedule the workflow to run every second day at midnight UTC
22+
- cron: '0 0 */2 * *'
23+
24+
jobs:
25+
build-ami:
26+
defaults:
27+
run:
28+
working-directory: infrastructure/ami
29+
runs-on: ubuntu-latest
30+
env:
31+
AWS_REGION: us-east-1
32+
steps:
33+
- name: Checkout
34+
uses: actions/checkout@v3
35+
with:
36+
# If the workflow is triggered manually or by schedule, uses the tag, otherwise uses the current branch
37+
ref: ${{ github.event.inputs.tag || github.ref }}
38+
39+
- name: Setup Packer
40+
uses: hashicorp/setup-packer@main
41+
42+
- name: configure aws credentials
43+
uses: aws-actions/configure-aws-credentials@v1
44+
with:
45+
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID_BUILD_AMI }}
46+
aws-secret-access-key: ${{ secrets.AWS_ACCESS_KEY_SECRET_BUILD_AMI }}
47+
aws-region: ${{ env.AWS_REGION }}
48+
49+
- name: Packer format
50+
id: format
51+
run: packer fmt hcl2-files
52+
continue-on-error: true
53+
54+
- name: Packer Init
55+
id: init
56+
run: packer init hcl2-files
57+
continue-on-error: true
58+
59+
- name: Packer Validate
60+
id: validate
61+
# If the workflow is triggered manually or scheduled, uses the tag, otherwise uses the main branch of optimum-neuron repo for building the AMI
62+
run: packer validate -var "optimum_version=${{ github.event.inputs.tag || github.event.repository.default_branch }}" -var "region=${{ env.AWS_REGION }}" hcl2-files
63+
continue-on-error: true
64+
65+
- name: Packer Build
66+
id: build
67+
# If the workflow is triggered manually or scheduled, uses the tag, otherwise uses the main branch of optimum-neuron repo for building the AMI
68+
run: |
69+
packer build -var "optimum_version=${{ github.event.inputs.tag || github.event.repository.default_branch }}" -var "region=${{ env.AWS_REGION }}" hcl2-files
70+
71+
- name: Slack Notification on Failure
72+
id: slack
73+
uses: slackapi/[email protected]
74+
if: ${{ failure() && github.event_name == 'schedule' }}
75+
with:
76+
channel-id: 'C06GAEQJLNN' #copied from slack channel
77+
payload: |
78+
{
79+
"text": "GitHub Action HuggingFace Neuron AMI Build result: ${{job.status}}"
80+
}
81+
env:
82+
SLACK_BOT_TOKEN: ${{ secrets.SLACK_CIFEEDBACK_BOT_TOKEN }}

‎infrastructure/ami/README.md

+78
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
# Building AMI with Packer
2+
3+
This directory contains the files for building AMI using [Packer](https://github.com/hashicorp/packer) that is later published as a AWS Marketplace asset.
4+
5+
6+
## Folder Structure
7+
8+
- [hcl2-files](./hcl2-files/) - Includes different files which are used by a Packer pipeline to build an AMI. The files are:
9+
- [build.pkr.hcl](./hcl2-files/build.pkr.hcl): contains the [build](https://developer.hashicorp.com/packer/docs/templates/hcl_templates/blocks/build) block, defining the builders to start, provisioning them using [provisioner](https://developer.hashicorp.com/packer/docs/templates/hcl_templates/blocks/build/provisioner), and specifying actions to take with the built artifacts using `post-process`.
10+
- [variables.pkr.hcl](./hcl2-files/variables.pkr.hcl): contains the [variables](https://developer.hashicorp.com/packer/docs/templates/hcl_templates/blocks/variable) block, defining variables within your Packer configuration.
11+
- [sources.pkr.hcl](./hcl2-files/sources.pkr.hcl): contains the [source](https://developer.hashicorp.com/packer/docs/templates/hcl_templates/blocks/source) block, defining reusable builder configuration blocks.
12+
- [packer.pkr.hcl](./hcl2-files/packer.pkr.hcl): contains the [packer](https://developer.hashicorp.com/packer/docs/templates/hcl_templates/blocks/packer) block, used to configure some behaviors of Packer itself, such as the minimum required Packer version needed to apply to your configuration.
13+
- [scripts](./scripts): contains scripts used by [provisioner](https://developer.hashicorp.com/packer/docs/templates/hcl_templates/blocks/build/provisioner) for installing additonal packages/softwares.
14+
15+
16+
### Prerequisites
17+
- [Packer](https://developer.hashicorp.com/packer/docs/intro): Packer is an open source tool for creating identical machine images for multiple platforms from a single source configuration.
18+
19+
- AWS Credentials: You need to have AWS credentials configured on your machine. You can configure AWS credentials using [AWS CLI](https://github.com/aws/aws-cli) or by setting environment variables.
20+
21+
#### Install Packer on Ubuntu/Debian
22+
```bash
23+
curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
24+
sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
25+
sudo apt-get update && sudo apt-get install packer
26+
```
27+
28+
You can also install Packer for other OS from [here](https://developer.hashicorp.com/packer/tutorials/docker-get-started/get-started-install-cli).
29+
30+
#### Configure AWS Credentials
31+
32+
Using Environment Variables:
33+
```bash
34+
export AWS_ACCESS_KEY_ID=<access_key>
35+
export AWS_SECRET_ACCESS_KEY=<secret_key>
36+
```
37+
38+
Using AWS CLI:
39+
```bash
40+
aws configure sso
41+
```
42+
43+
There are other ways to configure AWS credentials. You can read more about it [here](https://github.com/aws/aws-cli?tab=readme-ov-file#configuration).
44+
45+
### Build AMI
46+
47+
#### Format Packer blocks
48+
You can format your HCL2 files locally. This command will update your files in place.
49+
50+
Format a single file:
51+
```bash
52+
packer fmt build.pkr.hcl
53+
```
54+
55+
Format all files in a directory:
56+
```bash
57+
packer fmt ./hcl2-files
58+
```
59+
60+
#### Validate Packer blocks
61+
You can validate the syntax and configuration of your files locally. This command will return a zero exit status on success, and a non-zero exit status on failure.
62+
63+
```bash
64+
packer validate -var 'region=us-west-2' -var 'optimum_version=v0.0.17' ./hcl2-files
65+
```
66+
67+
#### Run Packer build
68+
You can run Packer locally. This command will build the AMI and upload it to AWS.
69+
70+
You need to set variables with no default values using `-var` flag. For example:
71+
```bash
72+
packer build -var 'region=us-west-2' -var 'optimum_version=v0.0.17' ./hcl2-files
73+
```
74+
75+
To trigger a github action workflow manually, you can use GitHub CLI:
76+
```bash
77+
gh workflow run build-ami.yml -f tag=<tag>
78+
```

‎infrastructure/ami/hcl2-files/build.pkr.hcl

+29
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
build {
2+
name = "build-hf-dl-neuron"
3+
sources = [
4+
"source.amazon-ebs.ubuntu"
5+
]
6+
provisioner "shell" {
7+
script = "scripts/validate-neuron.sh"
8+
}
9+
provisioner "shell" {
10+
script = "scripts/install-huggingface-libraries.sh"
11+
environment_vars = [
12+
"TRANSFORMERS_VERSION=${var.transformers_version}",
13+
"OPTIMUM_VERSION=${var.optimum_version}",
14+
]
15+
}
16+
provisioner "shell" {
17+
inline = ["echo 'source /opt/aws_neuron_venv_pytorch/bin/activate' >> /home/ubuntu/.bashrc"]
18+
}
19+
provisioner "file" {
20+
source = "scripts/welcome-msg.sh"
21+
destination = "/tmp/99-custom-message"
22+
}
23+
provisioner "shell" {
24+
inline = [
25+
"sudo mv /tmp/99-custom-message /etc/update-motd.d/",
26+
"sudo chmod +x /etc/update-motd.d/99-custom-message",
27+
]
28+
}
29+
}

‎infrastructure/ami/hcl2-files/packer.pkr.hcl

+8
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
packer {
2+
required_plugins {
3+
amazon = {
4+
version = ">= 1.2.8"
5+
source = "github.com/hashicorp/amazon"
6+
}
7+
}
8+
}

‎infrastructure/ami/hcl2-files/sources.pkr.hcl

+15
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
source "amazon-ebs" "ubuntu" {
2+
ami_name = "huggingface-neuron-{{isotime \"2006-01-02T15-04-05Z\"}}"
3+
instance_type = var.instance_type
4+
region = var.region
5+
source_ami = var.source_ami
6+
ssh_username = var.ssh_username
7+
launch_block_device_mappings {
8+
device_name = "/dev/sda1"
9+
volume_size = 512
10+
volume_type = "gp2"
11+
delete_on_termination = true
12+
}
13+
ami_users = var.ami_users
14+
ami_regions = var.ami_regions
15+
}

‎infrastructure/ami/hcl2-files/variables.pkr.hcl

+54
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
variable "region" {
2+
description = "The AWS region"
3+
type = string
4+
}
5+
6+
variable "instance_type" {
7+
default = "trn1.2xlarge"
8+
description = "EC2 machine type for building AMI"
9+
type = string
10+
}
11+
12+
variable "source_ami" {
13+
default = "ami-0fbea04d7389bcd4e"
14+
description = "Base Image"
15+
type = string
16+
/*
17+
To get latest value, run the following command:
18+
aws ec2 describe-images \
19+
--region us-east-1 \
20+
--owners amazon \
21+
--filters 'Name=name,Values=Deep Learning AMI Neuron PyTorch 1.13 (Ubuntu 20.04) ????????' 'Name=state,Values=available' \
22+
--query 'reverse(sort_by(Images, &CreationDate))[:1].ImageId' \
23+
--output text
24+
*/
25+
}
26+
27+
variable "ssh_username" {
28+
default = "ubuntu"
29+
description = "Username to connect to SSH with"
30+
type = string
31+
}
32+
33+
variable "optimum_version" {
34+
description = "Optimum Neuron version to install"
35+
type = string
36+
}
37+
38+
variable "transformers_version" {
39+
default = "4.36.2"
40+
description = "Transformers version to install"
41+
type = string
42+
}
43+
44+
variable "ami_users" {
45+
default = ["754289655784", "558105141721"]
46+
description = "AWS accounts to share AMI with"
47+
type = list(string)
48+
}
49+
50+
variable "ami_regions" {
51+
default = ["eu-west-1"]
52+
description = "AWS regions to share AMI with"
53+
type = list(string)
54+
}

‎infrastructure/ami/scripts/install-huggingface-libraries.sh

+39
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
#!/bin/bash
2+
3+
# Activate the neuron virtual environment
4+
source /opt/aws_neuron_venv_pytorch/bin/activate
5+
6+
echo "Step: install-hugging-face-libraries"
7+
8+
echo "TRANSFORMERS_VERSION: $TRANSFORMERS_VERSION"
9+
echo "OPTIMUM_VERSION: $OPTIMUM_VERSION"
10+
11+
pip install --upgrade --no-cache-dir \
12+
"transformers[sklearn,sentencepiece,vision]==$TRANSFORMERS_VERSION" \
13+
"datasets==2.16.1" \
14+
"accelerate==0.23.0" \
15+
"diffusers==0.25.0" \
16+
"evaluate==0.4.1" \
17+
"requests==2.31.0" \
18+
"notebook==7.0.6" \
19+
"markupsafe==2.1.1" \
20+
"jinja2==3.1.2" \
21+
"attrs==23.1.0"
22+
23+
echo 'export PATH="${HOME}/.local/bin:$PATH"' >> "${HOME}/.bashrc"
24+
25+
echo "Step: install-and-copy-optimum-neuron-examples"
26+
git clone -b $OPTIMUM_VERSION https://github.com/huggingface/optimum-neuron.git
27+
28+
cd optimum-neuron
29+
python setup.py install
30+
cd ..
31+
32+
mkdir /home/ubuntu/huggingface-neuron-samples/ /home/ubuntu/huggingface-neuron-notebooks/
33+
mv optimum-neuron/examples/* /home/ubuntu/huggingface-neuron-samples/
34+
mv optimum-neuron/notebooks/* /home/ubuntu/huggingface-neuron-notebooks/
35+
rm -rf optimum-neuron
36+
chmod -R 777 /home/ubuntu/huggingface-neuron-samples /home/ubuntu/huggingface-neuron-notebooks
37+
38+
echo "Step: validate-imports-of-huggingface-libraries"
39+
bash -c 'python -c "import transformers;import datasets;import accelerate;import evaluate;import tensorboard; import torch;"'

‎infrastructure/ami/scripts/validate-neuron.sh

+13
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
#!/bin/bash
2+
echo "Step: validate-neuron-devices"
3+
neuron-ls
4+
5+
# Activate the neuron virtual environment
6+
source /opt/aws_neuron_venv_pytorch/bin/activate
7+
8+
python -c 'import torch'
9+
python -c 'import torch_neuronx'
10+
11+
echo "Installing Tensorboard Plugin for Neuron"
12+
pip install --upgrade --no-cache-dir \
13+
"tensorboard-plugin-neuronx"

‎infrastructure/ami/scripts/welcome-msg.sh

+11
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
#!/bin/bash
2+
printf "=============================================================================\n"
3+
printf " __| __|_ )\n"
4+
printf " _| ( / HuggingFace Deep Learning Neuron AMI (Ubuntu 20.04)\n"
5+
printf " ___|\___|___|\n"
6+
printf "=============================================================================\n"
7+
printf "Welcome to the HuggingFace Deep Learning Neuron AMI (Ubuntu 20.04)\n"
8+
printf "* Examples: /home/ubuntu/huggingface-neuron-samples \n"
9+
printf "* Notebooks: /home/ubuntu/huggingface-neuron-notebooks \n"
10+
printf "* Documentation: https://huggingface.co/docs/optimum-neuron/ \n"
11+
printf "=============================================================================\n"

0 commit comments

Comments
 (0)