0. Environment Setup.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Environment Setup\n",
    "\n",
    "> *This notebook should work well in the `Python 3 (Data Science)` kernel in SageMaker Studio, or `conda_python3` in SageMaker Notebook Instances*\n",
    "\n",
    "Amazon Forecast will access source data from (and export forecasts to) Amazon S3... So before we start using Forecast, we should set up our data bucket(s) and permissions.\n",
    "\n",
    "Production environments will typically automate this setup via tools like [AWS CloudFormation](https://aws.amazon.com/cloudformation/) and the [AWS Cloud Development Kit](https://aws.amazon.com/cdk/). You could also perform it manually in the [Amazon S3 Console](https://s3.console.aws.amazon.com/s3/home) and [AWS IAM Console](https://console.aws.amazon.com/iam/home).\n",
    "\n",
    "Since we're just experimenting though, we'll instead use this notebook to keep the setup easily customizable for your environment:\n",
    "\n",
    "![Setup Architecture Overview Diagram](static/imgs/setup-architecture.png \"Setup Architecture Overview Diagram\")\n",
    "\n",
    "> ⚠️ This assumes you're running the notebook itself with sufficient administrative permissions (as defined by its *Execution Role*) to set up IAM and S3 resources\n",
    "\n",
    "To get started, we'll first import the appropriate Python libraries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Python Built-Ins:\n",
    "import json\n",
    "import secrets\n",
    "import string\n",
    "from time import sleep\n",
    "\n",
    "# External Dependencies:\n",
    "import boto3  # (AWS Python SDK)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Connecting to an AWS Region\n",
    "\n",
    "Assuming you're running this notebook on [Amazon SageMaker](https://aws.amazon.com/sagemaker/), it will already be associated with a particular [AWS Region](https://aws.amazon.com/about-aws/global-infrastructure/) and be running with certain [AWS IAM Permissions](https://aws.amazon.com/iam/) (defined by the **notebook execution role**).\n",
    "\n",
    "If you're running the notebook locally, you may need to explicitly log in e.g. using `aws configure` from the [AWS CLI](https://aws.amazon.com/cli/), and set the specific region you'd like to work with."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "session = boto3.Session(region_name=None)  # To set a specific region, replace None with e.g. \"us-east-1\"\n",
    "region = session.region_name  # We'll save the configured region to initialize later notebooks\n",
    "print(region)\n",
    "%store region\n",
    "\n",
    "iam = session.client(\"iam\")\n",
    "s3 = session.resource(\"s3\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## S3 Bucket(s)\n",
    "\n",
    "Amazon Forecast will read historical data from S3, and may export forecasts to S3.\n",
    "\n",
    "By default, we'll create a single bucket for both with a partially-randomized name (since S3 bucket names must be globally unique).\n",
    "\n",
    "You can customize this setup (e.g. to use an existing bucket instead) and/or configure through the [Amazon S3 Console](https://s3.console.aws.amazon.com/s3/home).\n",
    "\n",
    "Just be sure to `%store` a valid `bucket_name` and `export_bucket_name` which exist in the same `region`: We'll use this below and in later notebooks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Choose a source data bucket name:\n",
    "bucket_name = \"{}-forecastpoc-{}\".format(\n",
    "    # AWS Account ID:\n",
    "    session.client(\"sts\").get_caller_identity().get(\"Account\"),\n",
    "    # Random string:\n",
    "    ''.join(secrets.choice(string.ascii_lowercase + string.digits) for i in range(8))\n",
    ")\n",
    "print(bucket_name)\n",
    "%store bucket_name\n",
    "\n",
    "# Create the bucket (assuming it's new):\n",
    "if region != \"us-east-1\":\n",
    "    s3.create_bucket(Bucket=bucket_name, CreateBucketConfiguration={ \"LocationConstraint\": region })\n",
    "else:\n",
    "    s3.create_bucket(Bucket=bucket_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We'll assume forecast exports can go in the same bucket:\n",
    "\n",
    "export_bucket_name = bucket_name\n",
    "%store export_bucket_name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## IAM Role for Forecast\n",
    "\n",
    "To access data in these buckets, Amazon Forecast needs permissions. This means creating a **role** with appropriate access the buckets and which can be assumed by the Forecast service.\n",
    "\n",
    "By default, we'll create a new role and attach necessary permissions here.\n",
    "\n",
    "You can customize this setup and/or configure through the [AWS IAM Console](https://console.aws.amazon.com/iam/home).\n",
    "\n",
    "Just be sure to `%store` a valid `forecast_role_arn`: We'll use this in later notebooks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "forecast_role_name = \"ForecastRolePOC\"\n",
    "\n",
    "try:\n",
    "    create_role_response = iam.create_role(\n",
    "        RoleName=forecast_role_name,\n",
    "        AssumeRolePolicyDocument=json.dumps({\n",
    "            \"Version\": \"2012-10-17\",\n",
    "            \"Statement\": [\n",
    "                {\n",
    "                    \"Effect\": \"Allow\",\n",
    "                    \"Principal\": {\n",
    "                        \"Service\": \"forecast.amazonaws.com\",\n",
    "                    },\n",
    "                    \"Action\": \"sts:AssumeRole\",\n",
    "                },\n",
    "            ]\n",
    "        }),\n",
    "    )\n",
    "    forecast_role_arn = create_role_response[\"Role\"][\"Arn\"]\n",
    "\n",
    "except iam.exceptions.EntityAlreadyExistsException:\n",
    "    print(f\"Using already-existing role {forecast_role_name}\")\n",
    "    forecast_role_arn = iam.get_role(RoleName=forecast_role_name)[\"Role\"][\"Arn\"]\n",
    "\n",
    "print(forecast_role_arn)\n",
    "%store forecast_role_arn\n",
    "\n",
    "# Note that AmazonForecastFullAccess provides access to some specifically-named default S3 buckets\n",
    "# as well, but we just want it for the Forecast permissions themselves:\n",
    "iam.attach_role_policy(\n",
    "    RoleName=forecast_role_name,\n",
    "    PolicyArn=\"arn:aws:iam::aws:policy/AmazonForecastFullAccess\",\n",
    ")\n",
    "\n",
    "# By default (since we're experimenting), this code attaches over-generous S3 permissions:\n",
    "iam.attach_role_policy(\n",
    "    RoleName=forecast_role_name,\n",
    "    PolicyArn=\"arn:aws:iam::aws:policy/AmazonS3FullAccess\",\n",
    ")\n",
    "# You could instead use something like the below to give access to *only* the relevant buckets:\n",
    "# inline_s3_policy = {\n",
    "#     \"Version\": \"2012-10-17\",\n",
    "#     \"Statement\": [\n",
    "#         {\n",
    "#             \"Effect\": \"Allow\",\n",
    "#             \"Action\": \"s3:*\",\n",
    "#             \"Resource\": [\n",
    "#                 # (Assuming you're not running in a different partition e.g. aws-cn)\n",
    "#                 f\"arn:aws:s3:::{bucket_name}\",\n",
    "#                 f\"arn:aws:s3:::{bucket_name}/*\",\n",
    "#             ]\n",
    "#         },\n",
    "#     ],\n",
    "# }\n",
    "# if bucket_name != export_bucket_name:\n",
    "#     inline_s3_policy[\"Statement\"][0][\"Resource\"].append(f\"arn:aws:s3:::{export_bucket_name}\")\n",
    "#     inline_s3_policy[\"Statement\"][0][\"Resource\"].append(f\"arn:aws:s3:::{export_bucket_name}/*\")\n",
    "\n",
    "# iam.put_role_policy(\n",
    "#     RoleName=role_name,\n",
    "#     PolicyName=\"ForecastPoCBucketAccess\",\n",
    "#     PolicyDocument=json.dumps(inline_s3_policy)\n",
    "# )\n",
    "\n",
    "# IAM policy attachments *may* take up to a minute to propagate, so just to be safe:\n",
    "sleep(60) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## All done!\n",
    "\n",
    "We now have a bucket to store our data in, and an Amazon Forecast-ready IAM role which is allowed to read & write to it."
   ]
  }
 ],
 "metadata": {
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "Python 3 (Data Science)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:ap-southeast-1:492261229750:image/datascience-1.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}