2a. Getting Started with Forecast (Console).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Getting Started with Amazon Forecast (AWS Console)\n",
    "\n",
    "> *This notebook should work well in the `Python 3 (Data Science)` kernel in SageMaker Studio, or `conda_python3` in SageMaker Notebook Instances*\n",
    "\n",
    "Now the only mandatory dataset (the TTS) is prepared in compatible CSV format and uploaded to Amazon S3, we're ready to start our experiments with Amazon Forecast.\n",
    "\n",
    "**In this notebook** we'll use the [**Amazon Forecast Console**](https://console.aws.amazon.com/forecast/home) to:\n",
    "\n",
    "- Create the \"Dataset Group\" wrapper in Forecast to store our project\n",
    "- Define our TTS dataset schema and import the prepared data\n",
    "- Create some **predictors** - training models on our data\n",
    "- Evaluate metrics on how good our predictors seem from the training process\n",
    "- Create and export some **forecasts**\n",
    "\n",
    "Check out [Notebook 2b](2b.%20Getting%20Started%20with%20Forecast%20(Python%20SDK).ipynb) for an alternative guide through the same steps from code using the **AWS Python SDK** instead!\n",
    "\n",
    "To get started, navigate to the [Amazon Forecast Console](https://console.aws.amazon.com/forecast/home). You might see a landing page something like the below, or be taken straight to the *Dataset groups list* page if you've used the service before. You can navigate around the Forecast console using the expandable white sidebar menu on the left.\n",
    "\n",
    "![\"Amazon Forecast Landing Page Screenshot\"](static/imgs/guide/gs-00-forecast-console.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setting up the Dataset Group\n",
    "\n",
    "A **Dataset Group** is the highest level of abstraction in Amazon Forecast, and contains all the source datasets for a particular collection of Forecasts. A Dataset Group contains **up to one of each type** of dataset (Target Time-Series, Related Time-Series, and Item Metadata) and no information is shared between Dataset Groups - so if you'd like to try out various alternatives to the schemas we create below, you could create a new Dataset Group and make your changes inside its corresponding Datasets.\n",
    "\n",
    "▶️ **Click** the *\"Create dataset group\"* button on the landing page or dataset group list to start the process\n",
    "\n",
    "In the first screen, you'll be prompted for:\n",
    "\n",
    "- **A name** - you can use `forecast_poc`\n",
    "- **Our chosen [domain](https://docs.aws.amazon.com/forecast/latest/dg/howitworks-domains-ds-types.html)** - here we'll use `CUSTOM`, because the traffic forecasting use case doesn't have clear mapping to other predefined domains.\n",
    "\n",
    "When that's entered, as below, click *Next*\n",
    "\n",
    "![\"Create Dataset Group Screenshot\"](static/imgs/guide/gs-01-create-dsg.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Defining the TTS Dataset\n",
    "\n",
    "Next we'll define the **structure** of our TTS dataset, including:\n",
    "\n",
    "- A **name** - we'll use `forecast_poc_tts`\n",
    "- The [**frequency**](https://docs.aws.amazon.com/forecast/latest/dg/howitworks-datasets-groups.html#howitworks-data-alignment) (e.g. hourly, daily, etc), which will determine what frequencies of forecasts we can build from our data (e.g. can't build hourly forecasts from weekly source data!) - our data is `hourly`\n",
    "- The data **schema** including both the **mandatory** fields for our chosen [domain](https://docs.aws.amazon.com/forecast/latest/dg/howitworks-domains-ds-types.html) and any **optional** fields we've chosen to add.\n",
    "\n",
    "> ⚠️ The schema must **match the prepared data exactly**, *including the order of columns*, because Forecast will validate the data against it when importing!\n",
    "\n",
    "> ⚠️ **YOU MUST AMEND THE SCHEMA JSON** in the UI, because the default provided by the console has a different column order!\n",
    "\n",
    "![\"Create TTS Dataset Screenshot\"](static/imgs/guide/gs-02-create-tts-ds.png)\n",
    "\n",
    "As discussed in the ['Resolving Conflicts in Data Collection Frequency' doc](https://docs.aws.amazon.com/forecast/latest/dg/howitworks-datasets-groups.html#howitworks-data-alignment), the raw CSV data will be automatically mapped and aggregated to the chosen frequency time-steps if it doesn't match already... Just check the right aggregation is being applied in later steps, in case you have any mismatches!\n",
    "\n",
    "![Illustration of Time Step Binning](https://docs.aws.amazon.com/forecast/latest/dg/images/data-alignment.png)\n",
    "\n",
    "When you've entered the details and updated the schema, click *Next*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Importing the TTS Data\n",
    "\n",
    "We're now ready to populate our Amazon Forecast dataset with the contents of our CSV file from S3.\n",
    "\n",
    "Since this requires the Amazon Forecast service to access the Amazon S3 bucket, this is where we need the **service role** created in Notebook 0: Which has access to the target bucket and trusts the Forecast service.\n",
    "\n",
    "You can run the below cell to read your stored role ARN, or if you don't have such a role set up in your account yet, you'll be able to **create a new role** with permissions to access the required bucket, through the UI:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%store -r forecast_role_arn\n",
    "print(forecast_role_arn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we'll also need the **S3 URI** of the data file we uploaded earlier, which was printed out near the end of notebook 1 but you can view again by running the below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%store -r target_s3uri\n",
    "print(target_s3uri)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that every **dataset import job** we create in Forecast is a batch **overwriting** process that clears out any pre-existing data in the dataset: *not* appending data to existing records.\n",
    "\n",
    "To configure your initial import, enter:\n",
    "\n",
    "- A **Name** for the job - we'll use `poc_import_tts`\n",
    "- The **Timestamp Format** in the data, since some variations are permitted according to the [dataset guidelines](https://docs.aws.amazon.com/forecast/latest/dg/dataset-import-guidelines-troubleshooting.html) - our data uses the default\n",
    "- The **IAM Role** we created earlier\n",
    "- The **S3 URI** of the data we uploaded\n",
    "\n",
    "Once you've entered and checked the values, click *Start import*\n",
    "\n",
    "![\"Import TTS Data Screenshot\"](static/imgs/guide/gs-03-import-tts.png)\n",
    "\n",
    "You should be taken to the *dashboard page for your dataset group*, showing that the import is now pending. The DSG dashboard page gives a visual overview of which datasets have been populated in your dataset group, and how far you've got through the process of importing data, training models, and creating forecasts.\n",
    "\n",
    "![\"DSG Dashboard Screenshot showing TTS Import Pending\"](static/imgs/guide/gs-04-dsgdash-importing-tts.png)\n",
    "\n",
    "You can also navigate to the \"*Datasets*\" tab to view more details and drill in to individual datasets and import jobs:\n",
    "\n",
    "![\"Datasets view showing TTS Import Pending\"](static/imgs/guide/gs-05-datasets-importing-tts.png)\n",
    "\n",
    "▶️ We'll need to **wait** for this import to complete to move on to the next step\n",
    "\n",
    "> ⏰ The import process can **take a little time** (on the order of ~10-15 minutes for our sample dataset) because of validation, filling & aggregation, and the overhead of spinning up infrastructure to execute the import\n",
    "\n",
    "On small datasets like this, overheads can dominate the run-time and you should expect much better-than-linear scaling as dataset size is increased from this level.\n",
    "\n",
    "▶️ **Note** that, once a dataset import is complete, we can view per-field statistics in the console to check whether any missing values were found, or other unexpected problems with our data:\n",
    "\n",
    "![\"TTS import detail screen with item_id field metrics\"](static/imgs/guide/gs-05.1-tts-import-stats.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating and Training Predictors\n",
    "\n",
    "Once your target timeseries shows as **Active**, the button to *Start* predictor training will become enabled on the dataset group dashboard: because additional dataset types (related time-series and item metadata) are optional.\n",
    "\n",
    "▶️ Click **Start** next to Predictor training on the dashboard page for your dataset group\n",
    "\n",
    "We can create multiple predictors within our Dataset Group, and that's exactly what we'll do in this notebook to compare the results of a few different algorithms offered by the service.\n",
    "\n",
    "Amazon Forecast offers 6 (at the time of writing) algorithms as described in more detail on the [\"Choosing an Algorithm\" doc page](https://docs.aws.amazon.com/forecast/latest/dg/aws-forecast-choosing-recipes.html).\n",
    "\n",
    "Although an **AutoML** option is available which will automatically try each algorithm, list the metrics of each, and keep the best model; we'll create a set of models **manually** in this example so that we're able to generate a forecast output for each, to plot and compare."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ARIMA\n",
    "\n",
    "We'll start with one of the simpler baseline algorithms, **ARIMA**.\n",
    "\n",
    "▶️ In the \"*Train Predictor*\" screen, **enter**:\n",
    "\n",
    "- The **Predictor name** as `forecast_poc_arima_algo_1`\n",
    "- The **Forecast horizon** and **Forecast frequency** as `240` and `hour` (a 10-day, hourly forecast).\n",
    "    - Note that the forecast frequency cannot be less than the dataset frequencies, and the forecast horizon must not be more than 500 or 1/3 of the dataset length per the [quotas page](https://docs.aws.amazon.com/forecast/latest/dg/limits.html)\n",
    "- **Manual algorithm** selection and **ARIMA** for this first model\n",
    "- Set the **Country for holidays** to `United States`, since this data is from America and it's a country the Forecast service supports\n",
    "- Leave any other settings as default\n",
    "\n",
    "▶️ **Review** the \"*Featurizations*\" option in the *Advanced configurations* section.\n",
    "\n",
    "This is important configurations including [missing value handling](https://docs.aws.amazon.com/forecast/latest/dg/howitworks-missing-values.html) and [aggregation methods](https://docs.aws.amazon.com/forecast/latest/dg/howitworks-datasets-groups.html#howitworks-data-alignment) for standardizing data frequency are configured. We can use the defaults for this example, but it's good to be aware of where this is set!\n",
    "\n",
    "![Create Predictor screenshot for ARIMA config, part 1](static/imgs/guide/gs-06.1-create-arima-pt1.png)\n",
    "\n",
    "![Create Predictor screenshot for ARIMA config, part 2](static/imgs/guide/gs-06.2-create-arima-pt2.png)\n",
    "\n",
    "▶️ Once you've checked everything, click **Train predictor** to kick off the model training.\n",
    "\n",
    "You should be returned to the dataset group dashboard page, as below, showing your new predictor as \"Create Pending\":\n",
    "\n",
    "![DSG Dashboard Screenshot showing Pending Predictor](static/imgs/guide/gs-07-dsgdash-creating-arima.png)\n",
    "\n",
    "Note that the **Predictors tab** will now be enabled in the sidebar, and you can open this to review the status and details of any predictors in your DSG:\n",
    "\n",
    "![Predictors List Screenshot with only ARIMA (Pending)](static/imgs/guide/gs-08-predictors-list-arima.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prophet, DeepAR+ (and maybe CNN-QR)\n",
    "\n",
    "▶️ **Navigate** to the \"Predictors\" list for your dataset group using the sidebar\n",
    "\n",
    "▶️ **Click** the *Train new predictor* button and **repeat the process** from the ARIMA predictor to start training two more predictors:\n",
    "\n",
    "- `forecast_poc_prophet_algo_1`, with algorithm **Prophet** and all other settings as per the ARIMA predictor\n",
    "- `forecast_poc_deeparp_algo_1`, with algorithm **DeepAR+** and all other settings as per the ARIMA predictor\n",
    "\n",
    "Your predictors list should now look something like this, with all 3 models creating:\n",
    "\n",
    "![Predictors list screenshot showing 3 models creating](static/imgs/guide/gs-09-predictors-list-training.png)\n",
    "\n",
    "We've selected this set because in our tests the CNN-QR algorithm takes longer to train on the small sample dataset than DeepAR+ - with comparable accuracy. On many larger \"real\" datasets CNN-QR can be much faster though, so we'd recommend experimenting with it on your own data!\n",
    "\n",
    "▶️ You'll need to **wait** for at least the first predictor to finish training (enter `Active` status) before continuing\n",
    "\n",
    "> ⏰ Predictor training can **take some time**: Simpler algorithms like ARIMA or ETS will typically train faster (may be ready in ~20mins on this example dataset), whereas more complex algorithms like DeepAR+ will usually take longer (may be approx 1hr on this example dataset)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Examining the Predictors\n",
    "\n",
    "Once each of the Predictors is in an `Active` state you can get metrics about it to better understand its accuracy and behavior. These are computed based on the hold out periods we defined when building the Predictor. The metrics are meant to guide our decisions when we use a particular Predictor to generate a forecast.\n",
    "\n",
    "Summary *weighted quantile loss* (wQL) metrics are published on the predictor list itself:\n",
    "\n",
    "![Predictors list screenshot showing wQL0.1, wQL0.5 and wQL0.9 metrics](static/imgs/guide/gs-10-predictors-list-trained.png)\n",
    "\n",
    "...while additional metrics, and separate figures for each *backtesting window* (if multiple were configured) are available on each predictor details page - accessed by clicking the predictor name hyperlink:\n",
    "\n",
    "![ARIMA predictor detail page with additional metrics](static/imgs/guide/gs-11-arima-predictor-detail.png)\n",
    "\n",
    "A key point to remember when evaluating Amazon Forecast models is that they produce **probabilistic** forecasts, as we'll see later... So while some metrics (like [MAPE](https://en.wikipedia.org/wiki/Mean_absolute_percentage_error) or [RMSE](https://en.wikipedia.org/wiki/Root-mean-square_deviation)) might already be familiar to you from single-forecast generating approaches, they don't tell us the whole story about our predictor accuracy.\n",
    "\n",
    "A detailed discussion of the calculation and interpretation of predictor metrics is provided on the [\"Evaluating Predictor Accuracy\"](https://docs.aws.amazon.com/forecast/latest/dg/metrics.html) doc page. Both wQL/MAPE and RMSE are **loss/error** metrics, so lower values are better.\n",
    "\n",
    "Your results may vary a little vs these example trained predictors.\n",
    "\n",
    "In our test:\n",
    "\n",
    "- **ARIMA** (our baseline algorithm) scored a 50% quantile weighted loss (=MAPE) of 0.4708 and an RMSE of ~1950.\n",
    "- **Prophet** (a more modern statistical approach) achieved a slightly better RMSE of ~1900, but the wQL0.5/MAPE score was slightly worse than ARIMA, at 0.4847\n",
    "- **DeepAR+** (an advanced deep learning model) delivered significant improvements in both RMSE (~1490) and MAPE (0.3199)\n",
    "\n",
    "These 'central' metrics suggest that the *middle* of the forecast (the 50% probability quantile) is likely to match the actual data most closely in the DeepAR+ model; and perform quite similarly between ARIMA & Prophet... Although ARIMA might see more *large* deviations since the RMSE was higher, while the MAPE was lower.\n",
    "\n",
    "The wQL0.1 and wQL0.9 metrics describe performance of the 10% and 90% quantiles of the forecast: in other words, the 80% *confidence interval* around our central most-likely forecast. They're similar to MAPE figures, but penalize errors on the \"wrong\" side of the quantile much more than the distance to the actual data when it's on the \"right\" side... So these figures will be lowest when the confidence interval is narrow and the true data falls within it - but pushed up significantly by any points where the actual data falls outside the confidence interval.\n",
    "\n",
    "In our example results you might notice that while the DeepAR+ model achieves the lowest (best) scores on most metrics, its wQL0.1 score of 0.2045 is higher (worse) than both ARIMA at 0.1706 and Prophet at 0.1815. This suggests that the *lower bound* forecast of the DeepAR+ model may be less tight & correct than the other two algorithms.\n",
    "\n",
    "While these metrics are useful for hyperparameter tuning, AutoML, and initial model evaluation, we'd also like to actually generate and view some forecasts!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating Forecasts\n",
    "\n",
    "Inside Amazon Forecast a Forecast is a rendered collection of all of your items, at every time interval, for all selected quantiles, for your given forecast horizon. This process takes the Predictor you just created and uses it to generate these inferences and to store them in a useful state. Once a Forecast exists within the service you can query it and obtain a JSON response or use another API call to export it to a CSV that is stored in S3.\n",
    "\n",
    "▶️ To start creating a forecast, either **click** the *Create new forecast* button from your ARIMA predictor's detail page, or select the ARIMA predictor from your predictor list screen as shown and click *Create new forecast* there\n",
    "\n",
    "![Screenshot of predictors list with ARIMA model selected](static/imgs/guide/gs-12-predictors-list-select-arima.png)\n",
    "\n",
    "Next:\n",
    "\n",
    "- **Name** your forecast as e.g. `forecast_poc_arima_algo_forecast`\n",
    "- Use the ARIMA predictor, which should already be selected\n",
    "- Leave the **Forecast types** option as default, which will generate the 0.1, 0.5 and 0.9 quantiles as used in the predictor metrics\n",
    "\n",
    "When you're ready, go ahead and click *Create new forecast* to kick off the process.\n",
    "\n",
    "![Screenshot of create forecast view](static/imgs/guide/gs-13-create-arima-forecast.png)\n",
    "\n",
    "You'll be taken back to the dashboard screen, now showing that forecast generation is in progress:\n",
    "\n",
    "![Screenshot of DSG dashboard with forecast in progress](static/imgs/guide/gs-14-dsgdash-creating-forecast.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "▶️ Next **Repeat** this process for your other two predictors, Prophet and DeepAR+ - by navigating back to the *Predictors* list in the sidebar.\n",
    "\n",
    "You can name your additional forecasts `forecast_poc_prophet_algo_forecast` and `forecast_poc_deeparp_algo_forecast`.\n",
    "\n",
    "Note that the *Forecasts* sidebar tab is now enabled, and yours should have 3 forecasts pending like in the screenshot below:\n",
    "\n",
    "![Screenshot of Forecasts tab with 3 forecasts pending](static/imgs/guide/gs-15-forecast-list-creating.png)\n",
    "\n",
    "▶️ You'll need to **wait** for at least the first forecast to reach `Active` status before continuing\n",
    "\n",
    "> ⏰ Forecast creation can **take some time**: Simpler algorithms like ARIMA or ETS will typically build faster (may be ready in ~20mins on this example dataset), whereas more complex algorithms like DeepAR+ will usually take longer (may be approx 1hr on this example dataset).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Viewing Forecasts in the Console\n",
    "\n",
    "As soon as at least one forecast is finished creating, the *Forecast lookup* tab will become enabled in the console.\n",
    "\n",
    "The Forecast lookup page is a basic UI to help you explore your forecasts, which can also be accessed through the [QueryForecast API](https://docs.aws.amazon.com/forecast/latest/dg/API_forecastquery_QueryForecast.html) as soon as their creation is finished.\n",
    "\n",
    "For this example, you can select:\n",
    "\n",
    "- **Start date** of `2017/12/22 00:00:00`, to include some historical actuals into the plot\n",
    "- **End date** of `2018/01/10 23:00:00`, the latest available date within our Forecast Horizon\n",
    "- **Item ID** of `all` - the only item_id present in our prepared sample data\n",
    "- Whichever **Forecast** you'd like to query\n",
    "\n",
    "In our test, we saw a result like the below for the **ARIMA** model:\n",
    "\n",
    "![ARIMA forecast in Amazon Forecast console](static/imgs/guide/gs-16-arima-forecast-query.png)\n",
    "\n",
    "...The below view for **Prophet**:\n",
    "\n",
    "![Prophet forecast in Amazon Forecast console](static/imgs/guide/gs-17-prophet-forecast-query.png)\n",
    "\n",
    "...and the below for **DeepAR+**:\n",
    "\n",
    "![DeepAR+ forecast in Amazon Forecast console](static/imgs/guide/gs-18-deeparp-forecast-query.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exporting Forecasts\n",
    "\n",
    "Although forecasts may be either queried directly through the API (or the *Forecast lookup* tab of the [Amazon Forecast Console](https://console.aws.amazon.com/forecast/home)), **exporting** the forecast to S3 bucket is also possible and we can use the resulting CSV files to analyze the forecast in other ways\n",
    "\n",
    "Once the forecasts have entered `Active` status, they are ready to be exported.\n",
    "\n",
    "▶️ To start exporting a forecast, either **click** the *Create forecast export* button from your ARIMA forecast's detail page, or select the ARIMA forecast from your forecasts list screen as shown and click *Create forecast export* there\n",
    "\n",
    "![Forecasts list screenshot with 'Active' ARIMA forecast selected](static/imgs/guide/gs-19-forecast-list-active.png)\n",
    "\n",
    "You'll need to know what **S3 bucket name** to export the forecasts to. Run the cell below to print this out if you ran through notebook 0, or else check what bucket you have available to store your forecasts which your **Forecast execution role** has permission to access."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%store -r export_bucket_name\n",
    "print(export_bucket_name)\n",
    "\n",
    "print(forecast_role_arn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the *Create forecast export* screen:\n",
    "\n",
    "- Set **Export name** to `forecast_poc_arima_export`\n",
    "- Check your **Generated forecast** has the *ARIMA* forecast selected\n",
    "- Check your **IAM Role** is set as expected - the role must have permission to write to the S3 output location\n",
    "- Set **Export location** like `s3://{export_bucket_name}/exports/arima/tts-only`\n",
    "\n",
    "![Create forecast export screen filled out for ARIMA](static/imgs/guide/gs-20-create-forecast-export.png)\n",
    "\n",
    "When ready, go ahead and click **Create forecast export** again to start the export.\n",
    "\n",
    "You should be taken to the detail page for your chosen forecast, showing the new pending export job:\n",
    "\n",
    "![Screenshot of ARIMA forecast exporting](static/imgs/guide/gs-21-arima-forecast-exporting.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "▶️ **Repeat** these steps for your Prophet and DeepAR+ forecasts, creating:\n",
    "\n",
    "- `forecast_poc_prophet_export` export to `s3://{export_bucket_name}/exports/prophet/tts-only`\n",
    "- `forecast_poc_deeparp_export` export to `s3://{export_bucket_name}/exports/deeparp/tts-only`\n",
    "\n",
    "▶️ You'll need to **wait** for the forecast exports to complete, before you can use the files for validation\n",
    "\n",
    "> ⏰ The export process can **take a little time** (on the order of ~10-15 minutes for our sample dataset)\n",
    "\n",
    "There's no consolidated dashboard for exports at the time of writing, but you can check the status from each *Forecast detail page*. The export is complete when the job shows \"Active\" status, as shown below for ARIMA:\n",
    "\n",
    "![ARIMA forecast detail screen showing export in 'Active' state](static/imgs/guide/gs-22-arima-forecast-done.png)\n",
    "\n",
    "Once your export is complete, result files will be available in your selected S3 location as shown below in the [Amazon S3 console](https://s3.console.aws.amazon.com/s3/home) (also accessible via searching in the \"Services\" dropdown):\n",
    "\n",
    "![Amazon S3 screenshot showing result files](static/imgs/guide/gs-23-arima-forecast-s3.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Validation\n",
    "\n",
    "To work through an example plotting the forecast over the *actual* data held back in our validation dataset, you can pick up the ['Validation' section of Notebook 2b (Python SDK)](2b.%20Getting%20Started%20with%20Forecast%20(Python%20SDK).ipynb#Validation).\n",
    "\n",
    "Regardless of whether you query forecasts in real-time using the [QueryForecast API](https://docs.aws.amazon.com/forecast/latest/dg/API_forecastquery_QueryForecast.html), or export forecasts to CSV as we have here, there are many possible tools available for you to visualize your forecast and reconcile against actual data when it's available."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Recap and Next Steps\n",
    "\n",
    "We've now explored some initial models on the target timeseries alone, and can start exploring additional **related data** as a way to improve forecast accuracy. The [next notebook, #3](3.%20Preparing%20Related%20Time-Series%20Data.ipynb) will guide you through the process of preparing a *related time-series* file ready to upload to Amazon Forecast."
   ]
  }
 ],
 "metadata": {
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "Python 3 (Data Science)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:ap-southeast-1:492261229750:image/datascience-1.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}