Installation Guide

Introduction

This topic provides technical explanations around the Alert Manager app.

Prerequisites

• All indexers and search heads require Splunk Enterprise version 6.2 or later.

How does it work

The Alert Manager works as a generic scripted alert action in Splunk, callert alert_handler.py. When an alert fires, the alert_handler.py script catches all the parameters and job details from the alert search and writes aggregated data to an index and a state to a collection in Splunk's App Key Value Store.

Data generated by the Alert Manager

There are differents events written by the alert manager. All of them are stored in a configurable index.

• One metadata event per new incident with sourcetype alert_metadata (up to 5 kilobytes per incident)
• One event per each change on an incident (change owner, priority or status) with sourcetype incident_change (less than 1 kilobyte per change)

The total amout of data generated by the Alert Manager depends on number of fired alerts. All data is written to Splunk through the API and counts towards the license.

Indexes used by the Alert Manager

By default, the Alert Manager creates and uses an index named alerts. The index used by the App can be changed during the installation process.

Techniques used by the App

As described above, the App writes different data and uses the Splunk framework components to provide the end-user functionality.

• App Key Value Store
  • Used to save and track states of incidents
  • Used to store incident default and user settings
• Splunk REST API
  • Write events to the index
  • Add/retrieve data from the App Key Value Store
  • Manage app configuration
• Splunk JS Stack
  • Extend Splunk dashboards with workflow functionalities
  • Extend and enhance Splunk visualization components (Single value, Tables, ...)
  • Provide 3rd party visualizations

Deploy and configure

In this chapter, we provide information how to install the Alert Manager app. Have a look at the Configuration Guide to learn how to get data into the Alert Manager.

Before you install

1. Decide if you want to use the Alert Managers default index alerts or not
2. Check if your Splunk installation fits the prerequisites

Deployment Matrix

	Alert Manager	Technology Add-on for Alert Manager	Supporting Add-on for Alert Manager Demo Data
Search Head	x	x	x
Indexer		x

Install the Technology Add-on for Alert Manager

As a first step is to download and install the Alert manager add-on. The Add-on provides configuration for:

• Index
• Event breaking and timestamp recognition configuration for Alert Manager events
• Field extractions

1. Download the latest add-on
2. Unpack and upload the add-on according to the Deployment Matrix
3. IMPORTANT: Make sure, the app folder name in $SPLUNK_HOME/etc/apps is TA-alert_manager (Downloading apps from git and uploading them to Splunk will result in wrong folder names)
4. Configure the Add-on (see the chapter below)
5. Restart Splunk

Configure the Add-on

If you decided to not use the default index alerts, create a copy of $SPLUNK_HOME/etc/apps/TA-alert_manager/default/indexes.conf to $SPLUNK_HOME/etc/apps/TA-alert_manager/local/indexes.conf and disable the index:

[alerts]

homePath = $SPLUNK_DB/alerts/db

coldPath = $SPLUNK_DB/alerts/colddb

thawedPath = $SPLUNK_DB/alerts/thaweddb

disabled = true

Install the Alert Manager App

The Alert Manager App contains the core functionalities and configurations.

1. Download the latest app
2. Unpack and upload the app according to the Deployment Matrix
3. IMPORTANT: Make sure, the app folder name in $SPLUNK_HOME/etc/apps is alert_manager (Downloading apps from git and uploading them to Splunk will result in wrong folder names)
4. Configure the App (see the chapter below)
5. Restart Splunk

Configure App settings

There are two ways to configure the basic app settings:

1. Through the App settings page
2. With alert_manager.conf

We recommend to use the App settings page, as there will be a configuration validation. To use the App settings page, restart Splunk after you've installed the App and open the app. If you visit the App the first time, the App settings page will open automatically.

Notes:

• Change the index according to your decision whether to use the default one (named alerts) or your custom index. Either change it in the Alert Manager's setup page or in alert_manager.conf
• Have a look at $SPLUNK_HOME/etc/apps/alert_manager/README/alert_manager.conf.spec for a full configuration reference
• Set is_configured to the value "1" (without quotes) in $SPLUNK_HOME/etc/apps/alert_manager/local/app.conf inside the "[install]" stanza to hide the App setup page in case you configured the App with the config file

Link the alert_handler.py script

Note: This step is required to get the alert manager running.

Since the Alert Manager works as a scripted alert action and Splunk only accepts global alert scripts in $SPLUNK_HOME/bin/scripts, we decided to link our alert script from the app the the scripts folder. With the symlink we can ensure that app updates will also apply for the alert handler script. If you have any ideas how to improve this step, let us know!

For Linux systems:

1. Open a command line
2. Switch to $SPLUNK_HOME/bin/scripts
3. Create a symlink:

ln -s ../../etc/apps/alert_manager/bin/alert_handler.py alert_handler.py

For Windows systems:

1. Open a command prompt with administrative privileges (hit "Start", type "cmd", Right-click "cmd.exe", click "Run as administrator"
2. Switch to $SPLUNK_HOME/bin/scripts
3. Create a symlink:

mklink alert_handler.py ..\..\etc\apps\alert_manager\bin\alert_handler.py

After finishing this step, you're ready to configure Splunk alerts to appear in the Alert Manager. Please proceed to the Configuration Guide for further instructions.