Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add the ability to sync AlertManager configuration with Cortex #10

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 29 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@ This action is configured using environment variables defined in the workflow. T
| `CORTEX_API_KEY` | Optional password that is required for password protected Cortex clusters. An encrypted [github secret](https://help.github.com/en/actions/automating-your-workflow-with-github-actions/creating-and-using-encrypted-secrets ) is recommended. | `false` | N/A |
| `ACTION` | Which action to take. One of `lint`, `prepare`, `check`, `diff` or `sync` | `true` | N/A |
| `RULES_DIR` | Comma-separated list of directories to walk in order to source rules files | `false` | `./` |
| `COMPONENT` | `RULER` or `ALERTMANAGER`. This dictates whether to sync ruler rulegroups or AlertManager configurations | `true` | N/A |
| `ALERTMANAGER_CONFIG_PATH` | Path to the tenants AlertManager configuration | `true` If `COMPONENT` is set to `ALERTMANAGER` | `./alertManager.yml`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's missing a | at the end, in the diff, it looks like one column instead of two - am I missing something?


## Actions

Expand Down Expand Up @@ -74,7 +76,7 @@ jobs:
CORTEX_TENANT_ID: 1
CORTEX_API_KEY: ${{ secrets.CORTEX_API_KEY }} # Encrypted Github Secret https://help.github.com/en/actions/automating-your-workflow-with-github-actions/creating-and-using-encrypted-secrets
ACTION: diff
RULES_DIR: "./rules/" # In this example rules are stored in a rules directory in the repo
RULES_DIR: "./rules/" # In this example rules are stored in a rules directory in the repo
- name: comment PR
uses: unsplash/[email protected] # https://github.com/unsplash/comment-on-pr
env:
Expand Down Expand Up @@ -108,5 +110,30 @@ jobs:
CORTEX_TENANT_ID: 1
CORTEX_API_KEY: ${{ secrets.CORTEX_API_KEY }} # Encrypted Github Secret https://help.github.com/en/actions/automating-your-workflow-with-github-actions/creating-and-using-encrypted-secrets
ACTION: sync
RULES_DIR: "./rules/" # In this example rules are stored in a rules directory in the repo
RULES_DIR: "./rules/" # In this example rules are stored in a rules directory in the repo
```

The following workflow will sync the AlertManager configuration file in the `master` branch against the configured Cortex cluster
```yaml
name: sync_alertmanager_master
on:
push:
branches:
- master
jobs:
sync-master:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v2
with:
ref: master
- name: sync-alertmanager
uses: grafana//[email protected]
env:
CORTEX_ADDRESS: https://example-cluster.com/
CORTEX_TENANT_ID: 1
CORTEX_API_KEY: ${{ secrets.CORTEX_API_KEY }} # Encrypted Github Secret https://help.github.com/en/actions/automating-your-workflow-with-github-actions/creating-and-using-encrypted-secrets
ACTION: sync
COMPONENT: ALERTMANAGER
ALERTMANAGER_CONFIG_PATH: "./alertmanager/config.yml" # In this example the AlertManager config is stored in the `alertmanager/` directory in the repository```
78 changes: 48 additions & 30 deletions entrypoint.sh
Original file line number Diff line number Diff line change
Expand Up @@ -27,48 +27,66 @@ SYNC_CMD=sync
DIFF_CMD=diff
PRINT_CMD=print

if [ -z "${COMPONENT}" ]; then
echo "COMPONENT not set, select either RULER or ALERTMANAGER"
exit 1
Comment on lines +31 to +32
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
echo "COMPONENT not set, select either RULER or ALERTMANAGER"
exit 1
echo "COMPONENT not set, using RULER as a default."
COMPONENT=RULER

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would make sure we do not break existing setups.

fi

if [ -z "${RULES_DIR}" ]; then
echo "RULES_DIR not set, using './' as a default."
RULES_DIR="./"
fi

if [ ${COMPONENT} == "ALERTMANAGER" ] && [ -z "${ALERTMANAGER_CONFIG_PATH}" ]; then
echo "No ALERTMANAGER_CONFIG_PATH variable set. Defaulting to ./alertManager.yml"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
echo "No ALERTMANAGER_CONFIG_PATH variable set. Defaulting to ./alertManager.yml"
echo "ALERTMANAGER_CONFIG_PATH not set, using ./alertmanager.yml as a default."

ALERTMANAGER_CONFIG_PATH=./alertManager.yml
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
ALERTMANAGER_CONFIG_PATH=./alertManager.yml
ALERTMANAGER_CONFIG_PATH=./alertmanager.yml

fi

if [ -z "${ACTION}" ]; then
err "ACTION has not been set."
exit 1
fi

case "${ACTION}" in
$SYNC_CMD)
verifyTenantAndAddress
OUTPUT=$(/usr/bin/cortextool rules sync --rule-dirs="${RULES_DIR}" "$@")
STATUS=$?
;;
$DIFF_CMD)
verifyTenantAndAddress
OUTPUT=$(/usr/bin/cortextool rules diff --rule-dirs="${RULES_DIR}" "$@")
STATUS=$?
;;
$LINT_CMD)
OUTPUT=$(/usr/bin/cortextool rules lint --rule-dirs="${RULES_DIR}" "$@")
STATUS=$?
;;
$PREPARE_CMD)
OUTPUT=$(/usr/bin/cortextool rules prepare -i --rule-dirs="${RULES_DIR}" "$@")
STATUS=$?
;;
$CHECK_CMD)
OUTPUT=$(/usr/bin/cortextool rules check --rule-dirs="${RULES_DIR}" "$@")
STATUS=$?
;;
$PRINT_CMD)
OUTPUT=$(/usr/bin/cortextool rules print "$@")
if [ "${COMPONENT}" == "RULER" ]; then
case "${ACTION}" in
$SYNC_CMD)
verifyTenantAndAddress
OUTPUT=$(/usr/bin/cortextool rules sync --rule-dirs="${RULES_DIR}" "$@")
STATUS=$?
;;
*)
err "Unexpected action '${ACTION}'"
exit 1
;;
esac
$DIFF_CMD)
verifyTenantAndAddress
OUTPUT=$(/usr/bin/cortextool rules diff --rule-dirs="${RULES_DIR}" "$@")
STATUS=$?
;;
$LINT_CMD)
OUTPUT=$(/usr/bin/cortextool rules lint --rule-dirs="${RULES_DIR}" "$@")
STATUS=$?
;;
$PREPARE_CMD)
OUTPUT=$(/usr/bin/cortextool rules prepare -i --rule-dirs="${RULES_DIR}" "$@")
STATUS=$?
;;
$CHECK_CMD)
OUTPUT=$(/usr/bin/cortextool rules check --rule-dirs="${RULES_DIR}" "$@")
STATUS=$?
;;
$PRINT_CMD)
OUTPUT=$(/usr/bin/cortextool rules print "$@")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
OUTPUT=$(/usr/bin/cortextool rules print "$@")
verifyTenantAndAddress
OUTPUT=$(/usr/bin/cortextool rules print "$@")

STATUS=$?
;;
*)
err "Unexpected action '${ACTION}'"
exit 1
;;
esac
elif [ "${COMPONENT}" == "ALERTMANAGER" ]; then
OUTPUT=$(/usr/bin/cortextool alertmanager load ${ALERTMANAGER_CONFIG_PATH})
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you have time to add the other commands just like the previous block?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
OUTPUT=$(/usr/bin/cortextool alertmanager load ${ALERTMANAGER_CONFIG_PATH})
verifyTenantAndAddress
OUTPUT=$(/usr/bin/cortextool alertmanager load ${ALERTMANAGER_CONFIG_PATH})

STATUS=$?
else
echo "Invalid component selected, use RULER or ALERTMANAGER"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
echo "Invalid component selected, use RULER or ALERTMANAGER"
echo "COMPONENT is not valid, please use RULER or ALERTMANAGER."

exit 1
fi

echo "${OUTPUT}"
echo ::set-output name=detailed::"${OUTPUT}"
Expand Down