diff --git a/docs.kosli.com/content/tutorials/report_aws_envs.md b/docs.kosli.com/content/tutorials/report_aws_envs.md new file mode 100644 index 000000000..5563ca1d3 --- /dev/null +++ b/docs.kosli.com/content/tutorials/report_aws_envs.md @@ -0,0 +1,292 @@ +--- +title: "How to report ECS, Lambda and S3 environments" +bookCollapseSection: false +weight: 508 +--- + +# How to report ECS, Lambda and S3 environments + +Kosli environments allow you to track changes in your physical/virtual runtime environments. Such changes must be reported from the runtime environment to Kosli. + +This tutorial shows you how to set up reporting of running artifacts from a Kubernetes cluster to Kosli. + + +## Different ways for reporting + +There are two different ways to report what's running in a Kubernetes cluster: + +- Using Kosli CLI (suitable for testing only) +- Using the [Kosli terraform module](https://registry.terraform.io/modules/kosli-dev/kosli-reporter/aws/latest) to setup a Lambda function to be triggered on AWS changes and report to Kosli. + +We describe how to use the different options below and you can choose what suites your needs. + +## Prerequisites + +To follow this tutorial, you will need to: + +- Have access to AWS. +- [Create a Kosli account](https://app.kosli.com/sign-up) if you have not got one already. +- [Create an ECS, Lambda or S3 Kosli environment](getting_started/environments/#create-an-environment) named `aws-env-tutorial` +- [Get a Kosli API token](/getting_started/service-accounts/) +- [Install Kosli CLI](/getting_started/install/) (only needed if you will report using CLI) +- [Install Terraform](https://developer.hashicorp.com/terraform/install) (only needed if you will use the Kosli terraform module) + +## Report snapshots using Kosli CLI + +This option is **only suitable for testing purposes**. +You need to create an AWS static credentials or equivalent and export the following environments variables: + +```shell {.command} +export AWS_REGION=yourAWSRegion +export AWS_ACCESS_KEY_ID=yourAWSAccessKeyID +export AWS_SECRET_ACCESS_KEY=yourAWSSecretAccessKey +``` + +{{< tabs "snapshot env" "col-no-wrap" >}} + +{{< tab "ECS" >}} +```shell {.command} +$ kosli snapshot ecs aws-env-tutorial \ + --cluster \ + --api-token \ + --org +``` +{{< /tab >}} + +{{< tab "Lambda" >}} +```shell {.command} +$ kosli snapshot lambda aws-env-tutorial \ + --function-names function1,function2 \ + --api-token \ + --org +``` +{{< /tab >}} + +{{< tab "S3" >}} +```shell {.command} +$ kosli snapshot s3 aws-env-tutorial \ + --bucket \ + --api-token \ + --org +``` +{{< /tab >}} + +{{< /tabs >}} + + +## Report snapshots using Terraform module + +You can use the Kosli reporter terraform module to setup a Lambda function which is triggered every time your ECS cluster, Lambda function(s) or S3 bucket changes. The Lambda function will report the running artifacts to Kosli by running the Kosli CLI. + +To setup the Lambda function using terraform, you need to follow these steps: + +1. [Authenticate to AWS](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html) + +2. Store the Kosli API key value in an AWS SSM parameter (SecureString type). By default, the Lambda Reporter function will search for the kosli_api_token SSM parameter, but it is also possible to set custom parameter name using kosli_api_token_ssm_parameter_name variable. + +3. Create a Terraform configuration by copying one of the examples below into a `main.tf` file. + +{{< tabs "terraform aws env" "col-no-wrap" >}} + +{{< tab "ECS" >}} +```hcl {.command} +terraform { + required_version = ">= 1.0.0" + + required_providers { + aws = { + source = "hashicorp/aws" + version = ">= 4.63" + } + random = { + source = "hashicorp/random" + version = ">= 3.5.1" + } + } +} + +provider "aws" { + region = local.region + + # Make it faster by skipping some checks + skip_metadata_api_check = true + skip_region_validation = true + skip_credentials_validation = true + skip_requesting_account_id = true +} + +locals { + reporter_name = "reporter-${random_pet.this.id}" + region = "eu-central-1" +} + +data "aws_caller_identity" "current" {} + +data "aws_canonical_user_id" "current" {} + +resource "random_pet" "this" { + length = 2 +} + +module "lambda_reporter" { + source = "kosli-dev/kosli-reporter/aws" + version = "0.4.0" + + name = local.reporter_name + kosli_environment_type = "ecs" + kosli_cli_version = "2.7.5" + kosli_environment_name = "aws-env-tutorial" + kosli_org = "" + reported_aws_resource_name = "" +} +``` +{{< /tab >}} + + +{{< tab "Lambda" >}} +```hcl {.command} +terraform { + required_version = ">= 1.0.0" + + required_providers { + aws = { + source = "hashicorp/aws" + version = ">= 4.63" + } + random = { + source = "hashicorp/random" + version = ">= 3.5.1" + } + } +} + +provider "aws" { + region = local.region + + # Make it faster by skipping some checks + skip_metadata_api_check = true + skip_region_validation = true + skip_credentials_validation = true + skip_requesting_account_id = true +} + +locals { + reporter_name = "reporter-${random_pet.this.id}" + region = "eu-central-1" +} + +data "aws_caller_identity" "current" {} + +data "aws_canonical_user_id" "current" {} + +resource "random_pet" "this" { + length = 2 +} + +variable "my_lambda_functions" { + type = string + default = "function_name1, function_name2" +} + +module "lambda_reporter" { + source = "kosli-dev/kosli-reporter/aws" + version = "0.4.0" + + name = local.reporter_name + kosli_environment_type = "lambda" + kosli_cli_version = "2.7.5" + kosli_environment_name = "aws-env-tutorial" + kosli_org = "" + reported_aws_resource_name = var.my_lambda_functions + use_custom_eventbridge_pattern = true + custom_eventbridge_pattern = local.custom_event_pattern +} + +locals { + lambda_function_names_list = split(",", var.my_lambda_functions) + + custom_event_pattern = jsonencode({ + source = ["aws.lambda"] + detail-type = ["AWS API Call via CloudTrail"] + detail = { + requestParameters = { + functionName = local.lambda_function_names_list + } + responseElements = { + functionName = local.lambda_function_names_list + } + } + }) +} +``` +{{< /tab >}} + +{{< tab "S3" >}} +```hcl {.command} +terraform { + required_version = ">= 1.0.0" + + required_providers { + aws = { + source = "hashicorp/aws" + version = ">= 4.63" + } + random = { + source = "hashicorp/random" + version = ">= 3.5.1" + } + } +} + +provider "aws" { + region = local.region + + # Make it faster by skipping some checks + skip_metadata_api_check = true + skip_region_validation = true + skip_credentials_validation = true + skip_requesting_account_id = true +} + +locals { + reporter_name = "reporter-${random_pet.this.id}" + region = "eu-central-1" +} + +data "aws_caller_identity" "current" {} + +data "aws_canonical_user_id" "current" {} + +resource "random_pet" "this" { + length = 2 +} + +variable "my_lambda_functions" { + type = string + default = "my_lambda_function1, my_lambda_function_name2" +} + +module "lambda_reporter" { + source = "kosli-dev/kosli-reporter/aws" + version = "0.4.0" + + name = local.reporter_name + kosli_environment_type = "s3" + kosli_cli_version = "2.7.5" + kosli_environment_name = "aws-env-tutorial" + kosli_org = "" + reported_aws_resource_name = "" +} +``` +{{< /tab >}} + +{{< /tabs >}} + +4. Initialize and run Terraform by running: + +```shell {.command} +$ terraform init +$ terraform apply +``` + +5. To check Lambda reporter logs you can go to the AWS console -> Lambda service -> choose your lambda reporter function -> Monitor tab -> Logs tab. \ No newline at end of file diff --git a/docs.kosli.com/content/tutorials/report_k8s_envs.md b/docs.kosli.com/content/tutorials/report_k8s_envs.md new file mode 100644 index 000000000..34e34e58b --- /dev/null +++ b/docs.kosli.com/content/tutorials/report_k8s_envs.md @@ -0,0 +1,180 @@ +--- +title: "How to report Kubernetes Clusters" +bookCollapseSection: false +weight: 507 +--- + +# How to report Kubernetes Clusters to Kosli + +Kosli environments allow you to track changes in your physical/virtual runtime environments. Such changes must be reported from the runtime environment to Kosli. + +This tutorial shows you how to set up reporting of running artifacts from a Kubernetes cluster to Kosli. + + +## Different ways for reporting + +There are 3 different ways to report what's running in a Kubernetes cluster: + +- Using Kosli CLI (suitable for testing only) +- Using a Kubernetes cronjob configured with a helm chart (recommended for production use). +- Using an externally scheduled cron process (e.g. a scheduled CI workflow) + +We describe how to use the different options below and you can choose what suites your needs. + +## Prerequisites + +To follow this tutorial, you will need to: + +- Have access to a Kubernetes cluster. +- [Create a Kosli account](https://app.kosli.com/sign-up) if you have not got one already. +- [Create a Kubernetes Kosli environment](getting_started/environments/#create-an-environment) named `k8s-tutorial` +- [Get a Kosli API token](/getting_started/service-accounts/) +- [Install Kosli CLI](/getting_started/install/) (only needed if you will report using CLI) +- [Install Helm](https://helm.sh/docs/intro/install/) (only needed if you will use the Kosli helm chart) + +## Report snapshots using Kosli CLI + +This option is **only suitable for testing purposes**. + +> All the commands below will use the default `kubecontext` in "$HOME/.kube/config". You can change it with `--kubeconfig` + +To report the **artifacts running in an entire cluster**, you can run the following command: + +```shell {.command} +$ kosli snapshot k8s k8s-tutorial \ + --api-token \ + --org +``` + +To report **artifacts running in one or more namespaces**, you can run the following command: + +```shell {.command} +$ kosli snapshot k8s k8s-tutorial \ + --namespaces namespace1,namespace2 \ + --api-token \ + --org +``` + +To report **artifacts running in the entire cluster except from some namespaces**, you can run the following command: + +```shell {.command} +$ kosli snapshot k8s k8s-tutorial \ + --exclude-namespaces namespace1,namespace2 \ + --api-token \ + --org +``` + +## Report snapshots using the Kosli K8S reporter helm chart + +The recommended way to regularly report artifacts running in a cluster to Kosli is to use the [K8S reporter helm chart](/helm). + +The chart creates a cronjob that will run the Kosli CLI inside a pod to report the artifacts running in the cluster. + +1. Create a K8S secret to contain your Kosli API token. + +```shell {.command} +$ kubectl create secret generic kosli-api-token --from-literal=apikey= +``` + +> Make sure the secret value does not contain any trailing whitespace. + +2. Prepare the settings for the helm chart + +To customize how the helm chart creates the cronjob, you can create your own values file by copying and modifying the [default values file](https://github.com/kosli-dev/cli/blob/main/charts/k8s-reporter/values.yaml). + +We will use this file (named `tutorial-values.yaml`): + +```yaml {.command} +# -- the cron schedule at which the reporter is triggered to report to kosli +cronSchedule: "*/5 * * * *" + +kosliApiToken: + # -- the name of the secret containing the kosli API token + secretName: "kosli-api-token" + # -- the name of the key in the secret data which contains the kosli API token + secretKey: "apikey" + +reporterConfig: + # -- the name of the kosli org + kosliOrg: "" + # -- the name of kosli environment that the k8s cluster/namespace correlates to + kosliEnvironmentName: "k8s-tutorial" + # -- the namespaces which represent the environment. + # It is a comma separated list of namespace name regex patterns. + # e.g. `^prod$,^dev-*` reports for the `prod` namespace and any namespace that starts with `dev-` + # leave this unset if you want to report what is running in the entire cluster + namespaces: "" +``` + +3. Install the Kosli helm chart + +```shell {.command} +$ helm repo add kosli https://charts.kosli.com/ +$ helm repo update +$ helm install kosli-reporter kosli/k8s-reporter -f tutorial-values.yaml +``` + +4. Confirm the cronjob is created in the cluster: + +```shell {.command} +$ kubectl get cronjobs +``` + +Now, the cronjob will run every 5 minutes and report what is running in the entire cluster to Kosli. + + +## Report snapshots using externally scheduled cronjobs + +If you do not wish to run the Kosli reporter inside the cluster, you can run it from outside the cluster. This requires opening access to the cluster from the place you will run the CLI regularly. + +One option to send reports regularly from outside the cluster is to use Github Actions scheduled workflows. Here is an example workflow definition: + +> Note that the workflow below needs secrets to be added in Github actions. + +```yaml {.command} +name: Regular Kubernetes reports to Kosli + +on: + workflow_dispatch: + schedule: + - cron: '0 * * * *' # every one hour + +jobs: + k8s-report: + runs-on: ubuntu-latest + permissions: + id-token: write + contents: write + env: + KOSLI_API_TOKEN: ${{ secrets.MY_KOSLI_API_TOKEN }} + + steps: + - name: install kosli + uses: kosli-dev/setup-cli-action@v2 + + # connect to your cluster + # if not using GKE, replace this step with one that connects to your cluster + - name: Connect to GKE + uses: 'Swibi/connect-to-gke' + with: + GCP_SA_KEY: ${{ secrets.GKE_SA_KEY }} + GCP_PROJECT_ID: ${{ secrets.GKE_PROJECT }} + GKE_CLUSTER: + GKE_ZONE: + + - name: Scan artifacts and send K8S report to Kosli + run: + kosli snapshot k8s k8s-tutorial --org + + # send slack notifications on failure to report + - name: Slack Notification on Failure + if: ${{ failure() }} + uses: rtCamp/action-slack-notify@v2 + env: + SLACK_CHANNEL: kosli-reports-failure + SLACK_COLOR: ${{ job.status }} + SLACK_TITLE: Reporting K8S artifacts to Kosli has failed + SLACK_USERNAME: GithubActions + SLACK_WEBHOOK: ${{ secrets.SLACK_CI_FAILURES_WEBHOOK }} + SLACK_MESSAGE: "Reporting K8S artifacts to Kosli has failed. Please check the logs for more details." +```