Skip to content

Ship logs from cloud.gov into your custom logging solution

License

Notifications You must be signed in to change notification settings

GSA/datagov-logstack

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

cg-logstack

Drain logs from cloud.gov into your custom logging solution

Prerequisites

For deployment

For development, add

  • Docker (tested with Docker Engine v20.10.10)
  • Python (tested with v3.8)

Usage

Log drains

Use the drain plugin to configure the log drain for each app in a space.

cf drain <app-name> https://<username>:<password>@<drain-app-route>

Alternatively, you can auto-drain all apps in a given space by targeting that space, then running the ./create-space-drain.sh script.

cf target -s prod
./create-space-drain.sh

Note: at development space, it will use the logstack-shipper in development-ssb space:

cf target -s development
./create-space-drain.sh development-ssb

After a short delay, logs should begin to flow automatically.

Setup

Set your application name.

app_name=logstack

Copy vars.example.yml to vars.yml (or a space-specific version) and customize for your application.

Create an S3 bucket instance.

cf create-service s3 basic ${app_name}-s3

Create a user provided service for secrets.

Push the applications.

cf push --vars-file vars.yml

Secrets

Provide secrets for the logstack applications via a user-provided service.

cf cups ${app_name}-secrets -p DRAIN_USER,DRAIN_PASSWORD
Name Description Where to find?
DRAIN_PASSWORD Password for basic authentication on the Logstash proxy randomly generated
DRAIN_USER Username for basic authentication on the Logstash proxy randomly generated

Applications

The logstack application is made up of several smaller Cloud Foundry applications.

Name Description
logstack-shipper Logstash process that aggregates and parses log data.
logstack-space-drain Space drain monitors a CF space, and binds the log drain to applications. Created by the drains plugin.

Note: The logstack-space-drain application consumes 64MB.

Development

Run tests.

docker compose run --rm test

Contributing

See CONTRIBUTING for additional information.

Public domain

This project is in the worldwide public domain. As stated in CONTRIBUTING:

This project is in the public domain within the United States, and copyright and related rights in the work worldwide are waived through the CC0 1.0 Universal public domain dedication.

All contributions to this project will be released under the CC0 dedication. By submitting a pull request, you are agreeing to comply with this waiver of copyright interest.