Skip to content

Latest commit

 

History

History
141 lines (99 loc) · 4.23 KB

README.md

File metadata and controls

141 lines (99 loc) · 4.23 KB
Raw

fluent-bit s3 output plugin

Build Status Build status

Windows binaries are available in release pages.

This plugin works with fluent-bit's go plugin interface. You can use fluent-bit-go-s3 to ship logs into AWS S3.

The configuration typically looks like:

fluent-bit --> AWS S3

Usage

$ fluent-bit -e /path/to/built/out_s3.so -c fluent-bit.conf

Or,

$ docker build . -t fluent-bit/s3-plugin

and then, specify configuration parameters as environment variables:

$ docker run -it -e="FLUENT_BIT_ACCESS_KEY_ID=yourawsaccesskey" \
                 -e="FLUENT_BIT_SECRET_ACCESS_KEY=yourawsaccesssecret" \
                 -e="FLUENT_BIT_BUCKET_NAME=yourbucketname" \
                 -e="FLUENT_BIT_S3_PREFIX=yours3prefix" \
                 -e="FLUENT_BIT_REGION=awsregion" \
                 fluent-bit/s3-plugin

Using docker image from docker hub.

$ docker pull cosmo0920/fluent-bit-go-s3:latest

Other released images are available in DockerHub's fluent-bit-go-s3 image tags.

Or, using helm:

helm install [YOURRELEASENAME] ./helm/fluent-bit

Prerequisites

  • Go 1.11+
  • gcc (for cgo)
  • make

Building

$ make

Configuration Options

Key Description Default value Note
Credential URI of AWS shared credential "" (See Credentials)
AccessKeyID Access key ID of AWS "" (See Credentials)
SecretAccessKey Secret access key ID of AWS "" (See Credentials)
Bucket Bucket name of S3 storage - Mandatory parameter
S3Prefix S3Prefix of S3 key - Mandatory parameter
Region Region of S3 - Mandatory parameter
Compress Choose Compress method "" gzip or plainText("")
Endpoint Specify the endpoint URL "" URL with port or empty string
AutoCreateBucket Create bucket automatically false true/false
LogLevel Specify Log Level "info" trace/debug/info/warning/error/fatal/panic
TimeZone Specify TimeZone "" Specify TZInfo based region. e.g.) Asia/Tokyo

Example:

Add this section to fluent-bit.conf:

[Output]
    Name s3
    Match *
    # Credential    /path/to/sharedcredentialfile
    AccessKeyID     yourawsaccesskeyid
    SecretAccessKey yourawssecretaccesskey
    Bucket          yourbucketname
    S3Prefix yours3prefixname
    Region us-east-1
    Compress gzip
    # Endpoint parameter is mainly used for minio.
    # Endpoint http://localhost:9000
    # TimeZone Asia/Tokyo

fluent-bit-go-s3 supports the following credentials. Users must specify one of them:

Credentials

Specifying credentials is required.

This plugin supports the following credentials:

Shared Credentials

Create the following file which includes credentials:

[default]
aws_access_key_id = YOUR_AWS_ACCESS_KEY_ID
aws_secret_access_key = YOUR_AWS_SECRET_ACCESS_KEY

Then, specify the following parameter in fluent-bit configuration:

Credential    /path/to/sharedcredentialfile

Static Credentials

Specify the following parameters in fluent-bit configuration:

AccessKeyID     yourawsaccesskeyid
SecretAccessKey yourawssecretaccesskey

Environment Credentials

Specify AWS_ACCESS_KEY and AWS_SECRET_KEY as environment variables.

Useful links