|
| 1 | +# Configuration |
| 2 | + |
| 3 | +Compass binary contains both the CLI client and the server. Each has it's own configuration in order to run. Server configuration contains information such as database credentials, elastic search brokers, log severity, etc. while CLI client configuration only has configuration about which server to connect. |
| 4 | +## Server Setup |
| 5 | + |
| 6 | +There are several approaches to setup Compass Server |
| 7 | + |
| 8 | +1. [Using the CLI](#using-the-cli) |
| 9 | +1. [Using the Docker](#using-the-docker) |
| 10 | +2. [Using the Helm Chart](#using-the-helm-chart) |
| 11 | + |
| 12 | +#### General pre-requisites |
| 13 | + |
| 14 | +- PostgreSQL (version 13 or above) |
| 15 | +- ElasticSearch v7 (optional) |
| 16 | + |
| 17 | +## Using the CLI |
| 18 | +### Using config file |
| 19 | + |
| 20 | +Create a config file with the following command |
| 21 | + |
| 22 | +```sh |
| 23 | +$ compass config init |
| 24 | +``` |
| 25 | + |
| 26 | +alternatively you can [use `--config` flag](#using---config-flag) to customize to config file location. |
| 27 | + |
| 28 | +You can also [use environment variables](#using-environment-variable) to provide the server configuration. |
| 29 | + |
| 30 | +Setup up the Postgres database, and ElasticSearch instance and provide the details as shown in the example below. |
| 31 | + |
| 32 | +> If you're new to YAML and want to learn more, see [Learn YAML in Y minutes.](https://learnxinyminutes.com/docs/yaml/) |
| 33 | +
|
| 34 | +Following is a sample server configuration yaml: |
| 35 | + |
| 36 | +```yaml title="compass.yaml" |
| 37 | +log_level: info # debug|info|warning|error|fatal|trace|panic - default: info |
| 38 | + |
| 39 | +elasticsearch: |
| 40 | + brokers: http://localhost:9200 #required |
| 41 | + |
| 42 | +db: |
| 43 | + host: localhost #required |
| 44 | + port: 5432 #required |
| 45 | + name: compass #required |
| 46 | + user: compass #required |
| 47 | + password: compass_password #required |
| 48 | + sslmode: disable #optional |
| 49 | + |
| 50 | +service: |
| 51 | + host: localhost #required |
| 52 | + port: 8080 #required |
| 53 | + identity: |
| 54 | + headerkey_uuid: Compass-User-UUID #required |
| 55 | + headerkey_email: Compass-User-Email #optional |
| 56 | + provider_default_name: shield #optional |
| 57 | + grpc: |
| 58 | + port: 8081 #required |
| 59 | + max_send_msg_size: 33554432 |
| 60 | + max_recv_msg_size: 33554432 |
| 61 | +``` |
| 62 | +### Using environment variable |
| 63 | +
|
| 64 | +All the server configurations can be passed as environment variables using underscore _ as the delimiter between nested keys. |
| 65 | +
|
| 66 | +See [configuration reference](./reference/configuration.md) for the list of all the configuration keys. |
| 67 | +
|
| 68 | +```sh title=".env" |
| 69 | +LOG_LEVEL=info |
| 70 | +ELASTICSEARCH_BROKERS=http://localhost:9200 |
| 71 | +DB_HOST=localhost |
| 72 | +DB_PORT=5432 |
| 73 | +DB_NAME=compass |
| 74 | +DB_USER=compass |
| 75 | +DB_PASSWORD=compass_password |
| 76 | +DB_SSLMODE=disable |
| 77 | +SERVICE_HOST=localhost |
| 78 | +SERVICE_PORT=8080 |
| 79 | +SERVICE_IDENTITY_HEADERKEY_UUID=Compass-User-UUID |
| 80 | +SERVICE_IDENTITY_HEADERKEY_EMAIL=Compass-User-Email |
| 81 | +SERVICE_IDENTITY_PROVIDER_DEFAULT_NAME=shield |
| 82 | +SERVICE_GRPC_PORT=8081 |
| 83 | +SERVICE_GRPC_MAX_SEND_MSG_SIZE=33554432 |
| 84 | +SERVICE_GRPC_MAX_RECV_MSG_SIZE=33554432 |
| 85 | +``` |
| 86 | + |
| 87 | +Set the env variable using export |
| 88 | +```bash |
| 89 | +$ export DB_PORT = 5432 |
| 90 | +``` |
| 91 | +### Starting the server |
| 92 | + |
| 93 | +Database migration is required during the first server initialization. In addition, re-running the migration command might be needed in a new release to apply the new schema changes (if any). It's safer to always re-run the migration script before deploying/starting a new release. |
| 94 | + |
| 95 | +To initialize the database schema, Run Migrations with the following command: |
| 96 | +```bash |
| 97 | +$ compass server migrate |
| 98 | +``` |
| 99 | + |
| 100 | +To run the Compass server use command: |
| 101 | + |
| 102 | +```bash |
| 103 | +$ compass server start |
| 104 | +``` |
| 105 | + |
| 106 | +#### Using `--config` flag |
| 107 | + |
| 108 | +```bash |
| 109 | +$ compass server migrate --config=<path-to-file> |
| 110 | +``` |
| 111 | + |
| 112 | +```bash |
| 113 | +$ compass server start --config=<path-to-file> |
| 114 | +``` |
| 115 | + |
| 116 | +## Using the Docker |
| 117 | +To run the Compass server using Docker, you need to have Docker installed on your system. You can find the installation instructions [here](https://docs.docker.com/get-docker/). |
| 118 | + |
| 119 | +You can choose to set the configuration using environment variables or a config file. The environment variables will override the config file. |
| 120 | + |
| 121 | +If you use Docker to build compass, then configuring networking requires extra steps. Following is one of doing it by running postgres and elasticsearch inside with `docker-compose` first. |
| 122 | + |
| 123 | +Go to the root of this project and run `docker-compose`. |
| 124 | + |
| 125 | +```bash |
| 126 | +$ docker-compose up |
| 127 | +``` |
| 128 | +Once postgres and elasticsearch has been ready, we can run Compass by passing in the config of postgres and elasticsearch defined in `docker-compose.yaml` file. |
| 129 | + |
| 130 | +### Using config file |
| 131 | +Alternatively you can use the `compass.yaml` config file defined [above](#using-config-file) and run the following command. |
| 132 | + |
| 133 | +```bash |
| 134 | +$ docker run -d \ |
| 135 | + --restart=always \ |
| 136 | + -p 8080:8080 \ |
| 137 | + -v $(pwd)/compass.yaml:/compass.yaml \ |
| 138 | + --name compass-server \ |
| 139 | + odpf/compass:<version> \ |
| 140 | + server start -c /compass.yaml |
| 141 | +``` |
| 142 | + |
| 143 | +### Using environment variables |
| 144 | + |
| 145 | +All the configs can be passed as environment variables using underscore `_` as the delimiter between nested keys. See the example as discussed [above](#using-environment-variable) |
| 146 | + |
| 147 | +Run the following command to start the server |
| 148 | + |
| 149 | +```bash |
| 150 | +$ docker run -d \ |
| 151 | + --restart=always \ |
| 152 | + -p 8080:8080 \ |
| 153 | + --env-file .env \ |
| 154 | + --name compass-server \ |
| 155 | + odpf/compass:<version> \ |
| 156 | + server start |
| 157 | +``` |
| 158 | + |
| 159 | +## Using the Helm chart |
| 160 | + |
| 161 | +### Pre-requisites for Helm chart |
| 162 | +Compass can be installed in Kubernetes using the Helm chart from https://github.com/odpf/charts. |
| 163 | + |
| 164 | +Ensure that the following requirements are met: |
| 165 | +- Kubernetes 1.14+ |
| 166 | +- Helm version 3.x is [installed](https://helm.sh/docs/intro/install/) |
| 167 | + |
| 168 | +### Add ODPF Helm repository |
| 169 | + |
| 170 | +Add ODPF chart repository to Helm: |
| 171 | + |
| 172 | +``` |
| 173 | +helm repo add odpf https://odpf.github.io/charts/ |
| 174 | +``` |
| 175 | + |
| 176 | +You can update the chart repository by running: |
| 177 | + |
| 178 | +``` |
| 179 | +helm repo update |
| 180 | +``` |
| 181 | + |
| 182 | +### Setup helm values |
| 183 | + |
| 184 | +The following table lists the configurable parameters of the Compass chart and their default values. |
| 185 | + |
| 186 | +See full helm values guide [here](https://github.com/odpf/charts/tree/main/stable/compass#values) |
| 187 | + |
| 188 | +```yaml title="values.yaml" |
| 189 | +app: |
| 190 | + image: |
| 191 | + repository: odpf/compass |
| 192 | + pullPolicy: Always |
| 193 | + tag: "0.3.0" |
| 194 | + container: |
| 195 | + command: |
| 196 | + - compass |
| 197 | + args: |
| 198 | + - server start |
| 199 | + livenessProbe: |
| 200 | + httpGet: |
| 201 | + path: /ping |
| 202 | + port: tcp |
| 203 | + readinessProbe: |
| 204 | + httpGet: |
| 205 | + path: /ping |
| 206 | + port: tcp |
| 207 | + |
| 208 | + migration: |
| 209 | + enabled: true |
| 210 | + command: |
| 211 | + - compass |
| 212 | + args: |
| 213 | + - server migrate |
| 214 | + |
| 215 | + service: |
| 216 | + annotations: {} |
| 217 | + |
| 218 | + ingress: |
| 219 | + enabled: true |
| 220 | + annotations: |
| 221 | + kubernetes.io/ingress.class: contour |
| 222 | + hosts: |
| 223 | + - host: compass.example.com |
| 224 | + paths: |
| 225 | + - path: / |
| 226 | + pathType: ImplementationSpecific |
| 227 | + backend: |
| 228 | + service: |
| 229 | + # name: backend_01 |
| 230 | + port: |
| 231 | + number: 80 |
| 232 | + |
| 233 | + config: |
| 234 | + COMPASS_SERVICE_PORT: 8080 |
| 235 | + COMPASS_SERVICE_GRPC_PORT: 8081 |
| 236 | + # COMPASS_SERVICE_HOST: 0.0.0.0 |
| 237 | + # COMPASS_STATSD_ENABLED: false |
| 238 | + # COMPASS_STATSD_PREFIX: compass |
| 239 | + # COMPASS_NEWRELIC_ENABLED: false |
| 240 | + # COMPASS_NEWRELIC_APPNAME: compass |
| 241 | + # COMPASS_LOG_LEVEL: info |
| 242 | + |
| 243 | + secretConfig: {} |
| 244 | + # COMPASS_ELASTICSEARCH_BROKERS: ~ |
| 245 | + # COMPASS_STATSD_ADDRESS: ~ |
| 246 | + # COMPASS_NEWRELIC_LICENSEKEY: ~ |
| 247 | + # COMPASS_DB_HOST: ~ |
| 248 | + # COMPASS_DB_PORT: 5432 |
| 249 | + # COMPASS_DB_NAME: ~ |
| 250 | + # COMPASS_DB_USER: ~ |
| 251 | + # COMPASS_DB_PASSWORD: ~ |
| 252 | + # COMPASS_DB_SSLMODE: disable |
| 253 | +``` |
| 254 | + |
| 255 | +And install it with the helm command line along with the values file: |
| 256 | + |
| 257 | +```bash |
| 258 | +$ helm install my-release -f values.yaml odpf/compass |
| 259 | +``` |
| 260 | + |
| 261 | +## Client Initialisation |
| 262 | + |
| 263 | +Add client configurations in the same `~/compass.yaml` file in root of current directory. Open this file to configure client. |
| 264 | + |
| 265 | +```yml |
| 266 | +client: |
| 267 | + host: localhost:8081 |
| 268 | + serverheaderkey_uuid: Compass-User-UUID |
| 269 | + serverheadervalue_uuid: [email protected] |
| 270 | +``` |
| 271 | +
|
| 272 | +#### Required Header/Metadata in API |
| 273 | +Compass has a concept of [User](./concepts/user.md). In the current version, all HTTP & gRPC APIs in Compass requires an identity header/metadata in the request. The header key is configurable but the default name is `Compass-User-UUID`. |
| 274 | + |
| 275 | +Compass APIs also expect an additional optional e-mail header. This is also configurable and the default name is `Compass-User-Email`. The purpose of having this optional e-mail header is described in the [User](./concepts/user.md) section. |
| 276 | + |
| 277 | + |
| 278 | +If everything goes ok, you should see something like this: |
| 279 | +```bash |
| 280 | +time="2022-04-27T09:18:08Z" level=info msg="compass starting" version=v0.2.0 |
| 281 | +time="2022-04-27T09:18:08Z" level=info msg="connected to elasticsearch cluster" config="\"docker-cluster\" (server version 7.6.1)" |
| 282 | +time="2022-04-27T09:18:08Z" level=info msg="New Relic monitoring is disabled." |
| 283 | +time="2022-04-27T09:18:08Z" level=info msg="statsd metrics monitoring is disabled." |
| 284 | +time="2022-04-27T09:18:08Z" level=info msg="connected to postgres server" host=postgres port=5432 |
| 285 | +time="2022-04-27T09:18:08Z" level=info msg="server started" |
| 286 | +``` |
| 287 | + |
0 commit comments