diff --git a/README.md b/README.md index 54165d7..e6c79d3 100644 --- a/README.md +++ b/README.md @@ -1,44 +1,79 @@ ## Tasques [![Build Status](https://travis-ci.org/lloydmeta/tasques.svg?branch=master)](https://travis-ci.org/lloydmeta/tasques) [![codecov](https://codecov.io/gh/lloydmeta/tasques/branch/master/graph/badge.svg)](https://codecov.io/gh/lloydmeta/tasques) [![](https://images.microbadger.com/badges/commit/lloydmeta/tasques.svg)](https://microbadger.com/images/lloydmeta/tasques "tasques docker image details") -Task queues backed by ES: Tasques. +> Task queues backed by Elasticsearch (ES): Tasques +> +> Pronounced: /tɑːsks/, like "tasks" +> +> You know, for Background Tasks ! + +Why use ES as a Tasks data store? It's horizontally scalable, highly-available, and offers a lot of built-in ways to manage your data +(lifecycle management, snapshots, etc). + +

+ dashboard +

### Features: - Easily scalable: - - Servers are stateless; easily spin more up as needed + - Servers are stateless; just spin more up as needed - The storage engine is Elasticsearch, nuff' said. -- Tasks can be configured +- Tasks are configurable: - Priority - - When to run - - Retries - - Tasks can be configured to retry X times, with exponential increase in run times -- Timeouts - - Tasks that are picked up by workers that either don't report in or finish on time get timed out. -- Unclaiming - - Tasks that were picked up but can't be handled now can be requeued without consequence. -- Recurring Tasks - - Tasks that are repeatedly enqueued at configurable intervals (cron format with basic macro support) - -### Requirements - -1. Go 1.13+ + - Schedule to run later + - Retries with exponential increase in retry delays. +- Recurring Tasks that are repeatedly enqueued at configurable intervals (cron format with basic macro support à la + `@every 1m`) +- Timeouts for Tasks that are picked up by Workers but either don't report in or finish on time. +- Unclaiming allows Tasks that get picked up but can't be handled to be requeued without consequence. +- API is exposed as Swagger; easily generate clients in any language: + - Use the client to enqueue Tasks from your application + - Workers are just a loop around the client, then your own business logic to do the actual work. +- Pre-seeded Kibana Index Patterns and Dashboards for monitoring tasks. +- Simple configuration: use a config file, optionally override with environment variables (12 factor-ready). +- Application Performance monitoring: metrics are exported to [APM](https://www.elastic.co/apm) and available again from + Kibana (more below) ### Usage #### Running -1. Go to `docker/k8s` and run `make install-eck deploy`, and wait until the pods are all ready. +Tasques is available as a [small Docker image](https://hub.docker.com/repository/docker/lloydmeta/tasques), with images +published automatically by CI upon pushes/merges to `master` and tag pushes. + +To get a quick idea of what is included and how to get up and running with Kubernetes: + +1. Go to `docker/k8s` and run `make install-eck deploy`, and wait until the pods are all ready (`kubectl get pods`) 2. For Swagger, go to [localhost:8080/swagger/index.html](http://localhost:8080/swagger/index.html) ![Swagger](swagger.png) +3. To log into Kibana for dashboards and APM stats, run `make show-credentials` to get the `elastic` user password, and + go to [localhost:5601](http://localhost:5601) to log in. There is also an example project that demonstrates the application-tasques-worker relationship more thoroughly; please see `example/ciphers` for more details. -##### APM +### Monitoring + +

+ APM +

The server supports APM, as configured according to the [official docs](https://www.elastic.co/guide/en/apm/agent/go/current/getting-started.html#configure-setup). -#### Dev +### High availability + +The Tasque server is in principle stateless; but there are internal recurring jobs that need to be taken care off, like +monitoring claimed Tasks and timing them out as needed, and scheduling recurring Tasks. + +These internal jobs only occur on the Leader server, as determined by a leader lock. By spinning up more than one Tasque server, +you not only gain the benefits of being able to handle more load, but also shield yourself from potential disruptions in the running +of these internal Tasks, as a new Leader will be elected and take over if the current Leader loses connectivity or is terminated. + +Running multiple servers also allows for zero-downtime rollouts of new versions of Tasques server. + +### Dev + +Requires [Go](https://golang.org) 1.13+. 1. [Install `Go`](https://golang.org/doc/install) 2. Use your favourite editor/IDE @@ -50,4 +85,15 @@ The server supports APM, as configured according to the [official docs](https:// 4. For updating the Go Client: 1. Install [go-swagger](https://goswagger.io/generate/client.html) 2. Run `swagger generate client -f docs/swagger.yaml` - 3. Commit the generated files. \ No newline at end of file + 3. Commit the generated files. + +#### Code guidelines + +The code emphasises the following: + +1. **Safety**: the code needs to do the right thing. Use built-in features (locks, typed ids) and tests to help maximise safety. +2. **Efficiency**: the server and its code should be reasonably efficient so that it can handle high loads. +3. **Observability**: where reasonable, pass a `Context` around so we can export performance metrics. This is a must when making any kind of IO call. +4. **High availability**: the server needs to be able to run in a highly available way to maximise robusness. +5. **Simplicity**: the API needs to be simple and easy to use +6. **Maintainability**: the internals need to be as simple as possible and invite contributions. Use the rule of 3 to know when something should be generalised. If in doubt, repeat yourself. \ No newline at end of file diff --git a/apm.png b/apm.png new file mode 100644 index 0000000..8ce5554 Binary files /dev/null and b/apm.png differ diff --git a/config/tasques.example.yaml b/config/tasques.example.yaml index 64a3e0f..81fa8d3 100644 --- a/config/tasques.example.yaml +++ b/config/tasques.example.yaml @@ -5,7 +5,7 @@ # Note this is used as the default template in the Docker image, so settings # should not be added/removed haphazardly # -# Note that each setting can be overwritten using env vars with under score `_` +# Note that each setting can be overwritten using env vars with underscore `_` # substituting for the dot when nesting (e.g. `TASQUES_SERVER_BIND_ADDRESS="0.0.0.0:8080"`) tasques: diff --git a/dashboard.png b/dashboard.png new file mode 100644 index 0000000..21ef83c Binary files /dev/null and b/dashboard.png differ diff --git a/docker/k8s/README.md b/docker/k8s/README.md index fdd0068..b016bf1 100644 --- a/docker/k8s/README.md +++ b/docker/k8s/README.md @@ -5,3 +5,8 @@ This module allows you to run the a Tasques server for playing around. * `make install-eck` to install the [ECK](https://github.com/elastic/cloud-on-k8s) k8s operator * `make deploy` spins up a working env * `make teardown` tears ... it down.. + +### Configuration + +Tasques can be configured via a config file or via environment variable; see the `tasques.yml` k8s definition file +for examples. \ No newline at end of file diff --git a/swagger.png b/swagger.png index 5f3aede..c5fcb7d 100644 Binary files a/swagger.png and b/swagger.png differ