🤝 Handoff

Handoff is a library that allows you to bootstrap a server that runs scheduled and manually triggered e2e tests written in Go and is extensible through plugins.

Why Handoff?

More and more companies are building their software as distributed systems.

These are notoriously hard to test end-to-end as the number of services grow. At some point starting all of them and their dependencies locally on the developer's machine becomes unfeasible.

This means that if you want to make sure your system works as expected you need to run tests in an environment that is as close to production as possible, such as a development or staging cluster.

This is where Handoff comes in. You can run Handoff alongside your system as a standalone server that runs all your end-to-end tests either

• on demand (via cli, api or ui) or
• repeatedly through a configurable schedule

On top of that there is a baked in web ui where you can look up the test results and manually trigger new runs.

Other valuable (future/planned) features:

• Matching and linking test runs to logs / traces / metrics generated by the systems under test (SUT) for easier debugging.
• Running smoke tests on new deployments for automatic rollbacks if necessary (argocd, flux).
• Fighting flaky tests via
• auto detection
• triggering multiple scheduled runs of the same test suite to provoke a test failure which can be debugged thereafter
• Maintaining historic test run data for deeper insights on the stability of the platform and making it available via a /metrics endpoint.
• Triggering of alerts on test failures (e.g. via pagerduty).
• Automated notifications on test runs (e.g. via slack messages).
• Github Integration: adding test run results to relevant PRs after they were merged and deployed.

Example

Bootstrapping a server is simple, all you need to do is run this code:

package main

func main() {
	h := handoff.New()
	h.Run()
}

To pass in test suites and scheduled runs you can do that by passing in handoff.WithTestSuite and handoff.WithScheduledRun options to handoff.New().

Another way is to register them via handoff.Register before calling handoff.New(). This is especially convenient when you want to have your tests in the same repository as the system under test (SUT), which means they would be in a different repository (unless you have a monorepo). In this case the test package could register the tests in an init function like so:

func init() {
	handoff.Register(ts, scheduledRuns)
}

and then all the handoff server needs to do is import the test package with a blank identifier:

import _ "github.com/my-org/my-service/tests"

For examples see [./cmd/example-server-bootstrap/main.go] and [./internal/packagetestexample].

Build

templ generate
go build ./cmd/example-server-bootstrap/
./example-server-bootstrap

Run

./example-server-bootstrap

Live reload

Instead of generating templ files and building the binary you can also run the example server with live reload:

air

Web

If your server is running you can open your browser to e.g. http://localhost:1337/suites.

This will show you all available test suites.

To create a new test run you can use the post request in ./requests.http

httpyac requests.http

Local dev cluster

Prerequisites:

• Running Docker
• Tilt + Kind installed and on the path

Run

kind create cluster --config=kind-config.yaml
tilt up

If you press <space> the tilt UI will open up.

Once handoff is green in the dashboard you should be able to open up the ui here: http://localhost:1337/.

Test best practices

• Pass in the test context for longer running operations and check if it was cancelled.
• Only log messages via t.Log/t.Logf as other log messages will not show up in the test logs.
• Make sure that code in setup is idempotent as it can run more than once.

Planned features

See here.

Open questions

• How to add test timeouts (it's impossible to externally stop goroutines running user provided functions)?

Non goals

• Implement a new assertion library. We aim to be compatible with existing ones.

Metrics

Metrics are exposed via the /metrics endpoint.

Name	Type	Description	Labels
handoff_testsuites_running	gauge	The number of test suites currently running	namespace, suite_name
handoff_testsuites_started_total	counter	The number of test suite runs started	namespace, suite_name, result
handoff_tests_run_total	counter	The number of tests run	namespace, suite_name, result