ltest

A custom test runner for clojure.test with detailed, coloured output and summaries

Contents

• About
• Feature List
• Usage
  • lein Plugin
  • Function Calls
    • Running Multiple Tests
    • Running One Test
    • Running a Suite
    • Running Multiple Suites
  • Creating a Test Runner
    • Tagging Test Namespaces
    • A runner Namespace
    • Adding a lein Alias
    • Running without ltest
• License

About ↟

This project started as a complaint (a numbered list of them) on a Slack channel about the default test runner for clojure.test. Most of these relating to the fact that there's not quite enough information presented to maximally assist in debugging ... that one often has to piece things together. Thanks to @chris-durbin, who urged an implementation follow-up to make things better, ltest for Clojure is now a thing.

The Clojure ltest test runner was inspired by the LFE ltest runner, whence it got its name.

The ltest library is currently being used in the NASA Earthdata CMR project as a supplementary development/testing tool, and in the CMR Client library.

Feature List ↟

The basic needs ltest aims to resolve (admittedly important for only a subset of developers) are the following:

• a detailed and explicit reporting-while-testing on what's getting tested
  • ✅ namespace
  • ✅ function
  • ✅ assertion
  • text of (testing ...) call (see ticket #11)
• ✅ explicit test status for each assertion (OK, FAIL, ERROR)
• a separation of reporting-while-testing and failure/error details
  • ✅ the running status of tests should be kept visually clean
  • ✅ failures and errors should be grouped separately
  • ✅ failure and error reporting should be done at the end, after the summary, in their own sections
  • suites should hold off until all suites have finished before reporting (see ticket #12)
• ✅ failure and error reporting should include the full namespace + function of where the issue occurred for easier viewing/copying+pasting
• ✅ different status output, sections, etc., should use ANSI terminal colors to assist with quick and easy identification of issues, data, etc.
• ✅ tests should be ordered alphabetically
• ✅ users/developers should have the ability to form arbitrary high-level divisions of tests
  • ✅ useful for running unit tests and integration tests together
  • ✅ called "suites" in ltest
  • an opinionated default grouping for suites should be offered
    • ✅ by default, group by the top two elements of a namespace (e.g., a.b.c.d and a.b.e.f would both be grouped in a.b)
    • developers should have the ability to override this easily (see ticket #25)

Usage ↟

The functionality provided by this library may be used in several ways:

• As a tool (set of functions) from a development REPL
• As a utility library for creating a test runner for your project
• As the basis for a lein plugin (not yet created; see ticket #10)

The first two are discussed below. In both cases, the ltest library is utilized solely through its primary namespace, e.g.:

(require '[ltest.core :as ltest])

lein Plugin ↟

The simplest way to use ltest is with the lein plugin. The latest version is here:

Add this to your project as a lein plugin, e.g.:

...
:plugins [[lein-ltest "x.y.z"]]
...

And then run your tests with the following:

$ lein ltest

Function Calls ↟

Running Multiple Tests ↟

Collections of tests may be run with the (ltest/run-tests) function. The following example passing just one test namespace, but any number may be passed as additional arguments:

(ltest/run-tests ['ltest.group1.samples.sample1])

Here's is a screenshot of this call's result in the ltest dev environment (click for a larger view):

Note that this includes, in order:

• summary results
• failure listings
• error listings

Running One Test ↟

A similar approach with analagous reporting is available for running single tests, but instead of a namespace, a namespace-qualified test function (as var) is passed:

(ltest/run-test #'ltest.group1.samples.sample2/multiple-pass-test)

Screenshot:

Running a Suite ↟

In ltest, test suites are aribitrary named groupings of tests. As with run-tests, any number of namespaces my be provided in the :nss vector:

(ltest/run-suite {:name "Simple Suite"
                  :nss ['ltest.group1.samples.sample2]})

Screenshot:

Running Multiple Suites ↟

You can also define multiple suites and run them together (useful for unit and integration tests):

(def suite-1
  {:name "Arbitrary Division 1"
   :nss ['nogroup
         'ltest.group1.samples.sample0
         'ltest.group1.samples.sample1]})

(def suite-2
  {:name "Arbitrary Division 2"
   :nss [:ltest.group1.samples.sample2
         "ltest.group1.samples.sample3"
         'ltest.group2.samples.sample4
         'ltest.group2.samples.sample5
         'ltest.group2.samples.sample6
         'ltest.group2.samples.sample7]})

(def suites
  [suite-1 suite-2])

(ltest/run-suites suites)

Screenshot:

Creating a Test Runner ↟

The CMR client library has opted to use ltest to build a quick test runner that can be executed from the command line, via a lein alias. This is really just a workaround until ltest has an official lein plugin.

The steps for creating a test runner are given in the following sub-sections.

Tagging Test Namespaces ↟

For all namespaces you want to qualify as containing unit tests, simply update the namespace for the given file, e.g.,

from this:

(ns ur.proj.tests.util
  ...)

to this:

(ns :unit ur.proj.tests.util
  ...)

Likewse for integration and system tests:

(ns :integration ur.proj.tests.server
  ...)

(ns :system ur.proj.tests.services
  ...)

A runner Namespace ↟

In this particular runner, the "suite" functionality of ltest is not taken advantage of; instead, our example runner below relies upon metadata tags in the test namespaces we've made. Note that the ltest/run-*-tests functions are conveniences provided by ltest; if you should want to tag your tests in any other arbitrary manner, creating convenience functions for your tags (e.g., to be used by the test runner) is very easy (see the ltest.core source for hints).

Here is a sample runner namespace, intended to be called from the command line:

(ns ur.proj.testing.runner
  (:require
   [cmr.client.tests]
   [ltest.core :as ltest])
  (:gen-class))

(def tests-regex #"ur\.proj\.tests\..*")

(defn run-tests
  []
  (ltest/run-all-tests tests-regex))

(defn print-header
  []
  (println)
  (println (apply str (repeat 80 "=")))
  (println "Your Project Test Runner")
  (println (apply str (repeat 80 "=")))
  (println))

(defn -main
  "This can be run from `lein` in the following ways:
  * `lein run-tests`
  * `lein run-tests unit`
  * `lein run-tests integration`
  * `lein run-tests system`"
  [& args]
  (print-header)
  (case (keyword (first args))
    :unit (ltest/run-unit-tests tests-regex)
    :integration (ltest/run-integration-tests tests-regex)
    :system (ltest/run-system-tests tests-regex)
    (run-tests)))

Adding a lein Alias ↟

Let's add an alias to easily execute our test runner from the command line. In your project.clj file, add a new section (if you don't already have it) siblimg to the :profiles or :dependencies sections.

  ...
  :aliases {
    ...
    "run-tests"
      ^{:doc "Use the ltest runner for verbose, colourful test output"}
      ["with-profile" "+test" "run" "-m" "ur.proj.testing.runner"]
    ...}
  ...

Now you can use that to run the following, optionally limiting tests to what they have been tagged in their namespace:

• lein run-tests (will run all types of tests)
• lein run-tests unit
• lein run-tests integration
• lein run-tests system

Running without ltest ↟

In the example above, we used the arbitrary namespace metadata of :unit, :integration, and :system for our different types of tests. Now that you have made these annotations in the test namespaces, you can use them directly with lein, too (without using ltest, should you so choose). Simply add this to your project.clj file's :test profile:

  ...
  :profiles {
    ...
    :test {
      ...
      :test-selectors {
        :default :unit
        :unit :unit
        :integration :integration
        :system :system}
      }}}
  ...

Now you can run the following to just test the parts of the project you want:

• lein test (will just run unit tests)
• lein test :unit
• lein test :integration
• lein test :system

License ↟

Copyright © 2017, Clojure-Aided Enrichment Center

Distributed under the Apache License, Version 2.0.