Skip to content

Latest commit

 

History

History
2216 lines (1615 loc) · 80.2 KB

README.md

File metadata and controls

2216 lines (1615 loc) · 80.2 KB
Raw

ShellSpec: full-featured BDD unit testing framework

ShellSpec is a full-featured BDD unit testing framework for dash, bash, ksh, zsh and all POSIX shells that provides first-class features such as code coverage, mocking, parameterized test, parallel execution and more. It was developed as a dev/test tool for cross-platform shell scripts and shell script libraries. ShellSpec is a new modern testing framework released in 2019, but it's already stable enough. With lots of practical CLI features and simple yet powerful syntax, it provides you with a fun shell script test environment.

Ubuntu macOS FreeBSD NetBSD OpenBSD DragonFlyBSD Solaris11
gitbash cygwin msys busybox
CodeFactor Grade Code Climate Codecov Kcov Docker Cloud Automated build GitHub top language GitHub release License
bash bosh busybox dash ksh mksh posh yash zsh

Version 0.28.0 has a lot of enhancements in the CLI. It is basically compatible, but there are some changes that you need to be aware of. See Migration Guide to Version 0.28.0 for details.

Thank you for your interest in ShellSpec. Please visit 🚩the official website to know the impressive features!

Let's have fun testing your shell scripts! (Try Online Demo on your browser).

demo

Coverage report

Latest Update.

See CHANGELOG.md

NOTE: This documentation contains unreleased features. Check them in the changelog.

Table of Contents

Supported shells and platforms

Platform Test
Linux (Debian, Ubuntu, Fedora, CentOS, Alpine, Busybox, OpenWrt) GitHub Actions or Docker
macOS (Default installed shells, Homebrew) GitHub Actions
Windows (Git bash, msys2, cygwin, busybox-w32, WSL) GitHub Actions
BSD (FreeBSD, OpenBSD, NetBSD) Cirrus CI (FreeBSD) or Manual (Others)
Unix (Solaris, AIX) Manual only

Tested version details

Requirements

POSIX-compliant commands

ShellSpec uses shell built-in commands and only few basic POSIX-compliant commands to support wide range of environments (except kcov for optional code coverage).

Currently used external (not shell builtins) commands:

  • cat, date, env, ls, mkdir, od (or not POSIX hexdump), rm, sleep, sort, time
  • ps (use to auto-detect shells in environments that don't implement procfs)
  • ln, mv (use only when generating coverage report)
  • kill, printf (most shells except some are built-in)

Installation

Web installer (for developers)

Install the latest release version 

curl -fsSL https://git.io/shellspec | sh

or

wget -O- https://git.io/shellspec | sh

NOTE: https://git.io/shellspec is redirected to install.sh

The installation using the web installer is mainly intended for development use. For CI, it is recommended to use a specific version (tag) in git or archives to avoid unexpected failures.

Advanced installation / upgrade

Automatic installation 

curl -fsSL https://git.io/shellspec | sh -s -- --yes

Install the specified version 

curl -fsSL https://git.io/shellspec | sh -s 0.19.1

Upgrade to the latest release version 

curl -fsSL https://git.io/shellspec | sh -s -- --switch

Switch to the specified version 

curl -fsSL https://git.io/shellspec | sh -s 0.18.0 --switch
Uninstall

How to uninstall

  1. Delete the ShellSpec executable file [default: $HOME/.local/bin/shellspec].
  2. Delete the ShellSpec installation directory [default: $HOME/.local/lib/shellspec].
Other usage

Other usage 

$ curl -fsSL https://git.io/shellspec | sh -s -- --help
Usage: [sudo] ./install.sh [VERSION] [OPTIONS...]
  or : wget -O- https://git.io/shellspec | [sudo] sh
  or : wget -O- https://git.io/shellspec | [sudo] sh -s -- [OPTIONS...]
  or : wget -O- https://git.io/shellspec | [sudo] sh -s VERSION [OPTIONS...]
  or : curl -fsSL https://git.io/shellspec | [sudo] sh
  or : curl -fsSL https://git.io/shellspec | [sudo] sh -s -- [OPTIONS...]
  or : curl -fsSL https://git.io/shellspec | [sudo] sh -s VERSION [OPTIONS...]

VERSION:
  Specify install version and method

  e.g
    1.0.0           Install 1.0.0 from git
    master          Install master from git
    1.0.0.tar.gz    Install 1.0.0 from tar.gz archive
    .               Install from local directory

OPTIONS:
  -p, --prefix PREFIX   Specify prefix                 [default: $HOME/.local]
  -b, --bin BIN         Specify bin directory          [default: <PREFIX>/bin]
  -d, --dir DIR         Specify installation directory [default: <PREFIX>/lib/shellspec]
  -s, --switch          Switch version (requires installation via git)
  -l, --list            List available versions (tags)
      --pre             Include pre-release
      --fetch FETCH     Force command to use when installing from archive (curl or wget)
  -y, --yes             Automatic yes to prompts
  -h, --help            You're looking at it

Package manager

Arch Linux

Installation on Arch Linux from the AUR ShellSpec package using aura:

# Install the latest stable version
$ aura -A shellspec
Homebrew / Linuxbrew 
# Install the latest stable version
$ brew tap shellspec/shellspec
$ brew install shellspec
basher / bpkg

Installation with basher

The officially supported version is ShellSpec 0.19.1 and later.

# Install from master branch
$ basher install shellspec/shellspec

# To specify a version (example: 0.19.1)
$ basher install shellspec/[email protected]

Installation with bpkg

The officially supported version is ShellSpec 0.19.1 and later.

# Install from master branch
$ bpkg install shellspec/shellspec

# To specify a version (example: 0.19.1)
$ bpkg install shellspec/[email protected]

Manual installation

git / archive (source code)

Download from git or archive and create a symbolic link.

From git

$ cd /SOME/WHERE/TO/INSTALL
$ git clone https://github.com/shellspec/shellspec.git

$ ln -s /SOME/WHERE/TO/INSTALL/shellspec/shellspec /EXECUTABLE/PATH/

From archive

$ cd /SOME/WHERE/TO/INSTALL
$ wget https://github.com/shellspec/shellspec/archive/{VERSION}.tar.gz
$ tar xzvf shellspec-{VERSION}.tar.gz

$ ln -s /SOME/WHERE/TO/INSTALL/shellspec-{VERSION}/shellspec /EXECUTABLE/PATH/

Executable path: e.g. /usr/local/bin/, $HOME/bin/

Use make instead of symbolic link creation

Download from git or archive and use make command.

How to install.

Install to /usr/local/bin and /usr/local/lib

sudo make install

Install to $HOME/bin and $HOME/lib

make install PREFIX=$HOME

How to uninstall.

sudo make uninstall
make uninstall PREFIX=$HOME
For environments that do not support symbolic links

Download from git or archive and create the following shellspec file instead of the symbolic link.

$ cat<<'HERE'>/EXECUTABLE/PATH/shellspec
#!/bin/sh
exec /SOME/WHERE/TO/INSTALL/shellspec/shellspec "$@"
HERE
$ chmod +x /EXECUTABLE/PATH/shellspec

Distribution archive (runtime only)

See Releases page if you want to download distribution archive.

Tutorial

Just create your project directory and run shellspec --init to setup your project

# Create your project directory, for example "hello".
$ mkdir hello
$ cd hello

# Initialize
$ shellspec --init
  create   .shellspec
  create   spec/spec_helper.sh

# Write your first specfile (of course you can use your favorite editor)
$ cat<<'HERE'>spec/hello_spec.sh
Describe 'hello.sh'
  Include lib/hello.sh
  It 'says hello'
    When call hello ShellSpec
    The output should equal 'Hello ShellSpec!'
  End
End
HERE

# Create lib/hello.sh
$ mkdir lib
$ touch lib/hello.sh

# It will fail because the hello function is not implemented.
$ shellspec

# Write hello function
$ cat<<'HERE'>lib/hello.sh
hello() {
  echo "Hello ${1}!"
}
HERE

# It will success!
$ shellspec

ShellSpec CLI

runs specfile using /bin/sh by default

ShellSpec CLI runs specfiles with the shell running shellspec. Usually it is /bin/sh that is the shebang of shellspec. If you run bash shellspec, it will be bash. Include files from specfile will be executed in the same shell as well.

The purpose of this specification is to allow ShellSpec to easily change multiple types of shells and enable the development of cross-platform shell scripts that support multiple shells and environments.

If you want to test with a specific shell, use the -s (--shell) option. You can specify the default shell in the .shellspec file.

NOTE: If you execute a shell script file (not a shell function) from within the specfile, its shebang will be respected. Because in that case, it will be run as an external command. The -s (--shell) option also has no effect. If you are testing a external shell script file, you can use When run script or When run source. These ignore the shebang of external shell script file and run in the same shell that runs specfile.

command options

NOTE: Since version 0.28.0, getoptions is used to parse options, so all POSIX and GNU compatible option syntax can be used. For example, you can abbreviate a long option.

See more info: ShellSpec CLI

$ shellspec -h
Usage: shellspec [ -c ] [-C <directory>] [options...] [files or directories...]

  Using + instead of - for short options causes reverses the meaning

    -s, --shell SHELL               Specify a path of shell [default: "auto" (the shell running shellspec)]
        --require MODULE            Require a MODULE (shell script file)
    -O, --options PATH              Specify the path to an additional options file
    -I, --load-path PATH            Specify PATH to add to $SHELLSPEC_LOAD_PATH (may be used more than once)
        --helperdir DIRECTORY       The directory to load helper files (spec_helper.sh, etc) [default: "spec"]
        --path PATH                 Set PATH environment variable at startup
        --{no-}sandbox              Force the use of the mock instead of the actual command
        --sandbox-path PATH         Make PATH the sandbox path instead of empty [default: empty]
        --execdir @LOCATION[/DIR]   Specify the execution directory of each specfile | [default: @project]
    -e, --env NAME[=VALUE]          Set environment variable
        --env-from ENV-SCRIPT       Set environment variable from shell script file
    -w, --{no-}warning-as-failure   Treat warning as failure [default: enabled]
        --{no-}fail-fast[=COUNT]    Abort the run after first (or COUNT) of failures [default: disabled]
        --{no-}fail-no-examples     Fail if no examples found [default: disabled]
        --{no-}fail-low-coverage    Fail on low coverage [default: disabled]
        --failure-exit-code CODE    Override the exit code used when there are failing specs [default: 101]
        --error-exit-code CODE      Override the exit code used when there are fatal errors [default: 102]
    -p, --{no-}profile              Enable profiling and list the slowest examples [default: disabled]
        --profile-limit N           List the top N slowest examples [default: 10]
        --{no-}boost                Increase the CPU frequency to boost up testing speed [default: disabled]
        --log-file LOGFILE          Log file for %logger directive and trace [default: "/dev/tty"]
        --tmpdir TMPDIR             Specify temporary directory [default: $TMPDIR, $TMP or "/tmp"]
        --keep-tmpdir               Do not cleanup temporary directory [default: disabled]

  The following options must be specified before other options and cannot be specified in the options file

    -c, --chdir                     Change the current directory to the first path of arguments at the start
    -C, --directory DIRECTORY       Change the current directory at the start

  **** Execution ****

    -q, --{no-}quick                Run not-passed examples if it exists, otherwise run all [default: disabled]
    -r, --repair, --only-failures   Run failure examples only (Depends on quick mode)
    -n, --next-failure              Run failure examples and abort on first failure (Depends on quick mode)
    -j, --jobs JOBS                 Number of parallel jobs to run [default: 0 (disabled)]
        --random TYPE[:SEED]        Run examples by the specified random type | <[none]> [specfiles] [examples]
    -x, --xtrace                    Run examples with trace output of evaluation enabled [default: disabled]
    -X, --xtrace-only               Run examples with trace output only enabled [default: disabled]
        --dry-run                   Print the formatter output without running any examples [default: disabled]

  **** Output ****

        --{no-}banner               Show banner if exist "<HELPERDIR>/banner[.md]" [default: enabled]
        --reportdir DIRECTORY       Output directory of the report [default: "report"]
    -f, --format FORMATTER          Choose a formatter for display | <[p]> [d] [t] [j] [f] [null] [debug]
    -o, --output FORMATTER          Choose a generator(s) to generate a report file(s) [default: none]
        --{no-}color                Enable or disable color [default: enabled if the output is a TTY]
        --skip-message VERBOSITY    Mute skip message | <[verbose]> [moderate] [quiet]
        --pending-message VERBOSITY Mute pending message | <[verbose]> [quiet]
        --quiet                     Equivalent of --skip-message quiet --pending-message quiet
        --(show|hide)-deprecations  Show or hide deprecations details [default: show]

  **** Ranges / Filters / Focus ****

    You can run selected examples by specified the line numbers or ids

      shellspec path/to/a_spec.sh:10   # Run the groups or examples that includes lines 10
      shellspec path/to/a_spec.sh:@1-5 # Run the 5th groups/examples defined in the 1st group
      shellspec a_spec.sh:10:@1:20:@2  # You can mixing multiple line numbers and ids with join by ":"

    -F, --focus                     Run focused groups / examples only
    -P, --pattern PATTERN           Load files matching pattern [default: "*_spec.sh"]
    -E, --example PATTERN           Run examples whose names include PATTERN
    -T, --tag TAG[:VALUE]           Run examples with the specified TAG
        --default-path PATH         Set the default path where looks for examples [default: "spec"]

    You can specify the path recursively by prefixing it with the pattern "*/" or "**/"
      (This is not glob patterns and requires quotes. It is also available with --default-path)

      shellspec "*/spec"               # The pattern "*/" matches 1 directory
      shellspec "**/spec"              # The pattern "**/" matches 0 and more directories
      shellspec "*/*/**/test_spec.sh"  # These patterns can be specified multiple times

    -L, --dereference               Dereference all symlinks in in the above pattern [default: disabled]

  **** Coverage ****

        --covdir DIRECTORY          Output directory of the Coverage Report [default: coverage]
        --{no-}kcov                 Enable coverage using kcov [default: disabled]
        --kcov-path PATH            Specify kcov path [default: kcov]
        --kcov-options OPTIONS      Additional Kcov options (coverage limits, coveralls id, etc)

  **** Utility ****

        --init [TEMPLATE...]        Initialize your project with ShellSpec | [spec] [git] [hg] [svn]
        --gen-bin [@COMMAND...]     Generate test support commands in "<HELPERDIR>/support/bin"
        --count                     Count the number of specfiles and examples
        --list LIST                 List the specfiles/examples | [specfiles] [examples(:id|:lineno)]
        --syntax-check              Syntax check of the specfiles without running any examples
        --translate                 Output translated specfile
        --task [TASK]               Run the TASK or Show the task list if TASK is not specified
        --docker DOCKER-IMAGE       Run tests in specified docker image (EXPERIMENTAL)
    -v, --version                   Display the version
    -h, --help                      -h: short help, --help: long help

Project directory

All specfiles for ShellSpec must be under the project directory. The root of the project directory must have a .shellspec file. This file is that specify the default options to be used in the project, but an empty file is required even if the project has no options.

NOTE: The .shellspec file was described in the documentation as a required file for some time, but ShellSpec worked without it. Starting with version 0.28.0, this file is checked and will be required in future versions.

You can easily create the necessary files by executing the shellspec --init command in an existing directory.

Typical directory structure

This is the typical directory structure. Version 0.28.0 allows many of these to be changed by specifying options, supporting a more flexible directory structure.

<PROJECT-ROOT> directory
├─ .shellspec                       [mandatory]
├─ .shellspec-local                 [optional] Ignore from version control
├─ .shellspec-quick.log             [optional] Ignore from version control
├─ report/                          [optional] Ignore from version control
├─ coverage/                        [optional] Ignore from version control
│
├─ bin/
│   ├─ your_script1.sh
│              :
├─ lib/
│   ├─ your_library1.sh
│              :
│
├─ spec/ (also <HELPERDIR>)
│   ├─ spec_helper.sh               [recommended]
│   ├─ banner[.md]                  [optional]
│   ├─ support/                     [optional]
│   │
│   ├─ bin/
│   │   ├─ your_script1_spec.sh
│   │             :
│   ├─ lib/
│   │   ├─ your_library1_spec.sh

Options file

To change the default options for the shellspec command, create options file(s). Files are read in the order shown below, options defined last take precedence.

  1. $XDG_CONFIG_HOME/shellspec/options
  2. $HOME/.shellspec-options (version >= 0.28.0) or $HOME/.shellspec (deprecated)
  3. <PROJECT-ROOT>/.shellspec
  4. <PROJECT-ROOT>/.shellspec-local (Do not store in VCS such as git)

Specify your default options with $XDG_CONFIG_HOME/shellspec/options or $HOME/.shellspec-options. Specify default project options with .shellspec and overwrite to your favorites with .shellspec-local.

.shellspec - project options file

Specifies the default options to use for the project.

.shellspec-local - user custom options file

Override the default options used by the project with your favorites.

.shellspec-basedir - specfile execution base directory

Used to specify the directory in which the specfile will be run. See directory structure or --execdir option for details.

.shellspec-quick.log - quick execution log

If this file is present, Quick mode will be enabled and the log of Quick execution will be recorded. It created automatically when --quick option is specified. If you want to turn off Quick mode, delete it.

report/ - report file directory

The output location for reports generated by the --output or --profile options. This can be changed with the --reportdir option.

coverage/ - coverage reports directory

The output location for coverage reports. This can be changed with the --covdir option.

spec/ - (default) specfiles directory

By default, it is assumed that all specfiles are store under the spec directory, but it is possible to create multiple directories with different names.

NOTE: In Version <= 0.27.x, the spec directory was the only directory that contained the specfiles.

<HELPERDIR> (default: spec/)

The directory to store spec_helper.sh and other files. By default, the spec directory also serves as HELPERDIR directory, but you can change it to another directory with the --helperdir option.

spec_helper.sh - (default) helper file for specfile

The spec_helper.sh is loaded to specfile by the --require spec_helper option. This file is used to define global functions, initial setting for examples, custom matchers, etc.

banner[.md] - banner file displayed at test execution

If the file <HELPERDIR>/banner or <HELPERDIR>/banner.md exists, Display a banner when the shellspec command is executed. It can be used to display information about the tests. The --no-banner option can be used to disable this behavior.

support/ - directory for support files

This directory can be used to store files such as custom matchers and tasks.

bin - directory for support commands

This directory is used to store support commands.

Specfile (test file)

In ShellSpec, you write your tests in a specfile. By default, specfile is a file ending with _spec.sh under the spec directory.

The specfile is executed using the shellspec command, but it can also be executed directly. See self-executable specfile for details.

Example

Describe 'lib.sh' # example group
  Describe 'bc command'
    add() { echo "$1 + $2" | bc; }

    It 'performs addition' # example
      When call add 2 3 # evaluation
      The output should eq 5  # expectation
    End
  End
End

The best place to learn how to write a specfile is the examples/spec directory. You should take a look at it! (Those examples include failure examples on purpose.)

About DSL

ShellSpec has its own DSL to write tests. It may seem like a distinctive code because DSL starts with a capital letter, but the syntax is compatible with shell scripts, and you can embed shell functions and use ShellCheck to check the syntax.

You may feel rejected by this DSL, but It starts with a capital letter to avoid confusion with the command, and it does a lot more than you think, such as realizing scopes, getting shell-independent line numbers, and workarounds for bugs in some shells.

Execution directory

Since version 0.28.0, the working directory when running a specfile is the project root directory by default. Even if you run a specfile from a subdirectory in the project directory, it is the project root directory. Before 0.27.x, it was the current directory when the shellspec command was executed.

You can change this directory (location) by using the --execdir @LOCATION[/DIR] option. You can choose from the following locations and specify a path relative to the location if necessary. However, you cannot specify a directory outside the project directory.

  • @project Where the .shellspec file is located (project root) [default]
  • @basedir Where the .shellspec or .shellspec-basedir file is located
  • @specfile Where the specfile is located

If @basedir is specified, the parent directory is searched from the directory containing the specfile to be run, and the first directory where .shellspec-basedir or .shellspec is found is used as the execution directory. This is useful if you want to have a separate directory for each utility (command) you want to test.

NOTE: You will need to change under the project directory or use the -c (--chdir) or -C (--directory) option before running the specfile.

Embedded shell scripts

You can embed shell functions (or shell script code) in the specfile. This shell function can be used for test preparation and complex testing.

Note that the specfile implements scope using subshells. Shell functions defined in the specfile can only be used within blocks (e.g. Describe, It, etc).

If you want to use a global function, you can define it in spec_helper.sh.

Translation process

The specfile will not be executed directly by the shell, but will be translated into a regular shell script and output to a temporary directory (default: /tmp) before being executed.

The translation process is simple in that it only replaces forward-matched words (DSLs), with a few exceptions. If you are interested in the translated code, you can see with shellspec --translate.

Syntax formatter (altshfmt)

The specfile contains DSLs, so it cannot be properly formatted by general shell script formatters. If you want to format the specfile, use altshfmt.

DSL syntax

Basic structure

Describe, Context, ExampleGroup - example group block

ExampleGroup is a block for grouping example groups or examples. Describe and Context are aliases for ExampleGroup. They can be nested, and may contain example groups or examples.

Describe 'is example group'
  Describe 'is nestable'
    ...
  End

  Context 'is used to facilitate understanding depending on the context'
    ...
  End
End

The example groups can be optionally tagged. See Tagging for details.

Describe 'is example group' tag1:value1 tag2:value2 ...

It, Specify, Example - example block

Example is a block for writing an evaluation and expectations. It and Specify are aliases for Example.

An example is composed by up to one evaluation and multiple expectations.

add() { echo "$1 + $2" | bc; }

It 'performs addition'          # example
  When call add 2 3             # evaluation
  The output should eq 5        # expectation
  The status should be success  # another expectation
End

The examples can be optionally tagged. See Tagging for details.

It 'performs addition' tag1:value1 tag2:value2 ...

Todo - one liner empty example

Todo is the same as the empty example and is treated as a pending example.

Todo 'will be used later when we write a test'

It 'is an empty example, the same as Todo'
End

When - evaluation

Evaluation executes a shell function or command for verification. Only one evaluation can be defined for each example and also can be omitted.

See more details of Evaluation

NOTE: About executing aliases

call - call a shell function (without subshell)

It calls a function without subshell. Practically, it can also run commands.

When call add 1 2 # call `add` shell function with two arguments.
run - run a command (within subshell)

It runs a command within subshell. Practically, it can also call a shell function. The command does not have to be a shell script.

NOTE: This does not support coverage measurement.

When run touch /tmp/foo # run `touch` command.

Some commands below are specially handled by ShellSpec.

command - runs an external command

It runs a command, respecting shebang. It can not call a shell function. The command does not have to be a shell script.

NOTE: This does not support coverage measurement.

When run command touch /tmp/foo # run `touch` command.
script - runs a shell script

It runs a shell script, ignoring shebang. The script has to be a shell script. It will be executed in another instance of the same shell as the current shell.

When run script my.sh # run `my.sh` script.
source - runs a script by . (dot) command

It sources a shell script, ignoring its shebang. The script has to be a shell script. It is similar to run script, but with some differences. Unlike run script, function-based mocking is available.

When run source my.sh # source `my.sh` script.
About executing aliases

If you want to execute aliases, you need a workaround using eval.

alias alias-name='echo this is alias'
When call alias-name # alias-name: not found

# eval is required
When call eval alias-name

# When using embedded shell scripts
foo() { eval alias-name; }
When call foo

The - expectation

Expectation begins with The which does the verification. The basic syntax is as follows:

The output should equal 4

Use should not for the opposite verification.

The output should not equal 4
Subjects

The subject is the target of the verification.

The output should equal 4
      |
      +-- subject

There are output (stdout), error (stdout), status, variable, path, etc. subjects.

Please refer to the Subjects for more details.

Modifiers

The modifier concretizes the target of the verification (subject).

The line 2 of output should equal 4
      |
      +-- modifier

The modifiers are chainable.

The word 1 of line 2 of output should equal 4

If the modifier argument is a number, you can use an ordinal numeral instead of a number.

The first word of second line of output should equal 4

There are line, word, length, contents, result, etc. modifiers. The result modifier is useful for making the result of a user-defined function the subject.

Please refer to the Modifiers for more details.

Matchers

The matcher is the verification.

The output should equal 4
                   |
                   +-- matcher

There are many matchers such as string matcher, status matcher, variable matchers and stat matchers. The satisfy matcher is useful for verification with user-defined functions.

Please refer to the Matchers for more details.

Language chains

ShellSpec supports language chains like chai.js. It only improves readability, does not affect the expectation: a, an, as, the.

The following two sentences have the same meaning:

The first word of second line of output should valid number

The first word of the second line of output should valid as a number

Assert - expectation for custom assertion

The Assert is yet another expectation to verify with a user-defined function. It is designed for verification of side effects, not the result of the evaluation.

still_alive() {
  ping -c1 "$1" >/dev/null
}

Describe "example.com"
  It "responses"
    Assert still_alive "example.com"
  End
End

Pending, skip and focus

Pending - pending example

Pending is similar to Skip, but the test passes if the verification fails, and the test fails if the verification succeeds. This is useful if you want to specify that you will implement something later.

Describe 'Pending'
  Pending "not implemented"

  hello() { :; }

  It 'will success when test fails'
    When call hello world
    The output should "Hello world"
  End
End

Skip - skip example

Use Skip to skip executing the example.

Describe 'Skip'
  Skip "not exists bc"

  It 'is always skip'
    ...
  End
End
if - conditional skip

Use Skip if if you want to skip conditionally.

Describe 'Conditional skip'
  not_exists_bc() { ! type bc >/dev/null 2>&1; }
  Skip if "not exists bc" not_exists_bc

  add() { echo "$1 + $2" | bc; }

  It 'performs addition'
    When call add 2 3
    The output should eq 5
  End
End

'x' prefix for example group and example

xDescribe, xContext, xExampleGroup - skipped example group

xDescribe, xContext, xExampleGroup are skipped example group blocks. Execution of examples contained in these blocks is skipped.

Describe 'is example group'
  xDescribe 'is skipped example group'
    ...
  End
End
xIt, xSpecify, xExample - skipped example

xIt, xSpecify, xExample are skipped example blocks. Execution of the example is skipped.

xIt 'is skipped example'
  ...
End

'f' prefix for example group and example

fDescribe, fContext, fExampleGroup - focused example group

fDescribe, fContext, fExampleGroup are focused example group blocks. Only the examples included in these will be executed when the --focus option is specified.

Describe 'is example group'
  fDescribe 'is focus example group'
    ...
  End
End
fIt, fSpecify, fExample - focused example

fIt, fSpecify, fExample are focused example blocks. Only these examples will be executed when the --focus option is specified.

fIt 'is focused example'
  ...
End

About temporary pending and skip

Using Pending or Skip without a message is a "temporary pending" or "temporary skip". "x"-prefixed example groups and examples are also treated as temporary skips.

The non-temporary Pending and Skip (with a message) are used when the case will take a long time to resolve. It may be committed to a version control system. Temporary pending and skip are used during current work. We do not recommend committing them to a version control system.

These two types differ in the display of the report. Refer to --skip-message and --pending-message options.

# Temporary pending and skip
Pending
Skip
Skip # this comment will be displayed in the report
Todo
xIt
  ...
End

# Non-temporary pending and skip
Pending "reason"
Skip "reason"
Skip if "reason" condition
Todo "It will be implemented"

Hooks

BeforeEach (Before), AfterEach (After) - example hook

You can specify commands to be executed before / after each example by BeforeEach (Before), AfterEach (After).

NOTE: BeforeEach and AfterEach are supported in version 0.28.0 and later. Previous versions should use Before and After instead.

NOTE: AfterEach is for cleanup and not for assertions.

Describe 'example hook'
  setup() { :; }
  cleanup() { :; }
  BeforeEach 'setup'
  AfterEach 'cleanup'

  It 'is called before and after each example'
    ...
  End

  It 'is called before and after each example'
    ...
  End
End

BeforeAll, AfterAll - example group hook

You can specify commands to be executed before / after all examples by BeforeAll and AfterAll.

Describe 'example all hook'
  setup() { :; }
  cleanup() { :; }
  BeforeAll 'setup'
  AfterAll 'cleanup'

  It 'is called before/after all example'
    ...
  End

  It 'is called before/after all example'
    ...
  End
End

BeforeCall, AfterCall - call evaluation hook

You can specify commands to be executed before / after call evaluation by BeforeCall and AfterCall.

NOTE: These hooks were originally created to test ShellSpec itself. Please use the BeforeEach / AfterEach hooks whenever possible.

Describe 'call evaluation hook'
  setup() { :; }
  cleanup() { :; }
  BeforeCall 'setup'
  AfterCall 'cleanup'

  It 'is called before/after call evaluation'
    When call hello world
    ...
  End
End

BeforeRun, AfterRun - run evaluation hook

You can specify commands to be executed before / after run evaluation (run, run command, run script, and run source) by BeforeRun and AfterRun.

These hooks are executed in the same subshell as the "run evaluation". Therefore, you can access the variables after executing the evaluation.

NOTE: These hooks were originally created to test ShellSpec itself. Please use the BeforeEach / AfterEach hooks whenever possible.

Describe 'run evaluation hook'
  setup() { :; }
  cleanup() { :; }
  BeforeRun 'setup'
  AfterRun 'cleanup'

  It 'is called before/after run evaluation'
    When run hello world
    ...
  End
End

Pitfalls

The hooks may fail in subtle ways if there is output to stderr, even if the return code / exit code is 0.

Commands like git checkout routinely write to stderr, even if there was no actual failure, so be aware that your hooks may fail because of this.

Helpers

Dump - dump stdout, stderr, and status for debugging

Dump stdout, stderr, and status of the evaluation. It is useful for debugging.

When call echo hello world
Dump # stdout, stderr and status

Include - include a script file

Include a shell script to test.

Describe 'lib.sh'
  Include lib.sh # hello function defined

  Describe 'hello()'
    It 'says hello'
      When call hello ShellSpec
      The output should equal 'Hello ShellSpec!'
    End
  End
End

Set - set shell options

Set shell options before executing each example. The shell option name is the long name of set or the name of shopt:

NOTE: Use Set instead of the set command because the set command may not work as expected in some shells.

Describe 'Set helper'
  Set 'errexit:off' 'noglob:on'

  It 'sets shell options before executing the example'
    When call foo
  End
End

Path, File, Dir - path alias

Path is used to define a short pathname alias. File and Dir are aliases for Path.

Describe 'Path helper'
  Path hosts-file="/etc/hosts"

  It 'defines short alias for long path'
    The path hosts-file should exist
  End
End

Data - pass data as stdin to evaluation

You can use the Data Helper which inputs data from stdin for evaluation. The input data is specified after #| in the Data or Data:expand block.

Describe 'Data helper'
  It 'provides with Data helper block style'
    Data # Use Data:expand instead if you want expand variables.
      #|item1 123
      #|item2 456
      #|item3 789
    End
    When call awk '{total+=$2} END{print total}'
    The output should eq 1368
  End
End

You can also use a file, function or string as data sources.

See more details of Data

Parameters - parameterized example

Parameterized tests (aka Data-driven testing) are used to run the same test once for each set of parameters. Parameters defines a block of parameters.

Describe 'example'
  Parameters
    "#1" 1 2 3
    "#2" 1 2 3
  End

  Example "example $1"
    When call echo "$(($2 + $3))"
    The output should eq "$4"
  End
End

In addition to the default Parameters block, three additional styles are supported: Parameters:value, Parameters:matrix and Parameters:dynamic.

See more details of Parameters

NOTE: You can also combine the Parameters and Data:expand helpers.

Mock - create a command-based mock

See Command-based mock

Intercept - create an intercept point

See Intercept

Directives

Directives are instructions that can be used in embedded shell scripts. It is used to solve small problems of shell scripts in testing.

This is like a shell function, but not a shell function. Therefore, the supported grammar is limited and can only be used at the beginning of a function definition or at the beginning of a line.

foo() { %puts "foo"; } # supported

bar() {
  %puts "bar" # supported
}

baz() {
  any command; %puts "baz" # not supported
}

%const (%) - constant definition

%const (% is short hand) directive defines a constant value. The characters which can be used for variable names are uppercase letters [A-Z], digits [0-9] and underscore _ only. It can not be defined inside an example group nor an example.

The value is evaluated during the specfile translation process. So you can access ShellSpec variables, but you can not access variable or function in the specfile.

This feature assumes use with conditional skip. The conditional skip may run outside of the examples. As a result, sometimes you may need variables defined outside of the examples.

%text - embedded text

You can use the %text directive instead of a hard-to-use heredoc with indented code. The input data is specified after #|.

Describe '%text directive'
  It 'outputs texts'
    output() {
      echo "start" # you can write code here
      %text
      #|aaa
      #|bbb
      #|ccc
      echo "end" # you can write code here
    }

    result() { %text
      #|start
      #|aaa
      #|bbb
      #|ccc
      #|end
    }

    When call output
    The output should eq "$(result)"
    The line 3 of output should eq 'bbb'
  End
End

%puts (%-), %putsn (%=) - output a string (with newline)

%puts (put string) and %putsn (put string with newline) can be used instead of (not portable) echo. Unlike echo, it does not interpret escape sequences regardless of the shell. %- is an alias of %puts, %= is an alias of %putsn.

%printf - alias for printf

This is the same as printf, but it can be used in the sandbox mode because the path has been resolved.

%sleep - alias for sleep

This is the same as sleep, but it can be used in the sandbox mode because the path has been resolved.

%preserve - preserve variables

Use the %preserve directive to preserve the variables in subshells and external shell scripts.

In the following cases, %preserve is required because variables are not preserved.

  • When run evaluation - It runs in a subshell.
  • Command-based mock (Mock) - It is an external shell script.
  • Function-based Mock called by command substitution
Describe '%preserve directive'
  It 'preserves variables'
    func() { foo=1; bar=2; baz=3; }
    preserve() { %preserve bar baz:BAZ; }
    AfterRun preserve

    When run func
    The variable foo should eq 1 # This will be failure
    The variable bar should eq 2 # This will be success
    The variable BAZ should eq 3 # Preserved to different variable (baz:BAZ)
  End
End

%logger - debug output

Output log messages to the log file (default: /dev/tty) for debugging.

%data - define parameter

See Parameters:dynamic.

Mocking

There are two ways to create a mock, (shell) function-based mock and (external) command-based mock. The function-based mock is usually recommended for performance reasons. Both can be overwritten with an internal block and will be restored when the block ends.

Function-based mock

The (shell) function-based mock is simply (re)defined with a shell function.

Describe 'function-based mock'
  get_next_day() { echo $(($(date +%s) + 86400)); }

  date() {
    echo 1546268400
  }

  It 'calls the date function'
    When call get_next_day
    The stdout should eq 1546354800
  End
End

Command-based mock

The (external) command-based mock creates a temporary mock shell script and runs as an external command. This is slow, but there are some advantages over the function-based mock.

  • Can be use invalid characters as the shell function name.
    • e.g. docker-compose (- cannot be used as a function name in POSIX)
  • Can be invoke a mocked command from an external command (not limited to shell script).

A command-based mock creates an external shell script with the contents of a Mock block, so there are some restrictions.

  • It is not possible to mock shell functions or shell built-in functions.
  • It is not possible to call shell functions outside the Mock block.
    • Exception: Can be called exported (export -f) functions. (bash only)
  • To reference variables outside the Mock block, they must be exported.
  • To return a variable from a Mock block, you need to use the %preserve directive.
Describe 'command-based mock'
  get_next_day() { echo $(($(date +%s) + 86400)); }

  Mock date
    echo 1546268400
  End

  It 'runs the mocked date command'
    When call get_next_day
    The stdout should eq 1546354800
  End
End

NOTE: To achieve this feature, a directory for mock commands is included at the beginning of the PATH.

Support commands

Execute the actual command within a mock function

Support commands are helper commands that can be used in the specfile. For example, it can be used in a mock function to execute the actual command. It is recommended that the support command name be the actual command name prefixed with @.

Describe "Support commands example"
  touch() {
    @touch "$@" # @touch executes actual touch command
    echo "$1 was touched"
  }

  It "touch a file"
    When run touch "file"
    The output should eq "file was touched"
    The file "file" should exist
  End
End

Support commands are generated in the spec/support/bin directory by the --gen-bin option. For example run shellspec --gen-bin @touch to generate the @touch command.

This is the main purpose, but support commands are just shell scripts, so they can also be used for other purposes. You can freely edit the support command script.

Make mock not mandatory in sandbox mode

The sandbox mode forces the use of mocks. However, you may not want to require mocks for some commands. For example, printf is a built-in command in many shells and does not require a mock in the sandbox mode for these shells. But there are shells where it is an external command and then it requires to be mocked.

To allow printf to be called without mocking in certain cases, create a support command named printf (shellspec --gen-bin printf).

Resolve command incompatibilities

Some commands have different options between BSD and GNU. If you handle the difference in the specfile, the test will be hard to read. You can solve it with the support command.

#!/bin/sh -e
# Command name: @sed
. "$SHELLSPEC_SUPPORT_BIN"
case $OSTYPE in
  *darwin*) invoke gsed "$@" ;;
  *) invoke sed "$@" ;;
esac

Tagging

The example groups or examples can be tagged, and the --tag option can be used to filter the examples to be run. The tag name and tag value are separated by :, and the tag value is optional. You can use any character if quoted.

Describe "Checking something" someTag:someVal
  It "does foo" tagA:val1
    ...
  It "does bar" tagA:val2
    ...
  It "does baz" tagA
    ...
End
  1. Everything nested inside a selected element is selected in parent elements. e.g. --tag someTag will select everything above.
  2. Specifying a tag but no value selects everything with that tag whether or not it has a value, e.g. --tag tagA will select everything above.
  3. Specifying multiple tags will select the union of everything tagged, e.g. --tag tagA:val1,tagA:val2 will select does foo and does bar.
  4. Tests included multiple times are not a problem, e.g. --tag someTag,tagA,tagA:val1 just selects everything.
  5. If no tag matches, nothing will be run, e.g. --tag tagA: runs nothing (it does not match baz above, as empty values are not the same as no value).
  6. The --tag option can be used multiple times, e.g. --tag tagA:val1 --tag tagA:val2 works the same as --tag tagA:val1,tagA:val2

About testing external commands

ShellSpec is a testing framework for shell scripts, but it can be used to test anything that can be executed as an external command, even if it is written in another language. Even shell scripts can be tested as external commands.

If you are testing a shell script as an external command, please note the following.

  • It will be executed in the shell specified by the shebang not the shell running the specfile.
  • The coverage of the shell script will not be measured.
  • Cannot refer to variables inside the shell script.
  • Shell built-in commands cannot be mocked.
  • Functions defined inside the shell script cannot be mocked.
  • Only command-based mock can be used (if the script is calling an external command).
  • Interceptor is not available.

To get around these limitations, use run script or run source. See How to test a single file shell script.

How to test a single file shell script

If the shell script consists of a single file, unit testing becomes difficult. However, there are many such shell scripts.

ShellSpec has the ability to testing in such cases with only few modifications to the shell script.

Using run script

Unlike the case of executing as an external command, it has the following features.

  • It will run in the same shell (but another process) that is running specfile.
  • The coverage of the shell script will be measured.

There are limitations as follows.

  • Cannot refer to variables inside the shell script.
  • Shell built-in commands cannot be mocked.
  • Functions defined inside the shell script cannot be mocked.
  • Only command-based mock can be used (if the script is calling an external command).
  • Interceptor is not available.

Using run source

It is even less limitations than run script and has the following features.

  • It will run in the same shell and same process that is running specfile.
  • The coverage of the shell script will be measured.
  • Can be refer to variables inside the shell script.
  • Function-based mock and command-based mock are available.
  • Interceptor is available.
  • Shell built-in commands can be mocked.
  • Functions defined inside the shell script can be mocked using interceptor.

However, since it is simulated using the . command, there are some differences in behavior. For example, the value of $0 is different.

NOTE: Mocking of shell built-in commands can be done before run source. However, if you are using interceptor, mocking of the test command must be done in the __<name>__ function.

Testing shell functions

__SOURCED__

This is the way to test shell functions defined in a shell script.

Loading a script with Include defines a __SOURCED__ variable available in the sourced script. If the variable __SOURCED__ is defined, please return from the shell script.

#!/bin/sh
# hello.sh

hello() { echo "Hello $1"; }

# This is the writing style presented by ShellSpec, which is short but unfamiliar.
# Note that it returns the current exit status (could be non-zero).
${__SOURCED__:+return}

# The above means the same as below.
# ${__SOURCED__:+x} && return $?

# If you don't like the coding style, you can use the general writing style.
# if [ "${__SOURCED__:+x}" ]; then
#   return 0
# fi

hello "$1"
Describe "hello.sh"
  Include "./hello.sh"

  Describe "hello()"
    It "says hello"
      When call hello world
      The output should eq "Hello world"
    End
  End
End

Intercepting

Interceptor is a feature that allows you to intercept your shell script in the middle of its execution. This makes it possible to mock functions that cannot be mocked in advance at arbitrary timing, and to make assertions by retrieving the state of during script execution.

It is a powerful feature, but avoid using it as possible, because it requires you to modify your code and may reduce readability. Normally, it is not a good idea to modify the code just for testing, but in some cases, there is no choice but to use this.

#!/bin/sh
# ./today.sh

# When run directly without testing, the "__()" function does nothing.
test || __() { :; }

# the "now()" function is defined here, so it can't be mocked in advance.
now() { date +"%Y-%m-%d %H:%M:%S"; }

# The function you want to test
today() {
  now=$(now)
  echo "${now% *}"
}

# I want to mock the "now()" function here.
__ begin __

today=$(today)
echo "Today is $today"

__ end __
Describe "today.sh"
  Intercept begin
  __begin__() {
    now() { echo "2021-01-01 01:02:03"; }
  }
  __end__() {
    # The "run source" is run in a subshell, so you need to use "%preserve"
    # to preserve variables
    %preserve today
  }

  It "gets today's date"
    When run source ./today.sh
    The output should eq "Today is 2021-01-01"
    The variable today should eq "2021-01-01"
  End
End

Intercept

Usage: Intercept [<name>...]

Specify the name(s) to intercept.

NOTE: I will change Intercept to Interceptors to make it a declarative DSL.

test || __() { :; }

Define the __ function that does nothing except when run as a test (via ShellSpec). This allows you to run it as a production without changing the code.

The test command is the shell built-in test command. This command returns false (non-zero) when called with no arguments. This will allow who are not familiar with ShellSpec to will understand what the result will be, even if they don't know what the code is for. Of course, it is good practice to comment on what the code is for

When run via ShellSpec, the test command is redefined and returns true "only once" when called with no arguments. After that, it will return to its original behavior. This means that this code needs to be executed only once, at the start of the shell script.

__

Usage: __ <name> [arguments...] __

This is where the process is intercepted. You can define more than one. If the name matches the name specified in Intercept, the __<name>__ function will be called.

Note that if the name is not specified in Intercept, nothing will be done, but the exit status will be changed to 0.

spec_helper

The spec_helper can be used to set shell options for all specfiles, define global functions,check the execution shell, load custom matchers, etc.

The spec_helper is the default module name. It can be changed to any other name, and multiple modules can be used. Only characters accepted by POSIX as identifiers can be used in module names. The file name of the module must be the module name with the extension .sh appended. It is loaded from SHELLSPEC_LOAD_PATH using the --require option.

The following is a typical spec_helper. The following three callback functions are available.

# Filename: spec/spec_helper.sh

set -eu

spec_helper_precheck() {
  minimum_version "0.28.0"
  if [ "$SHELL_TYPE" != "bash" ]; then
    abort "Only bash is supported."
  fi
}

spec_helper_loaded() {
  : # In most cases, you won't use it.
}

spec_helper_configure() {
  import 'support/custom_matcher'
  before_each "global_before_each_hook"
}

# User-defined global function
global_before_each_hook() {
  :
}

# In version <= 0.27.x, only shellspec_spec_helper_configure was available.
# This callback function is still supported but deprecated in the future.
# Please rename it to spec_helper_configure.
# shellspec_spec_helper_configure() {
#  :
# }

The spec_helper will be loaded at least twice. The first time is at precheck phase, which is executed in a separate process before the specfile execution. The second time will be load at the beginning of the specfile execution. If you are using parallel execution, it will be loaded every specfile.

Within each callback function, there are several helper functions available. These functions are not available outside of the callback function. Also, these callback functions will be removed automatically when spec_helper is finished loading. (User-defined functions will not be removed.)

<module>_precheck

This callback function will be invoked only once before loading specfiles. Exit with exit or abort, or return non-zero to exit without executing specfiles. Inside this function, set -eu is executed, so an explicit return on error is not necessary.

Since it is invoked in a separate process from specfiles, changes made in this function will not be affected in specfiles.

minimum_version

  • Usage: minimum_version <version>

Specifies the minimum version of ShellSpec that the specfile supports. The version format is semantic version. Pre-release versions have a lower precedence than the associated normal version, but comparison between pre-release versions is not supported. The build metadata will simply be ignored.

NOTE: Since <module>_precheck is only available in 0.28.0 or later, it can be executed with earlier ShellSpecs even if minimum_version is specified. To avoid this, you can implement a workaround using --env-from.

# spec/env.sh
# Add `--env-from spec/env.sh` to `.shellspec`
major_minor=${SHELLSPEC_VERSION%".${SHELLSPEC_VERSION#*.*.}"}
if [ "${major_minor%.*}" -eq 0 ] && [ "${major_minor#*.}" -lt 28 ]; then
  echo "ShellSpec version 0.28.0 or higher is required." >&2
  exit 1
fi

error, warn, info

  • Usage: error [messages...]
  • Usage: warn [messages...]
  • Usage: info [messages...]

Outputs a message according to the type. You can also use echo or printf.

abort

  • Usage: abort [messages...]
  • Usage: abort <exit status> [messages...]

Display an error message and exit. If the exit status is omitted, it is 1. You can also exit with exit. exit 0 will exit normally without executing the specfiles.

setenv, unsetenv

  • Usage: setenv [name=value...]
  • Usage: unset [name...]

You can use setenv or unsetenv to pass or remove environment variables from precheck to specfiles.

environment variables

The following environment variables are defined.

  • VERSION - ShellSpec Version
  • SHELL_TYPE - Currently running shell type (e.g. bash)
  • SHELL_VERSION - Currently running shell version (e.g. 4.4.20(1)-release)

NOTE: Be careful not to confuse SHELL_TYPE with the environment variable SHELL. The environment variable SHELL is the user login shell, not the currently running shell. It is a variable set by the system, and which unrelated to ShellSpec.

<module>_loaded

It is called after loading the shellspec's general internal functions, but before loading the core modules (subject, modifier, matcher, etc). If parallel execution is enabled, it may be called multiple times in isolated processes. Internal functions starting with shellspec_ can also be used, but be aware that they may change.

This was created to perform workarounds for specific shells in order to test ShellSpec itself. Other than that, I have not come up with a case where this is absolutely necessary, but if you have one, please let me know.

<module>_configure

This callback function will be called after core modules (subject, modifier, matcher, etc) has been loaded. If parallel execution is enabled, it may be called multiple times in isolated processes. Internal functions starting with shellspec_ can also be used, but be aware that they may change. It can be used to set global hooks, load custom matchers, etc., and override core module functions.

import

  • Usage: import <module> [arguments...]

Import a custom module from SHELLSPEC_LOAD_PATH.

before_each, after_each

  • Usage: before_each [hooks...]
  • Usage: after_each [hooks...]

Register hooks to be executed before and after every example. It is the same as executing BeforeEach/AfterEach at the top of all specfiles.

before_all, after_all

  • Usage: before_all [hooks...]
  • Usage: after_all [hooks...]

Register hooks to be executed before and after all example. It is the same as executing BeforeAll/AfterAll at the top of all specfiles.

NOTE: This is a hook that is called before and after each specfile, not before and after all specfiles.

Self-executable specfile

Add eval "$(shellspec - -c) exit 1" to the top of the specfile and give execute permission to the specfile. You can use /bin/sh, /usr/bin/env bash, etc. for shebang. The specfile will be run in the shell written in shebang.

#!/bin/sh

eval "$(shellspec - -c) exit 1"

# Use the following if version <= 0.27.x
# eval "$(shellspec -)"

Describe "bc command"
  bc() { echo "$@" | command bc; }

  It "performs addition"
    When call bc "2+3"
    The output should eq 5
  End
End

The -c option is available since 0.28.0, and you can also pass other options. If you run the specfile directly, --pattern will be automatically set to *. These options will be ignored if run via shellspec command.

The use of shellspec as shebang is deprecated because it is not portable.

#!/usr/bin/env shellspec -c
Linux does not allow passing options

#!/usr/bin/env -S shellspec -c
The -S option requires GNU Core Utilities 8.30 (2018-07-01) or later.

Use with Docker

You can run ShellSpec without installation using Docker. ShellSpec and specfiles run in a Docker container.

See How to use ShellSpec with Docker.

Extension

Custom subject, modifier and matcher

You can create custom subject, custom modifier and custom matcher.

See examples/spec/support/custom_matcher.sh for custom matcher.

NOTE: If you want to verify using shell function, you can use result modifier or satisfy matcher. You don't need to create a custom matcher, etc.

Code Coverage

ShellSpec has integrated coverage feature. To use this feature Kcov (v38 or later) is required.

Supported shells

Supported only in bash, zsh, and ksh, where DEBUG trap is implemented. However, we recommend latest bash for the following reasons.

  • bash (older versions): There seems to be some code that cannot be measured correctly.
  • zsh: There seems to be some code that cannot be measured correctly, and the measurement rate will be the lowest.
  • ksh: ksh93u+ and ksh2020 may have side effects on exit status when DEBUG trap is enabled due to a bug. This bug has been fixed in ksh93u+m. These are also prone to instability, especially with ksh2020 (which has been abandoned).

In any shell, some code may not be measured correctly (e.g., code containing eval or newline). These are limitations and problems caused by shells and Kcov.

Measurement target

ShellSpec measures only the necessary codes to improve the measurement speed. Also, there are some things that cannot be measured due to implementation.

  • The shell scripts loaded by Include will be measured.
  • The shell functions called by the When evaluation will be measured.
  • The shell scripts executed by the When run script evaluation will be measured.
  • The shell scripts executed by the When run source evaluation will be measured.
  • The external commands executed by the When evaluation will NOT be measured.
    • Even if it is a shell script, it is not measured when it is executed as an external command.
  • If other than the above, it will not be measured.

By default only shell scripts whose names contain .sh are coverage targeted. If you want to include other files, you need to adjust options with --kcov-options.

# Default kcov (coverage) options
--kcov-options "--include-path=. --path-strip-level=1"
--kcov-options "--include-pattern=.sh"
--kcov-options "--exclude-pattern=/.shellspec,/spec/,/coverage/,/report/"

# Example: Include script "myprog" with no extension
--kcov-options "--include-pattern=.sh,myprog"

# Example: Only specified files/directories
--kcov-options "--include-pattern=myprog,/lib/"

Coverage report

[Coverage report][coverage] and cobertura.xml and sonarqube.xml files are generated under the coverage directory by Kcov. You can easily integrate with Coveralls, Code Climate, Codecov, etc.

For developers

Subprojects

ShellMetrics - Cyclomatic Complexity Analyzer for shell scripts

URL: https://github.com/shellspec/shellmetrics

ShellBench - A benchmark utility for POSIX shell comparison

URL: https://github.com/shellspec/shellbench

altshfmt - AltSH (alternative shell script) formatter

URL: https://github.com/shellspec/altshfmt

Related projects

getoptions - An elegant option parser and generator for shell scripts

URL: https://github.com/ko1nksm/getoptions

readlinkf - readlink -f implementation for shell scripts

URL: https://github.com/ko1nksm/readlinkf

portable-echo - Portable echo shell function for POSIX compliant shells

URL: https://github.com/ko1nksm/portable-echo.sh

Inspired frameworks

  • RSpec - Behaviour Driven Development for Ruby
  • Jest - Delightful JavaScript Testing
  • Mocha - the fun, simple, flexible JavaScript test framework
  • Jasmine - Behavior-Driven JavaScript
  • Ginkgo - A Golang BDD Testing Framework
  • JUnit 5 - The programmer-friendly testing framework for Java

Contributions

All contributions are welcome!

ShellSpec uses a peculiar coding style to assure high performance, reliability and portability, and the external commands allowed to use are greatly restricted.

We recommend that you create WIP PR early or offer suggestions in discussions to avoid ruining your work.

See CONTRIBUTING.md

Author: Koichi Nakashima (ko1nksm)