From 2897fc01a3c6a41d702fd432bba13efd0c65557a Mon Sep 17 00:00:00 2001 From: Casper Beyer Date: Fri, 15 Jan 2021 05:36:15 +0800 Subject: [PATCH 1/8] Create Testing.md --- docs/Testing.md | 127 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 127 insertions(+) create mode 100644 docs/Testing.md diff --git a/docs/Testing.md b/docs/Testing.md new file mode 100644 index 00000000..86a52c3e --- /dev/null +++ b/docs/Testing.md @@ -0,0 +1,127 @@ +# Testing + +A test case takes the form of a binary \.wasm file, next to that that there will always be a \.\ source file from which the test was originally compiled from which can be used as a reference in the event of an error. + +Additionally, any of the following optional auxilary files and directories may be present: +- \.arg +- \.env +- \.dir +- \.stdin +- \.stdout +- \.stderr +- \.status + +## Running tests + +To run a test do the following steps to prepare for execution: + +- Prepare inputs + - Given an \ module; take the \ of said module. + - If \.\ exists; take the program arguments from said file. + - If \.\ exists; take the program environment from said file. + - If \.\ exists; preopen the directory from said file. + - If \.\ exists; pipe said file into the program as stdin. +- Run program +- Collect results + - If \.\ exists; assert that the programs stdout matches + said file. + - If \.\ exists; assert that the programs stdout matches + said file. + - If \.\ exists; assert that the programs stdout matches + said file; otherwise assert that the program exited with status 0. +- Pass + +For example: + +```bash +# Usage: $1 +# $1 wasmtime proc_exit-success.wasm +# $1 wasmer proc_exit-failure.wasm +#!/usr/bin/env bash + +runtime=$1 +input=$2 + +prefix=${input%.*} +if [ -e "$prefix.stdin" ]; then + stdin="$prefix.stdin" +else + stdin="/dev/null" +fi + +output="$(mktemp -d)" +stdout_actual="$output/stdout" +stderr_actual="$output/stderr" +status_actual="$output/status" + +if [ -e "$prefix.arg" ]; then + arg=$(cat "$prefix.env") +else + arg="" +fi + +if [ -e "$prefix.env" ]; then + env=$(sed -e 's/^/--env /' < "$prefix.env") +else + env="" +fi + +if [ -e "$prefix.dir" ]; then + dir="--dir $prefix.dir" +else + dir="" +fi + +status=0 + +"$runtime" $dir $input -- $arg \ + < "$stdin" \ + > "$stdout_actual" \ + 2> "$stderr_actual" \ + || status=$? + +echo $status > "$status_actual" + +stdout_expected="$prefix.stdout" +if [ -e "$stdout_expected" ]; then + diff -u "$stderr_expected" "$stderr_actual" +fi + +stderr_expected="$prefix.stderr" +if [ -e "$stderr_expected" ]; then + diff -u "$stdout_expected" "$stdout_actual" +fi + +status_expected="$prefix.status" +if [ -e "$prefix.status" ]; then + diff -u "$status_expected" "$status_actual" +elif [ ! "$status" -eq "0" ]; then + cat $stderr_actual + exit 1 +fi +``` + +## Writing tests + +Any source language may be used to write tests as long as the compiler is +well-known and freely available. + +Each source file must be accompanied by a pre-compiled binary version of said +source file. + +Each source file must also start with a comment containing the complete command +that was used to compile the binary. + +Tests must follow the naming convention of \[-\].. + +Tests should be scoped to the smallest unit possible. + +In addition to the source and binary files, a test can have any of the following +auxilary files and directories: +- \.arg +- \.env +- \.dir +- \.stdin +- \.stdout +- \.stderr +- \.status From 77dc00b7b58d50855b3443d4dc434a0db97450bf Mon Sep 17 00:00:00 2001 From: Casper Beyer Date: Fri, 15 Jan 2021 01:57:50 +0000 Subject: [PATCH 2/8] Add note about wasm32-unknown-unknown --- docs/Testing.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/Testing.md b/docs/Testing.md index 86a52c3e..5dff5a58 100644 --- a/docs/Testing.md +++ b/docs/Testing.md @@ -103,8 +103,8 @@ fi ## Writing tests -Any source language may be used to write tests as long as the compiler is -well-known and freely available. +Any source language may be used to write tests as long as the compiler supports +the wasm32-unknown-unknown target and is well-known and freely available. Each source file must be accompanied by a pre-compiled binary version of said source file. From bb3e04beec8972f97eedab7dec1ad05b864699bd Mon Sep 17 00:00:00 2001 From: Casper Beyer Date: Fri, 15 Jan 2021 16:57:35 +0000 Subject: [PATCH 3/8] gqq --- docs/Testing.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/docs/Testing.md b/docs/Testing.md index 5dff5a58..29150c05 100644 --- a/docs/Testing.md +++ b/docs/Testing.md @@ -1,8 +1,12 @@ # Testing -A test case takes the form of a binary \.wasm file, next to that that there will always be a \.\ source file from which the test was originally compiled from which can be used as a reference in the event of an error. +A test case takes the form of a binary \.wasm file, next to that +that there will always be a \.\ source file from which the +test was originally compiled from which can be used as a reference in the event +of an error. -Additionally, any of the following optional auxilary files and directories may be present: +Additionally, any of the following optional auxilary files and directories may +be present: - \.arg - \.env - \.dir From 5a9e3f455f1037ffa58715c665a8dba7d65b64ba Mon Sep 17 00:00:00 2001 From: Casper Beyer Date: Fri, 15 Jan 2021 18:36:10 +0000 Subject: [PATCH 4/8] Quote instead of escape --- docs/Testing.md | 45 +++++++++++++++++++++++---------------------- 1 file changed, 23 insertions(+), 22 deletions(-) diff --git a/docs/Testing.md b/docs/Testing.md index 29150c05..5c852e8c 100644 --- a/docs/Testing.md +++ b/docs/Testing.md @@ -7,31 +7,31 @@ of an error. Additionally, any of the following optional auxilary files and directories may be present: -- \.arg -- \.env -- \.dir -- \.stdin -- \.stdout -- \.stderr -- \.status +- `.arg` +- `.env` +- `.dir` +- `.stdin` +- `.stdout` +- `.stderr` +- `.status` ## Running tests To run a test do the following steps to prepare for execution: - Prepare inputs - - Given an \ module; take the \ of said module. - - If \.\ exists; take the program arguments from said file. - - If \.\ exists; take the program environment from said file. - - If \.\ exists; preopen the directory from said file. - - If \.\ exists; pipe said file into the program as stdin. + - Given an `` module; take the of said module. + - If `.` exists; take the program arguments from said file. + - If `.` exists; take the program environment from said file. + - If `.` exists; preopen the directory from said file. + - If `.` exists; pipe said file into the program as stdin. - Run program - Collect results - - If \.\ exists; assert that the programs stdout matches + - If `.` exists; assert that the programs stdout matches said file. - - If \.\ exists; assert that the programs stdout matches + - If `.` exists; assert that the programs stdout matches said file. - - If \.\ exists; assert that the programs stdout matches + - If `.` exists; assert that the programs stdout matches said file; otherwise assert that the program exited with status 0. - Pass @@ -122,10 +122,11 @@ Tests should be scoped to the smallest unit possible. In addition to the source and binary files, a test can have any of the following auxilary files and directories: -- \.arg -- \.env -- \.dir -- \.stdin -- \.stdout -- \.stderr -- \.status + +- `.arg` +- `.env` +- `.dir` +- `.stdin` +- `.stdout` +- `.stderr` +- `.status` From 2686e48ccbc8b10a603d1627f41b72bb3410b628 Mon Sep 17 00:00:00 2001 From: Casper Beyer Date: Fri, 15 Jan 2021 18:37:31 +0000 Subject: [PATCH 5/8] Given an file --- docs/Testing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Testing.md b/docs/Testing.md index 5c852e8c..a20f65a1 100644 --- a/docs/Testing.md +++ b/docs/Testing.md @@ -20,7 +20,7 @@ be present: To run a test do the following steps to prepare for execution: - Prepare inputs - - Given an `` module; take the of said module. + - Given an `.wasm` file; take the `` of said module. - If `.` exists; take the program arguments from said file. - If `.` exists; take the program environment from said file. - If `.` exists; preopen the directory from said file. From 73ae0ef95c7c85a732d468cd41c26e06bdc2f9f3 Mon Sep 17 00:00:00 2001 From: Casper Beyer Date: Sat, 16 Jan 2021 03:05:41 +0800 Subject: [PATCH 6/8] Update docs/Testing.md --- docs/Testing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Testing.md b/docs/Testing.md index a20f65a1..91ef75a1 100644 --- a/docs/Testing.md +++ b/docs/Testing.md @@ -20,7 +20,7 @@ be present: To run a test do the following steps to prepare for execution: - Prepare inputs - - Given an `.wasm` file; take the `` of said module. + - Given an `.wasm` file; take the `` of said file. - If `.` exists; take the program arguments from said file. - If `.` exists; take the program environment from said file. - If `.` exists; preopen the directory from said file. From 76e1d407d463814e4d93a9078d1e1fe373849cb0 Mon Sep 17 00:00:00 2001 From: Casper Beyer Date: Sat, 16 Jan 2021 09:09:08 +0800 Subject: [PATCH 7/8] Add $env to the example runner --- docs/Testing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Testing.md b/docs/Testing.md index 91ef75a1..d0e9b43c 100644 --- a/docs/Testing.md +++ b/docs/Testing.md @@ -78,7 +78,7 @@ fi status=0 -"$runtime" $dir $input -- $arg \ +"$runtime" $dir $env $input -- $arg \ < "$stdin" \ > "$stdout_actual" \ 2> "$stderr_actual" \ From 6d24e2280732701836f1c43792198d9e639b686e Mon Sep 17 00:00:00 2001 From: Casper Beyer Date: Thu, 28 Jan 2021 15:58:06 +0800 Subject: [PATCH 8/8] Minor fixes --- docs/Testing.md | 20 +++++++++----------- 1 file changed, 9 insertions(+), 11 deletions(-) diff --git a/docs/Testing.md b/docs/Testing.md index d0e9b43c..4a14dba0 100644 --- a/docs/Testing.md +++ b/docs/Testing.md @@ -20,27 +20,25 @@ be present: To run a test do the following steps to prepare for execution: - Prepare inputs - - Given an `.wasm` file; take the `` of said file. - - If `.` exists; take the program arguments from said file. - - If `.` exists; take the program environment from said file. - - If `.` exists; preopen the directory from said file. - - If `.` exists; pipe said file into the program as stdin. + - Given an `` file; take the `` of said file. + - If `.arg` exists; take the program arguments from said file. + - If `.env` exists; take the program environment from said file. + - If `.dir` exists; preopen the directory from said file. + - If `.stdin` exists; pipe said file into the program as stdin. - Run program - Collect results - - If `.` exists; assert that the programs stdout matches + - If `.stdout` exists; assert that the programs stdout matches said file. - - If `.` exists; assert that the programs stdout matches + - If `.stderr` exists; assert that the programs stdout matches said file. - - If `.` exists; assert that the programs stdout matches + - If `.status` exists; assert that the programs stdout matches said file; otherwise assert that the program exited with status 0. - Pass For example: ```bash -# Usage: $1 -# $1 wasmtime proc_exit-success.wasm -# $1 wasmer proc_exit-failure.wasm +# Usage: $0 #!/usr/bin/env bash runtime=$1