正如我们上个章节里面看到的， Test::Nginx 提供了一个简单规范的格式来表达测试用例。
每个测试用例都用一个测试块来表示。一个测试块由一个标题、一个可选的描述和几个数据节[1]，
用于指定输入和期望的输出。在这一节我们将仔细看看对于不同的测试要求，如何准备这样的测试案例。
设计测试用例在很多方面都是一门艺术。有时候可能设计测试案例花费的时间和精力，多于实现要测试的功能，
这个和我们自己的经验有关。Test::Nginx 努力尝试让编写测试用例尽可能的简单，
但仍然做不到让整个测试用例设计的过程自动化。只有你才确切的知道测试什么，以及如何去测试。
本节讲侧重于 Test::Nginx 提供的基本原语，基于此你可以设计出巧妙和有效的测试用例。
在一个测试块里面，我们可以用不同的数据节来指定我们自定义的代码片段，
放在由 Test::Nginx 生成的最终的 nginx.conf 配置文件的不同位置中。
最常见的是 config 节，它用来在默认测试服务器的 server {} 配置块中插入自定义代码片段。
我们也可以在 nginx.conf 的 http {} 配置块中，用 http_config 节来插入自定义的内容。
main_config 节可以在 NGINX 配置的最上层范围中插入自定义内容。
一起看看下面这个例子。
=== TEST 1:
--- main_config
env MY_ENVIRONMENT;
--- http_config
init_worker_by_lua_block {
print("init")
}
--- config
location = /t {
echo ok;
}
--- request
GET /t
--- response_body
ok这个测试块会生成一个有如下基本结构的 nginx.conf 文件：
...
env MY_ENVIRONMENT;
http {
...
init_worker_by_lua_block {
print("init")
}
server {
...
location = /t {
echo ok;
}
}
}请注意 main_config, http_config, 和 config 这几个数据节的值，是如何映射到 NGINX 配置文件的不同位置的。
有疑问的时候，我们总是可以来检查这个测试框架生成的实际 nginx.conf 文件，
它位于当前工作目录（通常是当前项目的根目录）的 t/servroot/conf/nginx.conf 位置中。
Test::Nginx 会为每一个测试块生成一个新的 nginx.conf 文件，这会让每个测试块独立存在成为可能。
测试框架默认会在运行每一个测试块之前，自动启动一个新的 NGINX 服务，然后在这个测试块运行结束后立即关闭服务。
好在 NGINX 是一个轻量的服务器，通常启动和关闭都非常快。所以，测试块运行起来并没有看上去那么慢。
准备一个请求，最简单的是用 request 数据节,
比如
--- request
GET /t?a=1&b=2默认使用的是 HTTP/1.1 协议。如果你愿意，可以指明来使用 HTTP/1.0 协议:
--- request
GET /t?a=1&b=2 HTTP/1.0request 小节的值中，前导空格或者空白行都会被自动丢弃。你甚至可以在最前面加一个 # 字符来增加注释，比如
--- request
# this is a simple test:
GET /t你可以通过如下 more_headers 小节来同时增加一些额外的请求头部信息。
--- request
GET /t
--- more_headers
Foo: bar
Bar: bazPreparing pipelined HTTP requests are also possible. But you need to use
the pipelined_requests section instead of request. For instance,
=== TEST 1: pipelined requests
--- config
location = /t {
echo ok;
}
--- pipelined_requests eval
["GET /t", "GET /t"]
--- request_body eval
["ok\n", "ok\n"]It is worth noting that we use the eval filter with the pipelined_requests
section to treat the literal value of that section as Perl code. This way
we can construct a Perl array of the request strings, which is the expected
data format for the pipelined_requests section. Similarly we need a similar
trick for the response_body section when checking outputs. With an array
of expected response body data, we can expect and check different values
for different individual request in the pipeline. Note, however, not every
data section supports the same array-typed value semantics as response_body.
We have already visited the response_body and error_code data sections
for checking the response body data and response status code, respectively.
The response_body data section always performs an exact whole-string
comparison between the section value and the actual response body. It tries
to be clever when long string value comparison fails. Consider the following
sample output from prove.
t/foo.t .. 1/?
# Failed test 'TEST 1: long string test - response_body - response is expected (req 0)'
# at .../test-nginx/lib/Test/Nginx/Socket.pm line 1282.
# got: ..."IT 2.x is enabled.\x{0a}\x{0a}"...
# length: 409
# expected: ..."IT 2.x is not enabled.\x{0a}"...
# length: 412
# strings begin to differ at char 400 (line 1 column 400)
# Looks like you failed 1 test of 2.
/tmp/foo.t .. Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/2 subtests
Test Summary Report
-------------------
/tmp/foo.t (Wstat: 256 Tests: 2 Failed: 1)
Failed test: 2
Non-zero exit status: 1
Files=1, Tests=2, 0 wallclock secs (0.01 usr 0.00 sys + 0.09 cusr 0.03 csys = 0.13 CPU)
Result: FAIL
From this test report, we can clearly see that
-
it is the test block with the title
TEST 1: long string testthat is failing, -
it is the
response_bodydata section check that fails, -
the actual response body data is 409 bytes long while the expected value is 412 bytes, and
-
the expected value has an additional
notword in the string fragmentIT 2.x is enabledand the difference starts at the offset 400 in the long string.
Behind the scene, Test::Nginx uses the Perl module Test::LongString
to do the long string comparisons. It is also particularly useful while
checking response body data in binary formats.
If your response body data is in a multi-line textual format, then you
may also want to use a diff-style output when the data does not match.
To achieve this, we can call the no_long_string() Perl function before
the run_tests() function call in the prologue part of the test file.
Below is such an example.
use Test::Nginx::Socket 'no_plan';
no_long_string();
run_tests();
__DATA__
=== TEST 1:
--- config
location = /t {
echo "Life is short.";
echo "Moon is bright.";
echo "Sun is shining.";
}
--- request
GET /t
--- response_body
Life is short.
Moon is deem.
Sun is shining.Note the no_long_string() call in the prologue part. It is important
to place it before the run_tests() call otherwise it would be too late
for it to take effect, obviously.
Invoking the prove utility (or any shell wrappers for it) to run this
test file gives the following details about the test failure:
# Failed test 'TEST 1: - response_body - response is expected (req 0)' # at .../test-nginx/lib/Test/Nginx/Socket.pm line 1277. # @@ -1,3 +1,3 @@ # Life is short. # -Moon is deem. # +Moon is bright. # Sun is shining. # Looks like you failed 1 test of 2.
It is obvious that the second line of the response body output is different.
You can even further disable the diff-style comparison mode by adding
a no_diff() Perl function call in the prologue part. Then the failure
report will look like this:
# Failed test 'TEST 1: - response_body - response is expected (req 0)' # at .../test-nginx/lib/Test/Nginx/Socket.pm line 1277. # got: 'Life is short. # Moon is bright. # Sun is shining. # ' # expected: 'Life is short. # Moon is deem. # Sun is shining. # ' # Looks like you failed 1 test of 2.
That is, Test::Nginx just gives full listing of the actual response body
data and the expected one without any abbreviations or hand-holding.
When the request body may change in some ways or you just care about certain
key words in a long data string, you can specify a Perl regular expression
to do a pattern match against the actual request body data. This is achieved
by the response_body_like data section. For example,
--- response_body_like: age: \d+Be careful when you are using the multi-line data section value form. A
trailing newline character appended to your section value may make your
pattern never match. In this case the chomp filter we introduced in an
early section can be very helpful here. For example,
--- response_body_like chomp
age: \d+You can also use the eval filter to construct a Perl regular expression
object with a Perl expression, as in
--- response_body_like eval
qr/age: \d+/This is the most flexible form to specify a pattern.
|
Note
|
Perl uses the qr quoting structure to explicitly construct regular
expression objects. You can use various different quoting forms like qr/…/,
qr!…!, qr#…#, and qr{…}.
|
The response_headers data section can be used to validate response header
entries. For example,
--- response_headers
Foo: bar
Bar: baz
!BlahThis section dictates 3 tests actually:
-
The response header
Foomust appear and must take the valuebar; -
The response header
Barmust appear and must take the valuebaz; and -
The response header
Blahmust not appear or take an empty value.
In addition to responses, the NGINX error log file is also an important output channel for an NGINX server setup.
One immediate testing requirement is to check whether or not a piece of
text appears in any error log messages. Such checks can be done via the
data sections error_log and no_error_log, respectively. The former
ensures that some lines in the error log file contain the string specified
as the section value while the latter tests the opposite: ensuring that
no line contains the pattern.
For example,
--- error_log
Hello world from my serverThen the string Hello world from my server (without the trailing new-line)
must appear in at least one line of the NGINX error log. You can specify
multiple strings in separate lines of the section value to perform different
checks, for instance,
--- error_log
This is a dog!
Is it a cat?Then it performs two error log checks, one is to ensure that the string
This is a dog! appears in some error log lines. The order of these two
string patterns do not matter at all.
If one of the string pattern failed to match any lines in the error log
file, then we would get a test failure report from prove like below.
# Failed test 'TEST 1: simple test - pattern "This is a dog!" matches a line in error.log (req 0)'
If you want to specify a Perl regular expression (regex) as one of the
patterns, then you should use the eval section filter to construct a
Perl-array as the section value, as in
--- error_log eval
[
"This is a dog!",
qr/\w+ is a cat\?/,
]As we have seen earlier, Perl regexes can be constructed via the qr/…/
quoting syntax. Perl string patterns in the Perl array specified by double
quotes or single quotes are still treated as plain string patterns, as
usual. If the array contains only one regex pattern, then you can omit
the array itself, as in
--- error_log eval
qr/\w+ is a cat\?/Test::Nginx puts the error log file of the test NGINX server in the file
path t/servroot/logs/error.log. As a test writer, we frequently check
out this file directly when things go wrong. For example, it is common
to make mistakes or typos in the patterns we specify for the error_log
section. Also, scanning the raw log file can give us insight about the
details of the NGINX internal working when the NGINX debugging logs are
enabled in the NGINX build.
The no_error_log section is very similar to error_log but it checks
the nonexistence of the string patterns in the NGINX error log file. One
of the most frequent uses of the no_error_log section is to ensure that
there is no error level messages in the log file.
--- no_error_log
[error]If, however, there is a line in the nginx error log file that contains
the string [error], then the test fails. Below is such an example.
# Failed test 'TEST 1: simple test - pattern "[error]" should not match any line in error.log but matches line "2016/02/01 11:59:50 [error] 1788\#0: *1 lua entry thread aborted: runtime error: content_by_lua(nginx.conf:42):2: bad"'
This is a great way to find the details of the error quickly by just looking at the test report.
Like error_log, this section also supports Perl array values and Perl
regex values though the eval filter.
The error_log and no_error_log sections are very handy in quickly checking
the appearance of contain patterns in the NGINX error log file. But they
have serious limitations in that it is impossible to impose stronger constraints
on the relative order of the messages containing the patterns nor on the
number of their occurrences.
To address such limitations, Test::Nginx::Socket provides an alternative
way to check NGINX error logs in a way similar to the famous UNIX tool,
grep. The sections grep_error_log and grep_error_log_out are used
for this purpose. The test writer uses the grep_error_log section to
specify a pattern, with which the test framework scans through the NGINX
error log file and collect all the matched parts of the log file lines
along the way, forming a final result. This aggregated log data result
is then matched against the expected value specified as the value of the
grep_error_log_out section, in a similar way as with the response_body
section discussed above.
It is easiest to explain with a simple example.
=== TEST 1: simple grep test for error logs
--- config
location = /t {
content_by_lua_block {
print("it is matched!")
print("it is matched!")
print("it is matched!")
}
}
--- request
GET /t
--- grep_error_log: it is matched!
--- grep_error_log_out
it is matched!
it is matched!
it is matched!Here we use the Lua function print() provided by the
ngx_http_lua
module to generate NGINX error log messages at the notice level. This
test case tests the number of the log messages containing the string it
is matched!. It is important to note that only the matched part of the
log file lines are collected in the final result instead of the whole log
lines. This simplifies the comparison a lot since NGINX error log messages
can contain varying details like timestamps and connection numbers.
A more useful form of this test is to specify a Perl regex pattern in the
grep_error_log section. Consider the following example.
=== TEST 1: simple grep test for error logs
--- config
location = /t {
content_by_lua_block {
print("test: before sleeping...")
ngx.sleep(0.001) -- sleeping for 1ms
print("test: after sleeping...")
}
}
--- request
GET /t
--- grep_error_log eval: qr/test: .*?\.\.\./
--- grep_error_log_out
test: before sleeping...
test: after sleeping...We specify a Perl regex pattern, test: .*?\.\.\., here to filter out
all the error log messages starting with test: and ending with ….
And naturally in this test we also require the relative order of these
two messages, that is, before sleeping must appear before after sleeping.
Otherwise, we shall see failure reports like below:
# Failed test 'TEST 1: simple grep test for error logs - grep_error_log_out (req 0)'
# at ..../lib/Test/Nginx/Socket.pm line 1048.
# got: "test: after sleeping...\x{0a}test: before sleeping...\x{0a}"
# length: 49
# expected: "test: before sleeping...\x{0a}test: after sleeping...\x{0a}"
# length: 49
# strings begin to differ at char 7 (line 1 column 7)
As with the response_body section, we can also call the no_long_string()
Perl function before run_tests() in the test file prologue, so as to
disable the long string output mode and enable the diff mode. Then the
test failure would look like this:
# Failed test 'TEST 1: simple grep test for error logs - grep_error_log_out (req 0)' # at .../lib/Test/Nginx/Socket.pm line 1044. # @@ -1,2 +1,2 @@ # -test: before sleeping... # test: after sleeping... # +test: before sleeping...
Obviously, for this test case, the diff format looks better.
By default, Test::Nginx::Socket performs the NGINX error log checks not
long after it receives the complete HTTP response for the test request.
Sometimes, when the log messages are generated by the server after sending
out the response, the error log checks may be carried out too early
that the messages are not yet written into the log file. In this case,
we can specify an extra delay via the wait data section for the test
scaffold to wait for the error log messages. Here is an example:
=== TEST 1: wait for the timer
--- config
location = /t {
content_by_lua_block {
local function f(premature)
print("HERE!")
end
assert(ngx.timer.at(0.1, f))
}
}
--- request
GET /t
--- error_log
HERE!
--- no_error_log
[error]
--- wait: 0.12Here we create a timer via the ngx.timer.at Lua function, which expires
after 0.1 seconds. Due to the asynchronous nature of timers, the request
handler does not wait for the timer to expire and immediately finishes
processing the current request and sends out a response with an empty body.
To check for the log message HERE! generated by the timer handler f,
we have to specify an extra delay for the test scaffold to wait. The 0.12
seconds time is specified in this example but any values larger than 0.1
would suffice. Without the wait section, this test case would fail with
the following output:
# Failed test 'TEST 1: wait for the timer - pattern "HERE!" matches a line in error.log (req 0)'
Obviously the test scaffold checks the error log too soon, even before the timer handler runs.
Test::Nginx::Socket offers a rich set of data sections for specifying
various different input data and expected output data, ranging from NGINX
configuration file snippets, test requests, to expected responses and error
log messages. We have already demonstrated the power of data driven testing
and declarative test case crafting. We want to achieve multiple goals at
the same time, that is, not only to make the tests self-contained and highly
readable, but also to make the test report easy to interpret and analyze
when some of the tests fail. Raw files automatically generated by the test
scaffold, like t/servroot/conf/nginx.conf and t/servroot/logs/error.log,
should be checked frequently when manually debugging the test cases. The
next section extends the discussion of this section with a focus on testing
erroneous cases.