v0.26.0

k6 v0.26.0 is here! 🎉

This release contains mostly bug fixes, though it also has several new features and enhancements! They include a new JS compatibility mode option, exporting the end-of-test summary to a JSON report file, speedups to the InfluxDB and JSON outputs, http.batch() improvements, a brand new CSV output, multiple layered HTTP response body decompression, being able to use console in the init context, a new optional column in the summary, and Docker improvements!

Thanks to @Sirozha1337, @openmohan, @MMartyn, @KajdeMunter, @dmitrytokarev and @dimatock for contributing to this release!

New features and enhancements!

A new JavaScript compatibility mode option (#1206)

This adds a way to disable the automatic script transformation by Babel (v6.4.2) and loading of core-js (v2) polyfills, bundled in k6. With the new base compatibility mode, k6 will instead rely only on the goja runtime and what is built into k6.
This can be configured through the new --compatibility-mode CLI flag and the K6_COMPATIBILITY_MODE environment variable. The possible values currently are:

• extended: this is the default and current compatibility mode, which uses Babel and core.js to achieve ES6+ compatibility.
• base: an optional mode that disables loading of Babel and core.js, running scripts with only goja's native ES5.1+ compatibility. If the test scripts don't require ES6 compatibility (e.g. they were previously transformed by Babel), this option can be used to reduce RAM usage during test runs.

More info what this means can be found in the documentation.

Our benchmarks show a considerable drop in memory usage - around 80% for simple scripts, and around 50% in the case of 2MB script with a lot of static data in it. The CPU usage is mostly unchanged, except that k6 initializes test runs a lot faster. All of those benefits will be most noticeable if k6 is used with big number of VUs (1k+). More performance comparisons can be found in #1167.

JSON export of the end-of-test summary report (#1168)

This returns (from the very early days of k6) the ability to output the data from the end of test summary in a machine-readable JSON file.
This report can be enabled by the --summary-export <file_path> CLI flag or the K6_SUMMARY_EXPORT environment variable. The resulting JSON file will include data for all test metrics, checks and thresholds.

New CSV output (#1067)

There is an entirely new csv output that can be enabled by using the --out csv CLI flag. There are two things that can be configured: the output file with K6_CSV_FILENAME (by default it's file.csv), and the interval of pushing metrics to disk, which is configured with K6_CSV_SAVE_INTERVAL (1 second by default). Both of those can be configured by the CLI as well: --out csv=somefile.csv will output to somefile.csv and --out file_name=somefile.csv,save_interval=2s will output again to somefile.csv, but will flush the data every 2 seconds instead of every second.

The first line of the output is the names of columns and looks like:

metric_name,timestamp,metric_value,check,error,error_code,group,method,name,proto,status,subproto,tls_version,url,extra_tags
http_reqs,1573131887,1.000000,,,,,GET,http://httpbin.org/,HTTP/1.1,200,,,http://httpbin.org/,
http_req_duration,1573131887,116.774321,,,,,GET,http://httpbin.org/,HTTP/1.1,200,,,http://httpbin.org/,
http_req_blocked,1573131887,148.691247,,,,,GET,http://httpbin.org/,HTTP/1.1,200,,,http://httpbin.org/,
http_req_connecting,1573131887,112.593448,,,,,GET,http://httpbin.org/,HTTP/1.1,200,,,http://httpbin.org/,

All thanks to @Sirozha1337!

JSON output optimizations (#1114)

The JSON output no longer blocks the goroutine sending samples to the file, but instead (like all other outputs) buffers the samples and writes them at regular intervals (100ms and is currently not configurable). It also uses a slightly faster way of encoding the data, which should decrease the memory usage by a small amount.

Another improvement is the ability to compress the generated JSON file by simply adding .gz to the end of the file name. Compressed files are typically 30x smaller.

InfluxDB output improvements (#1113)

The InfluxDB output has been updated to use less memory and try to send smaller and consistent chunks of data to InfluxDB, in order to not drop packets and be more efficient. This is primarily done by sending data in parallel, as this seems to be better from a performance perspective, and more importantly, queuing data in separate packets, so that we don't send the data for a big time period all at once. Also, the used library was updated, which also decreased the memory usage.

Two new options were added:

• K6_INFLUXDB_PUSH_INTERVAL - configures at what interval the collected data is queued to be sent to InfluxDB. By default this is "1s".
• K6_INFLUXDB_CONCURRENT_WRITES - configures the number of concurrent write calls to InfluxDB. If this limit is reached the next writes will be queued and made when a slot is freed. By default this is 10.

console is now available in the init context (#982):

This wasn't supported for the longest time, which made debugging things outside of VU code much harder, but now it's here! 🎉

In order to get this feature shipped in a timely manner, it currently has a known bug. The output of console calls in the init context will always be written to the stderr, even if the --console-output option is specified. This bug is tracked in #1131

HTTP response body decompression with multiple layered algorithms (#1125)

In v0.25.0 compressing bodies was added and it had support for multiple layered algorithms. Now this is also true for decompressing bodies when k6 gets them as responses.

New optional count column in the end-of-test summary (#1143)

The --summary-trend-stats now also recognizes count as a valid column and will output the count of samples in all Trend metrics. This could be especially useful for custom Trend metrics, since with them you no longer need to specify a separate accompanying Counter metric.

Docker Compose refactor (#1183)

The example docker-compose that enabled easy running of InfluxDB+Grafana+k6 was refactored and all the images were updated to use the latest stable versions.

Thanks, @KajdeMunter!

Also the k6 Dockerfile Alpine version was bumped to 3.10. Thanks @dmitrytokarev!

http.batch() improvements and optimizations (#1259)

We made several small improvements to the mechanism for executing multiple HTTP requests simultaneously from a single VU:

• Calling http.batch() should now be more efficient, especially for many requests, because of reduced locking, type conversions, and goroutine spawning.
• The default value for batchPerHost has been reduced from 0 (unlimited) to 6, to more closely match browser behavior. The default value for the batch option remains unchanged at 20.
• Calling http.batch(arg), where arg is an array, would now return an array. Previously, this would have returned an object with integer keys, as explained in #767... Now http.batch() will return an array when you pass it an array, and return an object when you pass an object.

UX

• Better timeout messages for setup and teardown timeouts, including hints on how to fix them. (#1173)
• When a folder is passed to open(), the resulting error message will now include the path to the specified folder. (#1238)
• The k6 version output will now include more information - the git commit it was built from (in most cases), as well as the used Go version and architecture. (#1235)

Bugs fixed!

• Cloud: Stop sending metrics to the cloud output when the cloud returns that you have reached the limit. (#1130)
• JS: Fail a check if an uncaught error is thrown inside of it. (#1137)
• HTTP: Replace any user credentials in the metric sample tags with * when emitting HTTP metrics. (#1132)
• WS: Many fixes:
  • return an error instead of panicking if an error occurs during the making of the WebSocket connection (#1127)
  • calling the error handler on an error when closing the WebSocket, instead of calling with a null (#1118)
  • correctly handle server initiated close (#1186)
• JSON: Better error messages when parsing JSON fails. Now telling you at which line and row the error is instead of just the offset. Thanks, @openmohan! (#905)
• HTTP: Use Request's GetBody in order to be able to get the body multiple times for a single request as needed in 308 redirects of posts and if the server sends GOAWAY with no error. (#1093)
• JS: Don't export internal go struct fields of script options.(#1151)
• JS: Ignore minIterationDuration for setup and teardown. (#1175)
• HTTP: Return error on any request that returns 101 status code as k6 currently doesn't support any protocol upgrade behavior. (#1172)
• HTTP: Correctly capture TCP reset by peer and broken pipe errors and give them the appropriate error_code metric tag values. (#1164)
• Config: Don't interpret non-K6_ prefixed environment variables as k6 configuration, most notably DURATION and ITERATIONS. (#1215)
• JS/html: Selection.map was not wrapping the nodes it was outputting, which lead to wrongly using the internal Goquery.Selection instead of k6's Selection. Thanks to @MMartyn for reporting this! (#1198)
• HTTP: When there are redirects, k6 will now correctly set the cookie for the current URL, instead of for the one the current response is redirecting to. Thanks @dimatock! (#1201)
• Cloud: Add token to make calls to the cloud API idempotent. (#1208)
• Cloud: Improve aggregation of HTTP metrics for requests to different URLs, but with the same explicitly set name tag. (#1220)
• Cloud: Fix a bug where you weren't able to run a script, outputting to cloud, if it was using the shortcut URLs for github/cdnjs. (#1237)
• Config: The previous default value for batchPerHost of 20 wasn't propagated properly and was instead 0. (#1264)

Internals

• CI: Stop using external service for testing WebSockets (#1138) and remove the last use of the external httpbin.org. (#1213)
• Switched to Go 1.13.5 for building and testing k6, removed official support for 1.11.
• CI: Fix a test on MacOS. (#1142)
• CI: Fixing flaky tests. (#1149, #1223)
• Drop an external dependency for getting user's configdir. (#1162)
• Use bitmask for checking whether system tags are enabled, adding some small speedup where this is required. (#1148)
• Update envconfig as it was very old and the newer versions had fixes and features we want. (#1214)
• Metrics: Emit iterations as part of netext.NetTrail, instead of as a standalone one. Also cutting down on amount of bytes we sent to the cloud output. (#1203)
• JS: goja has been updated to the latest master version (commit 007eef3) (#1259)
• All official binary packages now are built with -trimpath and CGO_ENABLED=0. Previously the GitHub release assets were built with CGO_ENABLED=0, making them unsuitable for non-glibc systems (like Alpine Linux). (#1244, #1245)

Breaking changes

• The output of k6 version is different. It now contains not only the k6 version number, but also information about the git commit, build date, Go version and system architecture. For example, if previously the output of k6 version looked like k6 v0.25.1, now it is like this: k6 v0.26.0 (2019-12-16T10:58:59+0000/v0.26.0-0-gaeec9a7f, go1.13.5, linux/amd64). (#1235)

• We had to make a few minor breaking changes in the course of improving http.batch() (#1259):

  • The default value for batchPerHost has been reduced from 0 (unlimited) to 6, to more closely match browser behavior. The default value for the batch option remains unchanged at 20.
  • Calling http.batch(arg), where arg is an array, would now return an array. Previously, this would have returned an object with integer keys, as explained in #767... Now http.batch() will return an array when you pass it an array, and return an object when you pass an object.