test(perf): new CLI for running and comparing perf tests #1389

10xLaCroixDrinker · 2024-04-17T00:12:46Z

Description

Adds new CLI for running and comparing perf tests.

$ npm run perf -- --help

> @americanexpress/[email protected] perf
> node __performance__/bin/index.js --help

npm run perf -- <command>

Commands:
  npm run perf -- clean                Clean up performance test results
  npm run perf -- compare              Compare two performance test results
  npm run perf -- explain [metric]     Print descriptions of metrics
  npm run perf -- monitor              Monitor One App performance
  npm run perf -- summarize <results>  Summarize performance test results
                                                              [aliases: summary]
  npm run perf -- test                 Run a performance test

Options:
  -h, --help  Show help                                                [boolean]

$ npm run perf -- clean --help

> @americanexpress/[email protected] perf
> node __performance__/bin/index.js clean --help

npm run perf -- clean

Clean up performance test results

Options:
  -n, --dry-run  List files that would be deleted without deleting them[boolean]
  -f, --force    Force deletion of files                               [boolean]
  -q, --quiet    Suppress output                                       [boolean]
  -h, --help     Show help                                             [boolean]

$ npm run perf -- compare --help

> @americanexpress/[email protected] perf
> node __performance__/bin/index.js compare --help

npm run perf -- compare

Compare two performance test results

Options:
  -c, --control        Control test ID or path to control test results
                                                             [string] [required]
  -v, --variant        Variant test ID or path to variant test results
                                                             [string] [required]
      --markdown       Print the results table in markdown
                                                      [boolean] [default: false]
  -d, --description    Include a description of each metric in the results table
                                                      [boolean] [default: false]
  -a, --alpha          The significance level of the test
                                                        [number] [default: 0.05]
  -p, --processors     processors to run on each column of data
  [array] [choices: "sum", "mean", "median", "mode", "min", "max", "p95", "p90"]
                                                       [default: ["mean","max"]]
  -r, --raw            Output metrics unformatted (unitless)
                                                      [boolean] [default: false]
  -m, --metrics        metrics to compare
               [array] [default: Run npm run perf -- explain --defaults to list]
  -C, --control-label  Label for control test in report
                                                   [string] [default: "Control"]
  -V, --variant-label  Label for variant test in report
                                                   [string] [default: "Variant"]
      --show-hidden    Show hidden options                             [boolean]
  -h, --help           Show help                                       [boolean]

$ npm run perf -- explain --help

> @americanexpress/[email protected] perf
> node __performance__/bin/index.js explain --help

npm run perf -- explain [metric]

Print descriptions of metrics

Positionals:
  metric  name of the metric to explain                                 [string]

Options:
      --label     look up metric by label                               [string]
      --defaults  list the default metrics                             [boolean]
  -h, --help      Show help                                            [boolean]

$ npm run perf -- monitor --help

> @americanexpress/[email protected] perf
> node __performance__/bin/index.js monitor --help

npm run perf -- monitor

Monitor One App performance

Options:
  -t, --target      Host of the target metrics service
                                 [string] [required] [default: "localhost:3005"]
  -b, --background  Run the monitoring in the background               [boolean]
  -l, --tail        Tail backgrounded monitoring                       [boolean]
  -k, --kill        Kill the backgrounded monitoring                   [boolean]
  -h, --help        Show help                                          [boolean]

$ npm run perf -- summarize --help

> @americanexpress/[email protected] perf
> node __performance__/bin/index.js summarize --help

npm run perf -- summarize <results>

Summarize performance test results

Positionals:
  results  Test ID or path to test results                   [string] [required]

Options:
      --markdown     Print the results table in markdown
                                                      [boolean] [default: false]
  -d, --description  Include a description of each metric in the results table
                                                      [boolean] [default: false]
  -p, --processors   processors to run on each column of data
  [array] [choices: "sum", "mean", "median", "mode", "min", "max", "p95", "p90"]
                                           [default: ["min","mean","max","p95"]]
  -r, --raw          Output metrics unformatted (unitless)
                                                      [boolean] [default: false]
  -m, --metrics      metrics to compare
               [array] [default: Run npm run perf -- explain --defaults to list]
  -h, --help         Show help                                         [boolean]

$ npm run perf -- test --help

> @americanexpress/[email protected] perf
> node __performance__/bin/index.js test --help

npm run perf -- test

Run a performance test

Options:
      --type        Type of performance test to run
       [string] [required] [choices: "load", "smoke", "soak", "spike", "stress"]
      --target      URL of the target service                           [string]
      --identifier  Additional value included in the test ID            [string]
      --k6args      Additional arguments to pass to k6 run              [string]
  -h, --help        Show help                                          [boolean]

$ npm run perf -- summarize smoke-6.12.1-bdbc8e2-1713265335227

> @americanexpress/[email protected] perf
> node __performance__/bin/index.js summarize smoke-6.12.1-bdbc8e2-1713265335227

┌────────────────────────────────────┐
│ smoke-6.12.1-bdbc8e2-1713265335227 │
├────────────────────────────────────┤
│ Test type: Smoke                   │
│ Completed: Today at 4:02 AM        │
│ Duration 11 seconds                │
│ HTTP Requests: 11                  │
│ Data Received: 119.505kB           │
│ Failed Requests: 0                 │
└────────────────────────────────────┘

┌───────────────────────┬────────────┬────────────┬────────────┬─────────────────┐
│ Metric                │ Min        │ Average    │ Max        │ 95th Percentile │
├───────────────────────┼────────────┼────────────┼────────────┼─────────────────┤
│ Request Duration      │ 11.179ms   │ 13.843ms   │ 17.024ms   │ 16.161ms        │
│ Request Blocked       │ 2.928µs    │ 294.596µs  │ 3.199ms    │ 5.264µs         │
│ TTFB                  │ 11.031ms   │ 13.648ms   │ 16.768ms   │ 16.05ms         │
│ Request Receiving     │ 92.372µs   │ 172.937µs  │ 271.341µs  │ 235.034µs       │
│ Heap Total            │ 87.441MB   │ 87.441MB   │ 87.441MB   │ 87.441MB        │
│ Heap Used             │ 54.717MB   │ 55.506MB   │ 56.61MB    │ 56.61MB         │
│ Used Space            │ 5.549MB    │ 5.557MB    │ 5.563MB    │ 5.563MB         │
│ Event loop Lag        │ 5.115ms    │ 5.423ms    │ 5.856ms    │ 5.856ms         │
│ Process CPU           │ 6.061%     │ 6.076%     │ 6.096%     │ 6.096%          │
│ System CPU            │ 0.78%      │ 0.805%     │ 0.84%      │ 0.84%           │
│ User CPU              │ 5.255%     │ 5.27%      │ 5.281%     │ 5.281%          │
│ GC Rate               │ 14.421µs/s │ 21.441µs/s │ 31.269µs/s │ 31.269µs/s      │
│ GC Duration           │ 55ms       │ 63.437ms   │ 75.25ms    │ 75.25ms         │
│ Circuit Breaker Open  │ 0/s        │ 0/s        │ 0/s        │ 0/s             │
│ Circuit Breaker Close │ 0/s        │ 0/s        │ 0/s        │ 0/s             │
└───────────────────────┴────────────┴────────────┴────────────┴─────────────────┘

$ npm run perf -- summarize smoke-6.12.1-bdbc8e2-1713265335227 --markdown

> @americanexpress/[email protected] perf
> node __performance__/bin/index.js summarize smoke-6.12.1-bdbc8e2-1713265335227 --markdown

smoke-6.12.1-bdbc8e2-1713265335227
Test type: Smoke
Completed: Today at 4:02 AM
Duration 11 seconds
HTTP Requests: 11
Data Received: 119.505kB
Failed Requests: 0

Metric	Min	Average	Max	95th Percentile
Request Duration	11.179ms	13.843ms	17.024ms	16.161ms
Request Blocked	2.928µs	294.596µs	3.199ms	5.264µs
TTFB	11.031ms	13.648ms	16.768ms	16.05ms
Request Receiving	92.372µs	172.937µs	271.341µs	235.034µs
Heap Total	87.441MB	87.441MB	87.441MB	87.441MB
Heap Used	54.717MB	55.506MB	56.61MB	56.61MB
Used Space	5.549MB	5.557MB	5.563MB	5.563MB
Event loop Lag	5.115ms	5.423ms	5.856ms	5.856ms
Process CPU	6.061%	6.076%	6.096%	6.096%
System CPU	0.78%	0.805%	0.84%	0.84%
User CPU	5.255%	5.27%	5.281%	5.281%
GC Rate	14.421µs/s	21.441µs/s	31.269µs/s	31.269µs/s
GC Duration	55ms	63.437ms	75.25ms	75.25ms
Circuit Breaker Open	0/s	0/s	0/s	0/s
Circuit Breaker Close	0/s	0/s	0/s	0/s

$ npm run perf -- compare --control=spike-6.12.1-61d7a9f-1713277160607 --variant=spike-6.12.1-61d7a9f-1713277440024

> @americanexpress/[email protected] perf
> node __performance__/bin/index.js compare --control=spike-6.12.1-61d7a9f-1713277160607 --variant=spike-6.12.1-61d7a9f-1713277440024

┌────────────────────────────────────┬────────────────────────────────────┐
│ Control (c)                        │ Variant (v)                        │
├────────────────────────────────────┼────────────────────────────────────┤
│ Test type: Spike                   │ Test type: Spike                   │
│ Completed: Today at 7:23 AM        │ Completed: Today at 7:27 AM        │
│ Duration 4 minutes                 │ Duration 4 minutes                 │
│ HTTP Requests: 77932               │ HTTP Requests: 80208               │
│ Data Received: 538.279MB           │ Data Received: 553.261MB           │
│ Failed Requests: 0                 │ Failed Requests: 0                 │
│ spike-6.12.1-61d7a9f-1713277160607 │ spike-6.12.1-61d7a9f-1713277440024 │
└────────────────────────────────────┴────────────────────────────────────┘

┌───────────────────────┬─────────────┬─────────────┬────────────────────────┬─────────────┬─────────────┬────────────────────────┬─────────┐
│ Metric                │ Average (c) │ Average (v) │ Average (diff)         │ Max (c)     │ Max (v)     │ Max (diff)             │ p-value │
├───────────────────────┼─────────────┼─────────────┼────────────────────────┼─────────────┼─────────────┼────────────────────────┼─────────┤
│ Request Duration      │ 886.914ms   │ 868.93ms    │ -17.984ms (-2.028%)    │ 6.16s       │ 6.3s        │ +143.476ms (+2.33%)    │ *0.001  │
│ Request Blocked       │ 53.242ms    │ 52.051ms    │ -1.191ms (-2.236%)     │ 6.16s       │ 6.13s       │ -30.005ms (-0.487%)    │ 0.594   │
│ TTFB                  │ 886.799ms   │ 868.827ms   │ -17.972ms (-2.027%)    │ 6.16s       │ 6.3s        │ +143.577ms (+2.331%)   │ *0.001  │
│ Request Receiving     │ 92.265µs    │ 89.336µs    │ -2.929µs (-3.174%)     │ 25.781ms    │ 22.525ms    │ -3.256ms (-12.629%)    │ *0.009  │
│ Heap Total            │ 328.199MB   │ 333.858MB   │ +5.659MB (+1.724%)     │ 641.933MB   │ 537.35MB    │ -104.583MB (-16.292%)  │ 0.618   │
│ Heap Used             │ 205.6MB     │ 193.306MB   │ -12.293MB (-5.979%)    │ 598.694MB   │ 507.381MB   │ -91.313MB (-15.252%)   │ 0.255   │
│ Used Space            │ 6.593MB     │ 6.628MB     │ +35.624kB (+0.54%)     │ 6.868MB     │ 6.951MB     │ +83.2kB (+1.211%)      │ 0.121   │
│ Event loop Lag        │ 345.751ms   │ 229.596ms   │ -116.155ms (-33.595%)  │ 3.35s       │ 3.1s        │ -254.535ms (-7.598%)   │ 0.092   │
│ Process CPU           │ 108.965%    │ 111.532%    │ +2.567% (+2.356%)      │ 131.959%    │ 136.935%    │ +4.977% (+3.771%)      │ 0.404   │
│ System CPU            │ 11.56%      │ 12.097%     │ +0.538% (+4.652%)      │ 16.094%     │ 18.124%     │ +2.03% (+12.612%)      │ 0.091   │
│ User CPU              │ 97.406%     │ 99.435%     │ +2.029% (+2.083%)      │ 117.095%    │ 118.811%    │ +1.716% (+1.466%)      │ 0.473   │
│ GC Rate               │ 54.259µs/s  │ 68.221µs/s  │ +13.963µs/s (+25.734%) │ 124.838µs/s │ 164.576µs/s │ +39.738µs/s (+31.832%) │ *0.002  │
│ GC Duration           │ 11.716ms    │ 17.039ms    │ +5.322ms (+45.428%)    │ 59.5ms      │ 56.125ms    │ -3.375ms (-5.672%)     │ *<0.001 │
│ Circuit Breaker Open  │ 0.059/s     │ 0.056/s     │ -0.002/s (-4.06%)      │ 0.08/s      │ 0.08/s      │ -0/s (-0.188%)         │ 0.332   │
│ Circuit Breaker Close │ 0.038/s     │ 0.044/s     │ +0.006/s (+15.465%)    │ 0.08/s      │ 0.08/s      │ -0/s (-0.006%)         │ *<0.001 │
└───────────────────────┴─────────────┴─────────────┴────────────────────────┴─────────────┴─────────────┴────────────────────────┴─────────┘

$ npm run perf -- compare --control=spike-6.12.1-61d7a9f-1713277160607 --variant=spike-6.12.1-61d7a9f-1713277440024 --markdown

> @americanexpress/[email protected] perf
> node __performance__/bin/index.js compare --control=spike-6.12.1-61d7a9f-1713277160607 --variant=spike-6.12.1-61d7a9f-1713277440024 --markdown

Control (c)	Variant (v)
Test type: Spike	Test type: Spike
Completed: Today at 7:23 AM	Completed: Today at 7:27 AM
Duration 4 minutes	Duration 4 minutes
HTTP Requests: 77932	HTTP Requests: 80208
Data Received: 538.279MB	Data Received: 553.261MB
Failed Requests: 0	Failed Requests: 0
spike-6.12.1-61d7a9f-1713277160607	spike-6.12.1-61d7a9f-1713277440024

Metric	Average (c)	Average (v)	Average (diff)	Max (c)	Max (v)	Max (diff)	p-value
Request Duration	886.914ms	868.93ms	-17.984ms (-2.028%)	6.16s	6.3s	+143.476ms (+2.33%)	*0.001
Request Blocked	53.242ms	52.051ms	-1.191ms (-2.236%)	6.16s	6.13s	-30.005ms (-0.487%)	0.594
TTFB	886.799ms	868.827ms	-17.972ms (-2.027%)	6.16s	6.3s	+143.577ms (+2.331%)	*0.001
Request Receiving	92.265µs	89.336µs	-2.929µs (-3.174%)	25.781ms	22.525ms	-3.256ms (-12.629%)	*0.009
Heap Total	328.199MB	333.858MB	+5.659MB (+1.724%)	641.933MB	537.35MB	-104.583MB (-16.292%)	0.618
Heap Used	205.6MB	193.306MB	-12.293MB (-5.979%)	598.694MB	507.381MB	-91.313MB (-15.252%)	0.255
Used Space	6.593MB	6.628MB	+35.624kB (+0.54%)	6.868MB	6.951MB	+83.2kB (+1.211%)	0.121
Event loop Lag	345.751ms	229.596ms	-116.155ms (-33.595%)	3.35s	3.1s	-254.535ms (-7.598%)	0.092
Process CPU	108.965%	111.532%	+2.567% (+2.356%)	131.959%	136.935%	+4.977% (+3.771%)	0.404
System CPU	11.56%	12.097%	+0.538% (+4.652%)	16.094%	18.124%	+2.03% (+12.612%)	0.091
User CPU	97.406%	99.435%	+2.029% (+2.083%)	117.095%	118.811%	+1.716% (+1.466%)	0.473
GC Rate	54.259µs/s	68.221µs/s	+13.963µs/s (+25.734%)	124.838µs/s	164.576µs/s	+39.738µs/s (+31.832%)	*0.002
GC Duration	11.716ms	17.039ms	+5.322ms (+45.428%)	59.5ms	56.125ms	-3.375ms (-5.672%)	*<0.001
Circuit Breaker Open	0.059/s	0.056/s	-0.002/s (-4.06%)	0.08/s	0.08/s	-0/s (-0.188%)	0.332
Circuit Breaker Close	0.038/s	0.044/s	+0.006/s (+15.465%)	0.08/s	0.08/s	-0/s (-0.006%)	*<0.001

$ npm run perf -- explain http_req_sending

> @americanexpress/[email protected] perf
> node __performance__/bin/index.js explain http_req_sending

┌──────────────────────────────────────────────────────────────┐
│                       http_req_sending                       │
├──────────────────────────────────────────────────────────────┤
│                    Label: Request Sending                    │
│                                                              │
│   Description: time spent sending data to the remote host    │
└──────────────────────────────────────────────────────────────┘

$ npm run perf -- explain --label=ttfb

> @americanexpress/[email protected] perf
> node __performance__/bin/index.js explain --label=ttfb

┌──────────────────────────────────────────────────────────────┐
│                       http_req_waiting                       │
├──────────────────────────────────────────────────────────────┤
│                         Label: TTFB                          │
│                                                              │
│   Description: time spent waiting for response from remote   │
│        host (a.k.a. “time to first byte”, or “TTFB”)         │
└──────────────────────────────────────────────────────────────┘

$ npm run perf -- explain

> @americanexpress/[email protected] perf
> node __performance__/bin/index.js explain

┌───────────────────────┬──────────────────────────────────────────────────┬────────────────────────────────────────────────────┐
│ Label                 │ Metric                                           │ Description                                        │
├───────────────────────┼──────────────────────────────────────────────────┼────────────────────────────────────────────────────┤
│ HTTP Requests         │ http_reqs                                        │ How many total HTTP requests k6 generated          │
│ Request Duration      │ http_req_duration                                │ total time for the request. It's equal to          │
│                       │                                                  │ http_req_sending + http_req_waiting +              │
│                       │                                                  │ http_req_receiving (i.e. how long did the remote   │
│                       │                                                  │ server take to process the request and respond,    │
│                       │                                                  │ without the initial DNS lookup/connection times)   │
│ Request Blocked       │ http_req_blocked                                 │ time spent blocked (waiting for a free TCP         │
│                       │                                                  │ connection slot) before initiating the request     │
│ Request Connecting    │ http_req_connecting                              │ time spent establishing TCP connection to the      │
│                       │                                                  │ remote host                                        │
│ TLS Handshaking       │ http_req_tls_handshaking                         │ time spent handshaking TLS session with remote     │
│                       │                                                  │ host                                               │
│ Request Sending       │ http_req_sending                                 │ time spent sending data to the remote host         │
│ TTFB                  │ http_req_waiting                                 │ time spent waiting for response from remote host   │
│                       │                                                  │ (a.k.a. “time to first byte”, or “TTFB”)           │
│ Request Receiving     │ http_req_receiving                               │ time spent receiving response data from the remote │
│                       │                                                  │ host                                               │
│ Failed Requests       │ http_req_failed                                  │ the rate of failed requests according              │
│ Checks                │ checks                                           │ rate of successful checks                          │
│ VUs                   │ vus                                              │ active virtual users                               │
│ Max VUs               │ vus_max                                          │ max possible number of virtual users               │
│ Data Sent             │ data_sent                                        │ amount of data sent                                │
│ Data Received         │ data_received                                    │ amount of received data                            │
│ Iteration Duration    │ iteration_duration                               │ time to complete one full iteration, including     │
│                       │                                                  │ time spent in setup and teardown                   │
│ Iterations            │ iterations                                       │ aggregate number of times the VUs execute the JS   │
│                       │                                                  │ script                                             │
│ Heap Total            │ nodejs_heap_size_total_bytes                     │ total size of the allocated heap                   │
│ Heap Used             │ nodejs_heap_size_used_bytes                      │ actual memory used during the execution of our     │
│                       │                                                  │ process                                            │
│ Resident Set Size     │ process_resident_memory_bytes                    │ the total memory allocated for the process         │
│                       │                                                  │ execution                                          │
│ External              │ nodejs_external_memory_bytes                     │ memory used by "C++ objects bound to JavaScript    │
│                       │                                                  │ objects managed by V8"                             │
│ Available Space       │ nodejs_heap_space_size_available_bytes           │ process heap space size available from Node.js in  │
│                       │                                                  │ bytes                                              │
│ Used Space            │ nodejs_heap_space_size_used_bytes                │ process heap space size used from Node.js in bytes │
│ Total Space           │ nodejs_heap_space_size_total_bytes               │ process heap space size total from Node.js in      │
│                       │                                                  │ bytes                                              │
│ Event loop Lag        │ nodejs_eventloop_lag_seconds                     │ lag of event loop in seconds                       │
│ Process CPU           │ rate(process_cpu_seconds_total[1m])              │ total user and system CPU usage                    │
│ System CPU            │ rate(process_cpu_system_seconds_total[1m])       │ total system CPU usage                             │
│ User CPU              │ rate(process_cpu_user_seconds_total[1m])         │ total user CPU usage                               │
│ GC Rate               │ rate(nodejs_gc_duration_seconds_sum[1m])         │ rate of garbage collection (GC time spent per      │
│                       │                                                  │ second)                                            │
│ GC Duration           │ histogram_quantile(0.95,                         │ duration of garbage collection runs, 95th          │
│                       │ sum(rate(nodejs_gc_duration_seconds_bucket[1m])) │ percentile                                         │
│                       │ by (le))                                         │                                                    │
│ Circuit Breaker Open  │ rate(circuit{event="open"}[1m])                  │ rate of circuit breaker open events per second     │
│ Circuit Breaker Close │ rate(circuit{event="close"}[1m])                 │ rate of circuit breaker close events per second    │
└───────────────────────┴──────────────────────────────────────────────────┴────────────────────────────────────────────────────┘

$ npm run perf -- explain --processors

> @americanexpress/[email protected] perf
> node __performance__/bin/index.js explain --processors

┌─────────────────┬───────────┬────────────────────────────────────────────────────┐
│ Label           │ Processor │ Description                                        │
├─────────────────┼───────────┼────────────────────────────────────────────────────┤
│ Total           │ sum       │ sum total of all data points in the set            │
│ Average         │ mean      │ arithmetic mean of the data set                    │
│ Median          │ median    │ middle value (or midpoint) after all data points   │
│                 │           │ have been arranged in value order                  │
│ Mode            │ mode      │ the value that appears the most frequently in the  │
│                 │           │ data set                                           │
│ Min             │ min       │ smallest value in the data set                     │
│ Max             │ max       │ largest value in the data set                      │
│ 95th Percentile │ p95       │ the value below which 95% of the data set falls    │
│ 90th Percentile │ p90       │ the value below which 90% of the data set falls    │
└─────────────────┴───────────┴────────────────────────────────────────────────────┘

$ npm run perf -- explain --tests

> @americanexpress/[email protected] perf
> node __performance__/bin/index.js explain --tests

┌─────────────┬───────────┬──────────────────────────────────────────────────────────────┐
│ Label       │ Test Type │ Description                                                  │
├─────────────┼───────────┼──────────────────────────────────────────────────────────────┤
│ Load Test   │ load      │ Scenario: Up to 400 looping VUs for 20m0s over 3 stages      │
│             │           │                                                              │
│             │           │ About: Load Testing is primarily concerned with assessing    │
│             │           │ the current performance of your system in terms of           │
│             │           │ concurrent users or requests per second.                     │
│             │           │                                                              │
│             │           │ https://k6.io/docs/test-types/load-testing                   │
│             │           │                                                              │
│             │           │                                                              │
│ Smoke Test  │ smoke     │ Scenario: 1 looping VUs for 1m0s                             │
│             │           │                                                              │
│             │           │ About: Smoke test is a regular load test, configured for     │
│             │           │ minimal load. You want to run a smoke test as a sanity check │
│             │           │ every time you write a new script or modify an existing      │
│             │           │ script.                                                      │
│             │           │                                                              │
│             │           │ https://k6.io/docs/test-types/smoke-testing                  │
│             │           │                                                              │
│             │           │                                                              │
│ Soak Test   │ soak      │ Scenario: Up to 100 looping VUs for 12h20m0s over 3 stages   │
│             │           │                                                              │
│             │           │ About: Soak testing is concerned with reliability over a     │
│             │           │ long time.                                                   │
│             │           │                                                              │
│             │           │ https://k6.io/docs/test-types/soak-testing                   │
│             │           │                                                              │
│             │           │                                                              │
│ Spike Test  │ spike     │ Scenario: Up to 600 looping VUs for 3m40s over 7 stages      │
│             │           │                                                              │
│             │           │ About: Spike test is a variation of a stress test, but it    │
│             │           │ does not gradually increase the load, instead it spikes to   │
│             │           │ extreme load over a very short window of time. While a       │
│             │           │ stress test allows the SUT (System Under Test) to gradually  │
│             │           │ scale up its infrastructure, a spike test does not.          │
│             │           │                                                              │
│             │           │ https://k6.io/docs/test-types/stress-testing#spike-testing   │
│             │           │                                                              │
│             │           │                                                              │
│ Stress Test │ stress    │ Scenario: Up to 500 looping VUs for 38m0s over 9 stages      │
│             │           │                                                              │
│             │           │ About: The purpose of stress testing is to assess the        │
│             │           │ availability and stability of the system under heavy load.   │
│             │           │                                                              │
│             │           │ https://k6.io/docs/test-types/stress-testing                 │
└─────────────┴───────────┴──────────────────────────────────────────────────────────────┘

There is also a minor refactor in src to remove chalk and use colorette (the same lib used by pino-pretty)

Motivation and Context

First stage in a series of changes to automate performance testing.

The primary goal of the current commands in the CLI is to make it easier to run the tests and evaluate the performance.

How Has This Been Tested?

Running the new commands locally

Types of Changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Documentation (adding or updating documentation)
Dependency update
Security update

Checklist:

My change requires a change to the documentation and I have updated the documentation accordingly.
These changes should be applied to a maintenance branch.
This change requires cross browser checks.
Performance tests should be ran against the server prior to merging.
This change impacts caching for client browsers.
This change impacts HTTP headers.
This change adds additional environment variable requirements for One App users.
I have added the Apache 2.0 license header to any new files created.

What is the Impact to Developers Using One App?

github-actions · 2024-04-17T00:15:25Z

Size Change: 0 B

Total Size: 232 kB

ℹ️ View Unchanged

Filename	Size
`./build/app/app.js`	38.2 kB
`./build/app/app~vendors.js`	129 kB
`./build/app/runtime.js`	5.64 kB
`./build/app/service-worker-client.js`	5.46 kB
`./build/app/vendors.js`	54.1 kB

_{compressed-size-action}

__performance__/bin/commands/compare.js

__performance__/bin/commands/monitor.js

Matthew-Mallimo · 2024-04-18T19:58:08Z

package.json

@@ -110,7 +111,7 @@
    "accepts": "^1.3.8",
    "bytes": "^3.1.2",
    "cacheable-lookup": "^6.1.0",
-    "chalk": "^4.1.2",
+    "colorette": "^2.0.20",


Why the change from chalk to colorette?

For consistency. pino-pretty uses colorette, so most of the log coloring we do is actually already using colorette. It's also not ESM-only and we could actually consume any updates it gets.

…ss/one-app into feature/perf-suite

code-forger · 2024-04-19T14:09:44Z

package.json

    "standard-version": "^9.5.0",
    "tar": "^6.2.1",
    "thread-stream": "^2.4.1",
+    "ttest": "^4.0.0",


Are split2 and ttest intentional installs? These are the two most typo looking package names I've seen in my life

lol, yes they are intentional.

split2 is a fork of split, and ttest is for creating "t-tests"

test(perf): new perf testing CLI

8009be8

10xLaCroixDrinker requested review from a team as code owners April 17, 2024 00:12

github-actions bot added Feature one-app-team-review-requested labels Apr 17, 2024

10xLaCroixDrinker changed the title ~~test(perf): new CLI for runnining and comparing perf tests~~ test(perf): new CLI for running and comparing perf tests Apr 17, 2024

10xLaCroixDrinker added 3 commits April 16, 2024 21:24

test(perf): added --processors option to explain command

7332c56

test(perf): added --tests option to explain command

bff4bf4

Merge branch 'main' into feature/perf-suite

3a4e32e

Matthew-Mallimo reviewed Apr 18, 2024

View reviewed changes

10xLaCroixDrinker added 3 commits April 18, 2024 16:11

refactor(perf): optimize reading results

98a05dd

Merge branch 'feature/perf-suite' of https://github.com/americanexpre…

5fc6c18

…ss/one-app into feature/perf-suite

test(perf): use proxy agent when validating metrics availability

e9c8c81

Matthew-Mallimo approved these changes Apr 19, 2024

View reviewed changes

10xLaCroixDrinker requested a review from a team April 19, 2024 01:59

Merge branch 'main' into feature/perf-suite

c1d1700

code-forger reviewed Apr 19, 2024

View reviewed changes

code-forger approved these changes Apr 19, 2024

View reviewed changes

10xLaCroixDrinker merged commit a07372c into main Apr 19, 2024
9 checks passed

10xLaCroixDrinker deleted the feature/perf-suite branch April 19, 2024 14:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test(perf): new CLI for running and comparing perf tests #1389

test(perf): new CLI for running and comparing perf tests #1389

10xLaCroixDrinker commented Apr 17, 2024 •

edited

Loading

github-actions bot commented Apr 17, 2024 •

edited

Loading

Matthew-Mallimo Apr 18, 2024

10xLaCroixDrinker Apr 18, 2024

code-forger Apr 19, 2024

10xLaCroixDrinker Apr 19, 2024

test(perf): new CLI for running and comparing perf tests #1389

test(perf): new CLI for running and comparing perf tests #1389

Conversation

10xLaCroixDrinker commented Apr 17, 2024 • edited Loading

Description

Motivation and Context

How Has This Been Tested?

Types of Changes

Checklist:

What is the Impact to Developers Using One App?

github-actions bot commented Apr 17, 2024 • edited Loading

Matthew-Mallimo Apr 18, 2024

Choose a reason for hiding this comment

10xLaCroixDrinker Apr 18, 2024

Choose a reason for hiding this comment

code-forger Apr 19, 2024

Choose a reason for hiding this comment

10xLaCroixDrinker Apr 19, 2024

Choose a reason for hiding this comment

10xLaCroixDrinker commented Apr 17, 2024 •

edited

Loading

github-actions bot commented Apr 17, 2024 •

edited

Loading