HOWTO: Run regression tests

There are three main types of regression tests on CBTOOL. These tests cost money, time or both :-)

If you just fixed (or implemented a new) "core" feature, running the first regression test type should be enough.
If you added a new Cloud Adapter, the execution of the second regression test type will be required, but only applied to your new Cloud model. Exceptions to this rule are changes to the files cbtool/lib/clouds/libcloud_common.py and cbtool/lib/clouds/shared_functions.py: these will require a regression test over all Cloud models, which should set you back in a couple of tens of US$.
If you just added a new Virtual Application type, then you will have to run the third regression test type, applied only to your new VApp type. Exception to this rule are changes to the file cbtool/scripts/common/*: these will require a regression test over all Virtual Application types.

a) cd cbtool/regression; ./regression_test.py make;

b) ../cb --soft_reset --config configs/regression/simulated.txt --trace regression/regression_test_experiment_plan.txt 2>&1 > ~/teste.txt

c) ./regression_test.py validate ~/teste.txt

Tests the functionality of the cloud adapter by:

I) connecting to the cloud

II) creating a single instance (without checking network connectivity to the instance)

III) creating a second instance and checking if the ssh pubkey was properly injected

IV) creating a third instance with a volume attached

V) capturing this last instance with the image name "regressiontest"

VI) creating a fourth instance using the newly captured "regressiontest" image name

VII) creating a fifth instance pointing to a non-existent image (should fail)

VIII) creating a sixt instance with a valid image name, but forcing a failure at the very end of the creation.
After each test, CB checks if the number of images reported on its Object Store (Redis) and directly by querying the cloud is the same, and then removes the instance in case of success.
The last two tests ensure that instance cleanup upon failure is being properly executed.
Total expected run time is ~ 1.5 hours (for all Cloud Models).
In order to run this regression test, perform the following steps :

a) mkdir ~/multicloud_regression; cp cbtool/configs/regression/* ~/multicloud_regression/ (ONLY ONCE)

b) modify the files there with your credentials and IPs. (ONLY ONCE)

c) cd cbtool/regression; ./real_multicloud_regression.py configs/../../multicloud_regression/ sim,pcm,pdm,nop,osk,ec2,gce,do,slr high

d) just inspect the output in tabular format, looking for "PASS" for every line and column

Given an already attached cloud, and a list of Virtual Application types supplied it will attempt to:

I) Deploy a VApp AI type

II) Collect the management metrics samples

III) Collect at least N application (runtime) metrics (default N=3)

IV) Remove the AI

V) Collect the detachment management metric samples
Total expected run time is ~ 1.2 hours (when using the relatively fast "pure" Docker adapter).
In order to run this regression test, perform the following steps:

a) cd cbtoo/regression

b) ./real_application_regression_test.py --types nullworkload,btest,fio,ddgen,filebench,iperf,netperf,nuttcp,xping,unixbench,cassandra_yscb,redis_ycsb,mongodb_ycsb,hadoop,giraph,hpcc,linpack,parboil

c) just inspect the output in tabular format, looking for "PASS" for every line and column

Provide feedback