cloud_plugins.md

Built-in Cloud Plugins

Amazon EC2

mzb_api_ec2_plugin

The module allocates nodes in an Amazon EC2 cloud.

The AWS node images must have Erlang R17+, gcc, gcc-c++, git, and sudo installed. Sudo must be available for non-tty execution; put Defaults !requiretty in /etc/sudoers. The SSH and TCP ports 4801-4804 must be open; MZBench uses them internally to send logs and metrics data from nodes to the server.

There's a set of ready-to-use Amazon Linux images with all necessary dependencies for all availability zones:

Erlang 19:
us-west-2       ami-da24ffba
us-west-1       ami-77571c17
us-east-2       ami-86055fe3
us-east-1       ami-d0efb2c7
eu-west-1       ami-ab8ec7d8
eu-west-2       ami-e2c3d686
eu-central-1    ami-bdaa52d2
ap-northeast-1  ami-7a25841b
ap-northeast-2  ami-99eb3ff7
ap-southeast-1  ami-65c36206
ap-southeast-2  ami-11fac772
ap-south-1      ami-615e2a0e
sa-east-1       ami-a5c955c9

Erlang 18:
us-west-2       ami-ee8d718e
us-east-1       ami-61f11f0c
us-west-1       ami-fc28509c
eu-west-1       ami-4554c136
eu-central-1    ami-8d48a5e2
ap-northeast-1  ami-78a24419
ap-northeast-2  ami-ef579f81
ap-southeast-1  ami-66fd2b05
ap-southeast-2  ami-21634c42
sa-east-1       ami-7798101b

To use one of these images, specify it in the cloud plugin config as image_id:

{image_id, "ami-ee8d718e"}

You can, of course, build your own image based on the requirements listed above. Learn more in the official Amazon docs.

Configuration example:

{cloud_plugins, [{ec2, #{module => mzb_api_ec2_plugin,
                         instance_spec => [
                          {image_id, "ami-ee8d718e"},
                          {group_set, ""},
                          {key_name, "-"},
                          {subnet_id, "-"},
                          {instance_type, "t2.micro"},
                          {availability_zone, "us-west-2a"}
                        ],
                        config => [
                          {ec2_host, "ec2.us-west-2.amazonaws.com"},
                          {access_key_id, "-"},
                          {secret_access_key, "-"}
                         ]
                        instance_user => "ec2-user",
                    }}]},

Minimally, the config requires instance_spec and config keys specified. Learn more about AWS-specific config in the erlcloud documentation.

To make allocated nodes accessible: specify key_name in configuration above, put associated key to your MZBench server and make ssh use it. Usually it is about adding IdentityFile path/to/your/amazonkey.pem to ~/.ssh/config or running ssh-add path/to/your/amazonkey.pem.

Static Cloud

mzb_staticcloud_plugin

The module uses allocated nodes from a list of hosts.

Configuration example:

{cloud_plugins, [{static, #{module => mzb_staticcloud_plugin,
                           hosts => ["123.45.67.89", "hostname"]
                           }}]}

Dummy

mzb_dummycloud_plugin

The module does not allocate any hosts; localhost is used instead.

Configuration example:

{cloud_plugins, [{dummy, #{module => mzb_dummycloud_plugin}}]}

The main difference between dummycloud and staticcloud for localhost is that dummycloud does not provide node exclusivity. It means that you could run several benchmarks on your localhost simultaneously, which is not supported by staticcloud.

Multicloud

mzb_multicloud_plugin

Combine multiple plugins to allocate hosts from multiple sources.

Configration example:

{cloud_plugins, [
        {some_cloud_provider1, ...},
        {some_cloud_provider2, ...},
        {cloud_multi, #{module => mzb_multicloud_plugin,
                          clouds => [
                              {some_cloud_provider1, 3},
                              {some_cloud_provider2, 10}
                      ]}}
]}

Here, if the cluster is allocated using cloud_multi, it will contain 3 nodes from some_cloud_provider1 for every 10 nodes from some_cloud_provider2.

How to Write a Cloud Plugin

Cloud plugin is an Erlang module with at least three methods:

-spec start(Name, Opts) -> PluginRef when
    Name :: atom(),
    Opts :: #{},
    PluginRef :: term().

-spec create_cluster(PluginRef, NumNodes, Config) -> {ok, ClusterID, UserName, [Host]} when
    PluginRef :: term(),
    NumNodes :: pos_integer(),
    Config :: #{},
    ClusterID :: term()
    UserName :: string(),
    Host :: string().

-spec destroy_cluster(ClusterID) -> ok when
    ClusterID :: term().

start : Start a particular instance of the plugin and get an instance reference.

`Name`
:   The name of the particular instance of the plugin specified in the [configuration file](deployment.md#cloud_plugins).

`Opts`
:   Options passed from the server [configuration file](deployment.md#cloud_plugins) for the particular plugin instance.

create_cluster : Allocate the required number of nodes and return a tuple: {ok, ClusterID, UserName, HostList}.

`NumNodes`
:   Number of nodes to allocate.

`Config`
:   Map with keys `user`, `name` and `description`.

`ClusterID`
:   This term will be passed to `destroy_cluster/1` when it's time it deallocate the nodes. Its content is up to the plugin developer.

`UserName`
:   SSH username to connect to the allocated nodes.

`HostList`
:   List of hostnames or IPs of the allocated nodes.

destroy_cluster : Deallocate the required number of nodes and return ok.

`ClusterID`
:   Term returned by `create_cluster/1`.

Using the Cloud Plugin

Specify the plugin module and the path to it in the MZBench config file in the mzbench_api section:

    [
      {mzbench_api, [
        {cloud_plugins, [{my_cloud1, #{module => mycloud_plugin,
                                       Opt1 => Value1,
                                       Opt2 => Value2, ...}},
                         ...
                        ]},
        {plugins_dir, "/path/to/my/plugin"}
        ]},
    ].

The plugin binaries must be placed in a subdirectory inside plugins_dir, e.g. /mycloud-0.1.1/ebin/mycloud.ebin.

!!!note Plugin will be started using application:ensure_all_started/1 just before the benchmark start.