Skip to content

Monitoring Template Language

Jump to bottom
Vincent Zurczak edited this page Jul 31, 2015 · 5 revisions

Monitoring Template Language

Roboconf Specification Request: 232

Status: Archived (2015-07-31)

Editors:

Revisions

WARNING:
This specification is a DRAFT: a work in progress that is intended to be
published, which may or may not be ready for adoption. It should not
necessarily be considered factual or authoritative.

Introduction

This page proposes a template notation to generate Roboconf monitoring files. This language is needed to provide the Roboconf monitoring support feature, as requested by issue 155.

Context

While Roboconf can manage the deployment of applications on various cloud infrastructures, the physical and virtual machines that are part of the application still need to be monitored. This monitoring task is very specific to the actual infrastructure, and operators need to collect numerous data from the physical and virtual machines, components, instances and services deployed by Roboconf.

However operative maintenance is very touchy: operators use dedicated monitoring tools (e.g. Nagios, Shinken, Cacti …). They do not want to be distracted by (yet) another tool that messes up their carefully written configuration files with generated and unpredictable content. Instead of intruding the operator's universe, Roboconf must provide a simple and universal way to present relevant data to the monitoring tools. Operators are then free to extract the data they want, whenever they want.

The proposed Roboconf Monitoring Template Language must allow the extraction of every Roboconf model data (Application, Component, Instance …). Those data may be presented to multiple monitoring tools, that accept various input formats. Thus, the template language must provide a universal and extensible way to format data.

Formatted data are periodically written to files by the Roboconf Deployment Manager (DM). The operators/monitoring tools are then able to read those files, and/translate then into their native input format, and to act accordingly. So output files might be written into the monitoring native syntax, but most of the time, as more tool-specific information is needed, there are written as a pivot easy-to-parse intermediate file.

![Roboconf monitoring overview][overview]
**Roboconf monitoring overview**

Application monitoring

The monitoring of Roboconf applications can be configured during its execution. It is extended, at runtime, with new monitoring templates. The Deployment Manager register those templates in the application configuration directory (where application resources are extracted and stored).

![Runtime application monitoring][appMonitoring]
**Runtime application monitoring**

Generated files

When an application is configured to produce monitoring files, the DM generates and updates files in the application configuration directory. For instance an application named MyApp will have its generated monitoring files located in the monitoring-generated/MyApp/ directory. If the application configuration directory is /tmp (the default), then the generated monitoring files of this same application will be located in /tmp/monitoring-generated/MyApp/.

Each template file generates one output monitoring file. The exact name of the output file depends upon the name of the template file, which must end with a .tpl extension. For instance, a template file named nagios.conf.tpl will generate a file named nagios.conf:

monitoring-template/
   |- MyApp/
     |- nagios.conf.tpl
     |- config.json.tpl
     |- config.properties.tpl
monitoring-generated/
   |- MyApp/
     |- nagios.conf
     |- config.json
     |- config.properties

Template language overview

The Roboconf Monitoring Template Language must allow to address all the possible configuration input formats of various monitoring tools. It must not be tight to a specific configuration syntax. Quite the opposite: the template language must be universal and extensible, in order to cover changes in the monitoring tools configuration syntax, or to be able to feed newcomers.

The main concept is that a template file (TPL) is written in the target syntax, and contains special instructions, recognized by the template engine. These instructions will be substituted by some Roboconf application monitoring data (e.g. IP address, hostname …).

The Roboconf Monitoring Template Language relies on two things:

  • a hairy à la mode template engine, {{ mustache }}, which is used to define the layout of the generated monitoring files. In practice, an extension of {{ mustache }}, called handlebars, is used, as it provides more syntactic sugar, better expressiveness, and less tag nesting.
  • a monitoring data structure that can represent all the various Roboconf model entities (Application, Component, Instance…). Those structured data feed the template with Roboconf objects (instance name, path & status, exported configuration properties…), so they can be written down in the generated monitoring output file.

The {{ mustache }} template engine.

The Roboconf monitoring engine reads templates written using an extended mustache syntax. It allows to generate files in various output formats, e.g. JSON, properties…

Here is a quick example of a template that, for each root instance of the application, outputs its path, status and IP address:

{
  "VMs": [
    {{#instanceOf VM}}
    {
      "path":   "{{path}},
      "status": "{{status}}",
      "ip":     "{{exports.ip}}"
    },
    {{/instanceOf}}
  ]
}

For instance, when applied to the Roboconf "Getting Started Tutorial" application, this template could produce the following output:

{
  "VMs": [
    {
      "path":   "/Apache VM",
      "status": "DEPLOYED_STARTED",
      "ip":     "192.168.0.12"
    },
    {
      "path":   "/Tomcat VM 1",
      "status": "DEPLOYED_STARTED",
      "ip":     "192.168.0.29"
    },
    {
      "path":   "/Tomcat VM 2",
      "status": "DEPLOYED_STARTED",
      "ip":     "192.168.0.30"
    },
    {
      "path":   "/MySQL VM",
      "status": "DEPLOYED_STARTED",
      "ip":     "192.168.0.57"
    },
  ]
}

This section does not intend to provide a complete description of the mustache template engine syntax, but rather to show how it can actually be used to generate monitoring files for Roboconf applications. The mustache manual details more precisely all the syntactic constructions you can use in your templates.

You may also have a look at the documentation of handlebars.js template language, that extends and stretches the mustache, and at its Java implementation handlebars.java, which is internally used by the Roboconf monitoring support engine.

The following section details the underlying monitoring model: which mustache tags are defined, and how you can refer to them.

Roboconf Monitoring Data Model

The Roboconf template language must allow to access to all the entities of a Roboconf application model. First class entities are Application and Instances.

Application

The Roboconf application can be accessed by using the {{application}} tag. Inside a monitoring, this tag is always defined to the application being monitored. Within this tag, it is possible to access several properties of the Roboconf application:

Syntax Description
Navigation:
{{rootComponents}} The names of the root components of the application.
Type = list of String
{{rootInstances}} The root instances of the application.
Type = list of Instances
Attributes:
{{name}} The name of the application.
Type = String
{{qualifier}} The qualifier of the application.
Type = String
{{description}} The description of the application.
Type = String
{{dslId}} The DSL identifier of the application.
Type = String
{{namespace}} The name space of the application.
Type = String

Open topic: multi-application monitoring
May we provide an access to all the applications managed by the DM from one template, e.g. {{applications}} (type: list of Applications) ? With this feature we could write generic templates that enables monitoring for a family of/all applications.

Instance

When monitoring an application, instances are first class citizens. It is indeed the main source of information, and directly reflects the actual state of the living application.

Inside the {{application}} tag, instances can be accessed via the {{rootInstances}} tag, which list the root instance of the application. From a root instance, it is possible to navigate through the instance hierarchy via the {{parent}} and {{children}} tags. Those tags allows to access to the entire instance space of the application. It is sometimes more convenient to list leaf instances, or instance of a specific component. The instance requests section describes a syntactic sugar that allows such a quick and useful access to instances.

Syntax Description
Navigation:
{{parent}} The parent instance.
Type = Instance. Empty if this instance is a root instance.
{{children}} The children instances.
Type = list of Instances. Empty if this instance has no children.
{{component}} The name of the actual component type of this instance.
Type = String
Attributes:
{{name}} The name of the instance.
Type = String
{{path}} The path of this instance.
Type = String
{{status}} The current status of the instance.
Type = String, one of {"NOT_DEPLOYED", "DEPLOYING", "DEPLOYED_STOPPED", "UNRESOLVED", "STARTING", "DEPLOYED_STARTED", "STOPPING", "UNDEPLOYING", "PROBLEM"}
{{isStable}} Flag indicating if the current status of the instance is a stable one.
Type = boolean
{{exports}} The exports overridden by this instance.
Type = Map<String, String>
The iterating section explains how to extract information from such maps.
{{data}} Custom data of this instance.
Type = Map<String, String>
The iterating section explains how to extract information from such maps.

Instance requests

Iterating over the {{rootInstances}} of an application, and then navigating through the instance hierarchy is quite verbose and not very natural. We would like, as an example, to list all the instances of type VM, or instances which exposes the Storage facet.

The Roboconf template language defines a special instanceOf tag that allows to select instances that are descendants, either directly or indirectly, of the provided component or facet name.

Open topic: multiple inheritance contraint
May we provide a way to select instances according to multiple criteria in a single instruction, e.g. {{#instanceOf VM & Storage}}, or {{#instanceOf Apache | Tomcat}} ?

Open topic: complex contraints
It may be necessary to allow complex constraints, based on the properties of the components/facets implemented by the instances, e.g. {{#instanceOf VM with installerName==bash}}, or {{#instanceOf JEE with exportedVariables.jee.version>=1.4}}. This syntax may also be useful in other situations, as the selection of the children of an instance: {{children with status==DEPLOYED_STARTED}}.

Example:

<div id="storage">
{#instanceOf Storage}
  <h3>Instance name: {{name}}</h3>
  <ul>
    <li>Status: {{status}}</li>
    <li>Component: {{component}}</li>
  </ul>
{/instanceOf}
</div>

… could generate the following monitoring file:

<div id="storage">
  <h3>Instance name: MySQL VM</h3>
  <ul>
    <li>Status: DEPLOYED_STARTED</li>
    <li>Component: MySQL</li>
  </ul>
  <h3>Instance name: PostgreSQL VM</h3>
  <ul>
    <li>Status: DEPLOYED_STOPPED</li>
    <li>Component: PostgreSQL</li>
  </ul>
</div>

As a side note, and as you can see right above, Roboconf monitoring templates can be used to generate human-readable documentation too, not just configuration files.

Syntactic constructions

The following sections describe how the {{ mustache }} and Handlebars syntaxes can be applied to solve/ease many situations. They are not part of the specification itself, but are rather reminders of how you can use those tools.

Path tags

The Handlebars template engine allows to use tags with are constructed as paths, e.g {{exports.ip}}, or {{application.rootInstances.Apache VM}}. Such constructions are very helpful when you want to access a very specific sub-property of a Roboconf model entity.

For instance you can use such a syntax to directly access a data of the parent instance, using the following template:

  {{#instanceOf Tomcat}}
    "{{name}}.ip": {{parent.data.ip}},
  {{/instanceOf}}
Iterating over instance's exports & data

The {{ mustache }} syntax does not allow to iterate over an arbitrary map: the keys must be known in advance. If we know the name of the exported variable, for instance IP or port, that's OK… But what if we want to list all the exported variables?

This section defines a specific notation, inspired from Handlebars.js, which allows to iterate over such loosely-structured maps. The special each tag allows to make the template iterate over all the entries of a map, specified as an argument. Inside this tag, the {{key}} and {{value}} tags allows to access respectively to the key and the value of the entry.

For instance, the following template

  …
  {{#each exports}}
    {{key}} = {{value}}
  {{/each}}
  …
Template comments

Depending on the targeted language, you can insert comments inside your template. For instance, for a Java properties files, you could write:

  # This is a language-specific comment.

The template engine will not interpret the above line as an instruction, and will print it out verbatim. However it may be necessary to comment the template instructions themselves, without the need to include such comments in the final generated file. The {{ mustache }} engine provides a comment syntax that allows you to include such comments:

  {{! This is a template comment! }}

The above line will not be printed out in the generated monitoring file.

Examples

This section recaps all the syntaxes and model properties defined above, and show the Roboconf template language in action through the study of three real-life examples.

Configuration of a single root instance.

In this example, the output monitoring file must contain the configuration of a single root instance of an application. The application contains several root instances, but only a specific one is needed here. This instance has its configuration stored in specific data entries, such as ip, mac…

  {{! This line selects the root instance "Apache VM" of the application}}
  {{application.rootInstances.Apache VM}}
    "ip":       {{data.ip}},
    "mac":      {{data.mac}}
    …
  {{/application}}

Configuration of all instances of a specific component.

In this example, we will show how to the output the configuration of all the instances of a specific component. The application defines a Tomcat component, which may have several instances.

  "tomcats": [
  {{! This line selects all the instances of type "Tomcat" inside the application}}
  {{#instanceOf Tomcat}}
    {{! For each instance, the content of this block is output once.}}
    {
      "name":          {{name}},             {{! The instance name}}
      "ajpPort":       {{exports.ajpPort}}   {{! The ajpPort exported variable}}
    },
  {{/instanceOf}}
  ]

Nesting instance selections

In this more intricate example, we would like, for each instance a specific component (here: VM), output the configuration of all its children with a specific facet (here: Storage).

![Storage application layout][storageAppLayout]
**Storage application layout** 
  "allStorages": [
  {{! This line selects all the instances of type "VM" inside the application}}
  {{#instanceOf VM}}
    {{! For each instance, the content of this block is output once.}}
    {
      "vm.name": "{{name}}",
      "storages": [
        {{! This line selects all the instances of type "Storage", children of this instance}}
        {{#instanceOf Storage}}
          {{! For each storage, output its name and capacity.}}
          {
            "name":     "{{name}}",
            "capacity": "{{exports.capacity}}"
          },
        {{/instanceOf}}
      ]
    },
  {{/instanceOf}}
  ]

For an application with the layout described by the figure above, the output of this template could look like this:

  "allStorages": [
    {
      "vm.name": "Big Storage",
      "storages": [
        {
          "name":     "Huge but slow disk",
          "capacity": "500TB"
        },
        {
          "name":     "Big and fast disk",
          "capacity": "5TB"
        },
      ]
    },
    {
      "vm.name": "Small Storage",
      "storages": [
        {
          "name":     "Big disk",
          "capacity": "1TB"
        },
        {
          "name":     "Solid state",
          "capacity": "200GB"
        },
      ]
    },
  ]

Coming soon…

Clone this wiki locally