Add "minimesos logs" command

[riccardomc]

Add minimesos logs command

This PR implements the enhancement in issue #191

Dude, where are my logs?

In a Mesos cluster, the stdout/stderr of tasks are kept in a directory on the agent called sandbox. The content of this directory is exposed through the agent API via the /files/download?path=... endpoint. For more details, see the Mesos documentation.

Here's an example:

$ curl http://172.17.0.8:5051/files/download?path=%2Fvar%2Flib%2Fmesos%2Fagent-1637653773%2Fslaves%2Ff6fbe47d-dbb4-477a-b855-4480b80ddb7b-S0%2Fframeworks%2Ff6fbe47d-dbb4-477a-b855-4480b80ddb7b-0000%2Fexecutors%2Fhttp-ports-static-assigned-to-31002.89452b29-a3bc-11e7-9371-0242ac110007%2Fruns%2F1c6733cd-0c94-476d-a56c-16d86f930495%2Fstdout
[...]
Registered docker executor on 172.17.0.8
Starting task http-ports-static-assigned-to-31002.89452b29-a3bc-11e7-9371-0242ac110007
Web Server started. Listening on 0.0.0.0:80
[...]

The general format is as follows:

<agentServiceUrl>/files/download?path=<workDir>/slaves/<slaveId>/framework/<frameworkId>/executors/<executorId>/runs/<runId>/<filename> 

Access the logs programmatically

Generating the full URL to retrieve the stdout/stderr turned out to be trickier than expected. We need two parts:

1. the agentServiceUrl of the agent on which the task is running
2. the path to the sandbox

Figure out agentServiceUrl

For agentServiceUrl, we match the slaveId in Task objects retrieved in the Mesos master state with the id in each MesosAgent.getState(). Once found the agent, a simple MesosAgent.getServiceUrl() will give us what we need. This required adding parsing of id field in State objects.

Figure out path

It would be tempting to build the path using the components already we already know (i.e. slaveId, frameworkId, ...). However, the runId is only exposed in the agent state. To retrieve this bit of information, we need to parse frameworkId and executorId in Task objects and the directory of the right executor in the executors[] section in the agent state. This is a stripped down example:

{
  "id": "f6fbe47d-dbb4-477a-b855-4480b80ddb7b-S0", (slaveId)
  "flags": {},
  "frameworks": [
    {
      "id": "f6fbe47d-dbb4-477a-b855-4480b80ddb7b-0000", (frameworkId)
      "name": "marathon",
      "executors": [
        {
          "id": "http-ports-static-assigned-to-31002.89452b29-a3bc-11e7-9371-0242ac110007", (executorId/taskId)
          "container": "1c6733cd-0c94-476d-a56c-16d86f930495",
          ---> "directory": "/var/lib/mesos/agent-1637653773/slaves/f6fbe47d-dbb4-477a-b855-4480b80ddb7b-S0/frameworks/f6fbe47d-dbb4-477a-b855-4480b80ddb7b-0000/executors/http-ports-static-assigned-to-31002.89452b29-a3bc-11e7-9371-0242ac110007/runs/1c6733cd-0c94-476d-a56c-16d86f930495",

          "tasks": [
            {
              "id": "http-ports-static-assigned-to-31002.89452b29-a3bc-11e7-9371-0242ac110007",
              "name": "http-ports-static-assigned-to-31002",
              "framework_id": "f6fbe47d-dbb4-477a-b855-4480b80ddb7b-0000",
              "executor_id": "",
              "slave_id": "f6fbe47d-dbb4-477a-b855-4480b80ddb7b-S0",
              "state": "TASK_RUNNING"
            }
         ]
      }
   ]
}

Download the content

Once we have the URL, I simply used Unirest to get the content of the file. This could eventually be improved in terms of memory usage. An idea could be using the offset and length and the endpoint /files/read?path=... described here.

CLI

Look!

$ minimesos --help
[...]
    logs      Fetches the stdout logs of the specified task
      Usage: logs [options]
        Options:
          --stderr
             Fetch the stderr logs instead of stdout
             Default: false
        * --task
             Substring of a task ID
[...]

Tests

All the code was developed with a TDD approach. A full set of test is included. I also run all the test from the IDE and run a few manual test against a minimesos instance.

[frankscholten]

LGTM

[frankscholten]

Thanks @riccardomc!