Guide to Debugging Production API with Logs

Table of Contents generated with DocToc

• Important Notes
  • Use json.tx.txTimestamp
  • Use Grid View
• Trace Builds
  • With contextVersionId
    • Short Version
      • Query
        • Example
    • Detailed Version
      • Query
        • Example
  • With POST /builds/:id/actions/build request TID
    • Short Version
      • Query
        • Example
    • Detailed Version
• Trace Deployments - Example - Example
• Was My Dock Marked Unhealthy?

Guide to tracing API worker operations in Loggly

Important Notes

###Use json.tx.txTimestamp Don't rely on loggly's timestamp value. For performance reasons we batch send logs to loggly. The effect is the loggly-provided timestamp value is unreliable. In grid-view add the json.tx.txTimestamp field as a column and sort to get proper chronologically ordered logs.

###Use Grid View Use grid view and add json.msg and json.module and json.tx.txTimestamp as display columns. Sort by txTimestamp.

Trace Builds

Relevant workers:

• create-image-builder-container
• on-image-builder-container-create
• on-image-builder-container-die

###With contextVersionId Fetch contextVersionId:

• Ex: http://api.runnable.io/instances/?githubUsername=codenow&name=api

Set query time range to include time of build. This could be several days if deploy uses a deduped build that was built several days ago.

####Short Version Query for logs from only the relevant worker modules. This does not include logs from methods in other modules invoked from these worker modules. Quick way to verify success or failure of build logic flow.

#####Query (json.module:"lib/workers/base-worker.js" OR json.module:"lib/workers/create-image-builder-container.js" OR json.module:"lib/workers/on-image-builder-container-create.js" OR json.module:"lib/workers/on-image-builder-container-die.js") AND "{{contextVersionId}}"

######Example Successful build, passing through all workers

####Detailed Version Query for logs from all worker modules and invoked methods in other modules. Useful if failing operation occurs outside of worker.

#####Query

1. Query: json.msg:"CreateImageBuilderContainerWorker.prototype.handle" AND "{{contextVersionId}}"
2. Take json.tx.tid value from first log record and query: json.tx.tid:"{{tid}}"

######Example Successful build, passing through all workers

###With POST /builds/:id/actions/build request TID Copy TID from a PUT /build/:id/actions/build request response header

####Short Version Query for logs from only the relevant worker modules. This does not include logs from methods in other modules invoked from these worker modules. Quick way to verify success or failure of build logic flow.

#####Query (json.module:"lib/workers/create-image-builder-container.js" OR json.module:"lib/workers/on-image-builder-container-create.js" OR json.module:"lib/workers/on-image-builder-container-die.js") AND "{{tid}}"

######Example Successful build, passing through all workers

####Detailed Version Isolate one log using above logging steps, then use steps outlined previous detailed version

Trace Deployments

Relevant workers:

• deploy-instance
• create-instance-container
• on-instance-container-create
• start-instance-container
• on-instance-container-start

Deployments can be initiated from several places.

• Github hooks
• on-image-builder-container-die worker
• POST /instances/ with built build.

Knowing which workers are started & completed can aid in locating a fault in the system. Faults typically include missing images, image transfer failures, repeated docker timeouts, etc.

deploymentUuid is a unique key assigned to all logs across a single run of these workers. Find the deploymentUuid of the deployment you’re interested in tracing. Use datetime ranges and following query.

json.module:”lib/workers/create-instance-container.js” AND json.environment:”{{production or production-beta}}” AND json.msg:”CreateInstanceContainerWorker.prototype.handle” {{instance name}}

#####Example Querying deployments of a given instance, "update-isolation-ui-runnable-angular"

Grab the deploymentUuid from any of the logs and query: (json.module:lib/workers/* AND "{{ deploymentUuid }}" #####Example

Was My Dock Marked Unhealthy?

• https://github.com/CodeNow/devops-scripts/wiki/Guide-to-Debugging-Production-API-with-Logs