The Anatomy of a Connector

A Connector has only two high-level duties:

1. Sync a local copy of all data down from an external source.
2. Provide bare bones access to the current state and historical archive of that data via events and a RESTful API.

There are additional requirements (such as handling authentication with the external service), but those serve to support one of the high-level duties.

The Bits

A Connector has several parts (using Twitter as an example):

• twitter.js - The main startup module that is run by Core and loads and organizes all the other modules.
• auth.js - The module containing all authentication code (including webserivce endpoints).
• api.js - The data access API. Provides very basic RESTful API access to all stored data.
• sync.js - Contains all logic for robustly syncing data. Handles all aspects of this process including error handling, calming, scheduling, etc.
• webservice.js - Provides a simple HTTP wrapper around sync.js functions so that they can be called by core (via /at, etc)

The Flow

These modules are added dynamically and only as needed. The startup process of the Connector is as follows (again, using Twitter as an example):

1. Core receives a request for one of the Connector's endpoints, buffers it, and spawns a process with node twitter.js.
2. The Connector loads basic modules, creates an empty web server, loads webservice.js (which only handles / at this point).
3. Core writes basic process info (port, workingDirectory, lockerUrl) to the process's stdin (as JSON).
4. The Connector reads the basic process info, moves itself to the provided working directory, initializes the locker client, and loads the me data from me.json.
5. The Connector loads the data access API (via api.js - note: this always happens, regardless of auth state).
6. The Connector calls the authAndRun function of the auth module.
   • If the Connector has already been auth'd, this function calls back synchronously.
   • If the Connector has not been auth'd, it adds the necessary, web service endpoints for authentication. When complete it calls back.
7. When the auth lib calls back, the Connector calls webservice.authComplete, which adds the rest of the sync API endpoints.
8. The Connector listens on a port, and writes process info back to stdout, which indicates to core that it is ready.
9. Core passes on the orignal buffered request and any other requests that have been buffered during the startup process to the Connector.