TwitterMap documentation

TwitterMap

Based on the Play Framework requirement, the HTML source is located in the neo/app/views/index.scala.html. It is a scripted HTML (.scala.html) which will be rendered by the framework. Since we are using Angular to control the main logic, we don't put too many scripts here. The main logic is implemented in the javascript.

The javascript codes located in neo/public/javascripts/ folder. The app.js is the entrance of the js. Each front-end component is implemented as an Angular Directive. The meaning of each folder is introduced in below.

common

The common module defines an Angular service that communicates with the back-end server by using JSON request via web socket connection.

It defines

• a query function that can be called to send the JSON requests to the Neo server;
• a ws.onmessage function that receives the JSON messages from the Neo server and updates the corresponding global values;

It also defines several global values (e.g. mapResults, timeResults, etc) to store the results. The dependent modules UI can be bound to specific values by using Angular watch function

map

The map directive is implemented by extends the existing Angular leaflet-directive. Initially, it loads the state and the county shapes by asking the resource file from Neo server. Then if the map has the zoom-in, zoom-out, or drag actions, it calls query function in common module. It also watches the mapResults values that the draw function will be called once the results has changed.

searchbar

The directive to control the search box.

timeserials

The directive to show the time serial chart that is implemented using dc.js.

sidebar

It controls the hashtag and the sample tweets parts.

More future front-ends

dashboard

An experimental demo to let each state clickable.

AsterixDB Feed

To use AsterixDB’s data feed, we need to open a socket using AQL to listen to connections. Example AQL, see cloudberry/noah/src/main/resources/aql/feed.aql. Then create a socketAdapterClient to connect to AsterixDB’s socket and send records to AsterixDB through the socket.

FeedSocketAdapterClient

FeedSocketAdapterClient could initialize a socket connection with AsterixDB and send records to AsterixDB. It contains three important functions:

• initialize(): should be called after new a FeedSocketAdapterClient object. It sets up socket connection with AsterixDB.
• ingest(String record): sends a record to AsterixDB through the socket.
• finalized(): should be called after the feed ends. It closes the socket.

Both FileFeedDriver and TwitterFeedStreamDriver create a FeedSocketAdapterClient object and call ingest function to send records to AsterixDB.

FileFeedDriver

It feeds data from an adm file to AsterixDB. First, it initializes a FeedSocketAdapterClient. Then, it reads record from file line by line and calls FeedSocketAdapterClient.ingest to send the record to AsterixDB.

To use the FileFeedDriver, run fileFeed.sh

TwitterFeedStreamDriver

This class is the current pipeline which fetches real time twitter data and feeds the data to AsterixDB. The procedure is:

• Use twitter streaming API to fetch real time twitter data.
• For every tweet, geotag it, convert it from json format to adm format.
• Call FeedSocketAdapterClient.ingest to send the record to AsterixDB.

To use TwitterFeedStreamDriver, modify and run streamFeed.sh

Twitter driver documentation: https://docs.google.com/document/d/1j2vXRL8WeSoqzUKb2Kv4sebKHA0rQIvJZviUSH5cAo4/edit

Use Kafka to ingest to AsterixDB

See this page