Skip to content

Cloudberry Middleware

Jump to bottom Edit New page
Jianfeng Jia edited this page Feb 3, 2017 · 4 revisions

Neo

This module is the entry point to the client. It sets up the web server that wraps up the middleware logics. The server is implemented by using Play Framework.

conf

This folder contains the following configuration files

  • routes : defines the HTTP entries.
  • logback.xml : defines the log level.
  • application.conf : defines the develop application related configurations.
  • production.conf : the same as application.conf, but is used in the online machines.

app

The server related Scala codes.

controllers

It defines a web server application. The HTTP entries functions defined in the routes are implemented here.

views

It contains the *.scala.html scripted HTML files that will be rendered by Play.

actor

[To be changed] Currently, it contains a NeoActor which translate the JSON request from the web page to the Cloudberry JSON request. It's mainly used to simplify the front-end JS logics since the writer was more comfortable with the strong-typed language. It can be moved to the web page and let the JS send the Cloudberry request directly.

db

It connects the AsterixDB and checks if the berry.meta dataset is there. The metadata will be loaded once into the memory and will be created if not found.

models

Some utility case classes.

public

The frontend resource folder. Please read the frontend documentation.

Zion

Zion contains the kernel of the middleware work. It is composed of the following packages:

actor

It has three types of actors to handle all the workflows.

BerryClient

Each web connection creates one BerryClient that utilizes the JSONParser and QueryPlanner to get a group of AQLs for DataSetAgent to execute. It also takes care of the slicing logic by updating the state of the actor.

DataSetAgent

Each AsterixDB dataset is connected to one unique DataSetAgent. It runs AQLs queries and updates on that dataset. The uniqueness guarantees the read and update consistency.

DataStoreManager

One global actor that sync with the berry.meta AsterixDB dataset which stores the view description and relations.

common

Configuration file (Asterix URL, timeouts)

model

The object model codes

datastore

It defines the interface of the datastore related query model.

imple

  • AQLGenerator: It parses a Query by calling generate function that generates a correspondent AQL query with some syntax validation. It uses AQLFuncVisitor util to handle functions such as relation functions(e.g., contains, in) and aggregation functions(e.g., count, min, max). Reference 3

  • JSONParser: It parses a given Query to a JSON record. In addition, because query is part of the DataSetInfo which requires to be serialized and deserialized to and from AsterixDB, we also implement the write interface which can convert a Query back to a JSONRecord. Reference 1, 2

  • DataSetInfo: It is the dataset metadata that contains Schema, CreateQuery(if it is a view), statistical information(e.g., creation time, update time, cardinality). We implemented the corresponding JSON Formater to serialize/deserialize a DataSetInfo to/from a JSONRecord, so it can be stored into AsterixDB. Reference 1, 2.

schema

Schema model: Query, Schema, Functions types. Reference 2

References:

  1. JSON Serializer / Deserializer: https://www.playframework.com/documentation/2.5.x/ScalaJsonCombinators#JSON-Reads/Writes/Format-Combinators
  2. Query Model Doc: https://docs.google.com/document/d/1-xTBxdIg_y_nHMpztZdEG3i_IOpeKhwIT6UQJVDL0UM/edit?usp=sharing
  3. AsterixDB doc (AQL language): https://ci.apache.org/projects/asterixdb/aql/manual.html
  4. Test cases: https://twitter.github.io/scala_school/specs.html
Add a custom footer
Clone this wiki locally