Keep polishing the QuickStart page (#335)

Author: JavierJia

docs/README.md

@@ -30,6 +30,7 @@ navbar-links:
   Demo :
     - TwitterMap: "http://cloudberry.ics.uci.edu/demos/twittermap/"
   Resources:
+    - Quick Start: "quick-start"
     - Documentation: "documentation"
     - GitHub: "https://github.com/ISG-ICS/cloudberry"
   Pubs: "pubs"

docs/_config.yml

@@ -12,12 +12,17 @@ subtitle:
 ## Contributors
 * [Jianfeng Jia](https://github.com/JavierJia) (Ph.D. student)
 * [Taewoo Kim](https://github.com/waans11) (Ph.D. student)
+* [Chen Luo](luochen01.github.io) (Ph.D. student)
 * [Hao Chen](https://github.com/haochen07)
-* [Nishad Gurav](https://github.com/nishadg)
+* [Te-Yu Chen](https://github.com/DeyuChen)
 * [Vidhyasagar Thirumaraiselvan](https://github.com/vidhya567)
+* [Vignesh Sankar](https://github.com/vignesh-sankar)
 * [Shengjie Xu](https://github.com/HotLemonJuice)
+* [Liangju Chu](https://github.com/liangjuc)
+* [Sicong Liu](https://github.com/lsclovecode)
 
 ## Earlier Contributors
+* [Nishad Gurav](https://github.com/nishadg)
 * [Aishwarya Kapse](https://github.com/aishwaryakapse)
 * [Chen Li (The student version :-))](https://github.com/JeremyLi28)
 * [Kaiyi Ma](https://github.com/kaiyim)

docs/aboutme.md

@@ -1,73 +1,8 @@
 ---
 layout: page
-title: Quick Start
+title: Documentation
 toc: true
 ---
-## Setup TwitterMap locally
-
-This page includes instructions on how to setup a small instance of the
-[TwitterMap](http://cloudberry.ics.uci.edu/demos/twittermap/) on a local machine.
-
-System requirements:
-
- - Linux or Mac
- - At least 4GB memory
-
-Step 1: Install `sbt` by following the instructions on this [`page`](http://www.scala-sbt.org/release/docs/Setup.html).
-
-Step 2: Clone the codebase.
-
-```
-shell> git clone https://github.com/ISG-ICS/cloudberry.git
-```
-
-Suppose the repostory is cloned to the folder `~/cloudberry`.
-
-Step 3: Use the following steps to install an AsterixDB cluster on the local machine in order to run the Cloudberry middleware.  
-
-   1. Install [Docker](https://www.docker.com/products/docker) (version at least 1.10) on the local machine;
-   2. Run the following commands to create an AsterixDB cluster locally:
-
-```
-~> cd cloudberry
-~/cloudberry> ./script/dockerRunAsterixDB.sh
-```
-This command will download and run a prebuilt AsterixDB docker image from [here](https://hub.docker.com/r/jianfeng/asterixdb/). This step may take 5-10 minutes or even longer, depending on your network speed.
-
-Step 4: Run the following command to ingest sample tweets (about 324K) and US population data into AsterixDB.
-
-```
-~/cloudberry> ./script/ingestAllTwitterToLocalCluster.sh
-```
-
-This step is downloading about 70MB of data, and it may take 5 minutes, again, depending on your network speed.  This step is successful after you see a message "Data ingestion completed!" in the shell.
-
-Step 5: Compile and run the Cloudberry server.
-
-```
-~/cloudberry> sbt compile
-~/cloudberry> sbt "project neo" "run"
-```
-
-Wait until the shell prints a message "Server started, use Ctrl+D to stop and go back to the console....".
-
-Step 6: Start the TwitterMap frontend by running the following command in another shell:
-
-```
-~/cloudberry> sbt "project twittermap" "run 9001"
-```
-
-Step 7: Open a browser to access [http://localhost:9001](http://localhost:9001) to see the TwitterMap frontend.  Notice that the first time you open the page, it could take up to several minutes (depending on your machine) to load the front-end data.
-
-**Congratulations!** You have successfully set up TwitterMap using AsterixDB and Cloudberry!
-
-
-
-* Run your own front-end server
-
-TwitterMap is our homemade front-end that shows how to use Cloudberry server. You can implement own front-end service
-and let it talk to Cloudberry to achieve the same interactive user experience.
-
 
 ## Concepts
 
@@ -157,8 +92,9 @@ The following JSON request can be used to register the Twitter dataset inside As
 
 The front-end application can send the ddl JSON file to Cloudberry `/admin/register` path by using `POST` HTTP method.
 E.g., we can register the previous ddl using the following command line:
+
 ```
-curl -XPOST -d @JSON_FILE_NAME http://localhost:9000/berry
+curl -XPOST -d @JSON_FILE_NAME http://localhost:9000/admin/register
 ```
 
 *Note*:
@@ -197,11 +133,18 @@ After defining the dataset, the front-end can `POST` a JSON request to `/berry`
 illustration purpose, clients can use the `curl` command to send the JSON file as following.
 
 ```
-curl -XPOST -d @JSON_FILE --header "Content-Type:application/json" http://localhost:9000/berry
+curl -XPOST -d @JSON_FILE http://localhost:9000/berry
 ```
 
 In the production system, the front-end application can send the request by JavaScripts to the `/berry` path.
-We also provide the websocket connection at `ws://cloudberry_host_name/ws`. It will return the same result as HTTP POST requests.
+We also provide the websocket connection at `ws://cloudberry_host_name/ws`. You can let the front-end to directly talk
+to Cloudberry server in Javascript as following:
+
+```
+var ws = new WebSocket("ws://localhost:9000/ws"");
+```
+
+It will return the same result as HTTP POST requests.
 
 
 A request is composed of the following parameters:
@@ -263,7 +206,14 @@ A request is composed of the following parameters:
 }
 ```
 
-Using `curl` command, you should see the following responses:
+You can test the query by putting the above JSON record into a file and using `curl` command to send it to Cloudberry.
+
+```
+curl -XPOST -d @JSON_FILE http://localhost:9000/berry
+```
+
+You should see the following responses:
+
 ```
 [[
     {"state":6,"hour":"2016-04-09T10:00:00.000Z","count":1},
@@ -310,6 +260,7 @@ Using `curl` command, you should see the following responses:
 ```
 
 The expected results are as following:
+
 ```
 [[
   {"tag":"Zika","count":6},
@@ -339,6 +290,7 @@ The expected results are as following:
 ```
 
 The expected results are as following:
+
 ```
 [[
  {"create_at":"2016-10-04T10:00:17.000Z","id":783351045829357568},
@@ -363,6 +315,7 @@ Cloudberry supports automatic query-slicing on the `timeField`. The front-end ca
 ```
 
 For example, the following query asks the top-10 hashtags with an option to accept an updated results every 200ms.
+
 ```json
 {
     "dataset": "twitter.ds_tweet",
@@ -398,6 +351,7 @@ For example, the following query asks the top-10 hashtags with an option to acce
 ```
 
 There will be a stream of results return from Cloudberry as following:
+
 ```
 [[{"tag":"Zika","count":3},{"tag":"ColdWater","count":1},{"tag":"Croatia","count":1}, ... ]]
 [[{"tag":"Zika","count":4},{"tag":"Croatia","count":1},{"tag":"OperativoNU","count":1}, ... ]]
@@ -495,17 +449,16 @@ queries should be sliced synchronized.
 ```
 
 The response is as following:
+
 ```
 [
   [ {"state":6,"hour":"2016-08-05T10:00:00.000Z","count":1}, {"state":12,"hour":"2016-07-26T10:00:00.000Z","count":1}, ...],
   [ {"tag":"trndnl","count":6},{"tag":"Zika","count":5},{"tag":"ColdWater","count":1}, ...]
 ]
-
 [
   [ {"state":72,"hour":"2016-05-06T10:00:00.000Z","count":1},{"state":48,"hour":"2016-09-09T10:00:00.000Z","count":2}, ...],
   [ {"tag":"trndnl","count":6},{"tag":"Zika","count":6},{"tag":"Croatia","count":1}, ...]
 ]
-
 ...
 ```
 
@@ -558,38 +511,12 @@ The response is as below:
 
 `wrap` transformation is often preferable when the front-end send many different requests in the same WebSocket interface. 
 
-### Advanced users
-
-#### Multi-node AsterixDB cluster
+### Deregister Dataset
 
-Some applications may require a multi-node AsterixDB cluster.
-You can follow the official [documentation](https://ci.apache.org/projects/asterixdb/install.html) to set it up.
-
-After the cluster is set up, you should make the following changes
-
-* Change the AsterixDB NC name for feed connection
-
-In the script `./script/ingestTwitterToLocalCluster.sh`, line 86:
+You may also deregister a dataset from Cloudberry. To do so, you can send the JSON record as following to the `/admin/deregister` path.
 
 ```
-("sockets"="my_asterix_nc1:10001")
-```
-
-where *my_asterix* is the name of your cluster instance, and *nc1* is the name of one NC node.
-
-
-* Modify the AsterixDB hostname
-
-In configuration file `neo/conf/application.conf`, chang the `asterixdb.url` value to the previously set AsterixDB CC RESTFul API address.
-
-```
-asterixdb.url = "http://YourAsterixDBHostName:19002/query/service"
-```
-
-#### Deregister Tweets and US Population Data Models
-
-You may also deregister these data models from the Cloudberry to experiment with other data models. But be careful when doing this as the TwitterMap won't work without these data models.
-
-```
-./script/deregisterTwitterMapDataModel.sh
+{
+    "dataset": "twitter.dsCountyPopulation"
+}
 ```

docs/documentation.md

@@ -3,4 +3,105 @@ layout: page
 title: Quick Start
 ---
 
-## TODO
+## Setup TwitterMap locally
+
+This page includes instructions on how to setup a small instance of the
+[TwitterMap](http://cloudberry.ics.uci.edu/demos/twittermap/) on a local machine. 
+The relation between TwitterMap and Cloudberry is shown in the following figure:
+![architecture][architecture]
+
+System requirements:
+
+ - Linux or Mac
+ - At least 4GB memory
+
+**Step 1**: Install `sbt` by following the instructions on this [`page`](http://www.scala-sbt.org/release/docs/Setup.html).
+
+**Step 2**: Clone the codebase.
+
+```
+shell> git clone https://github.com/ISG-ICS/cloudberry.git
+```
+
+Suppose the repostory is cloned to the folder `~/cloudberry`.
+
+**Step 3**: Use the following steps to install an AsterixDB cluster on the local machine in order to run the Cloudberry middleware.  
+
+   1. Install [Docker](https://www.docker.com/products/docker) (version at least 1.10) on the local machine;
+   2. Run the following commands to create an AsterixDB cluster locally:
+
+```
+~> cd cloudberry
+~/cloudberry> ./script/dockerRunAsterixDB.sh
+```
+This command will download and run a prebuilt AsterixDB docker image from [here](https://hub.docker.com/r/jianfeng/asterixdb/). This step may take 5-10 minutes or even longer, depending on your network speed.
+After it finishes, you should see the messages as shown in the following screenshot:
+![docker][docker]
+
+**Step 4**: Run the following command to ingest sample tweets (about 324K) and US population data into AsterixDB.
+
+
+```
+~/cloudberry> ./script/ingestAllTwitterToLocalCluster.sh
+```
+
+This step is downloading about 70MB of data, and it may take 5 minutes, again, depending on your network speed.  This step is successful after you see a message "Data ingestion completed!" in the shell.
+After it finishes, you should see the messages as shown in the following screenshot:
+![ingestion][ingestion]
+
+**Step 5**: Compile and run the Cloudberry server.
+
+```
+~/cloudberry> sbt compile
+~/cloudberry> sbt "project neo" "run"
+```
+
+Wait until the shell prints the messages as shown in the following screenshot:
+![neo][neo]
+
+**Step 6**: Start the TwitterMap frontend by running the following command in another shell:
+
+```
+~/cloudberry> sbt "project twittermap" "run 9001"
+```
+
+Wait until the shell prints the messages as shown in the following screenshot:
+![twittermap][twittermap]
+
+
+**Step 7**: Open a browser to access [http://localhost:9001](http://localhost:9001) to see the TwitterMap frontend.  Notice that the first time you open the page, it could take up to several minutes (depending on your machine) to show the following webpage:
+![web][web]
+
+
+**Congratulations!** You have successfully set up TwitterMap using AsterixDB and Cloudberry!
+
+
+### Connect to your own AsterixDB cluster 
+
+To connect with an existing AsterixDB cluster, you can modify the AsterixDB hostname in the 
+configuration file `neo/conf/application.conf` and change the `asterixdb.url` value to the AsterixDB hostname.
+
+```
+asterixdb.url = "http://YourAsterixDBHostName:19002/query/service"
+```
+
+### Run your own front-end server
+
+TwitterMap is our homemade front-end that shows how to use Cloudberry server. You can implement own front-end service
+and let it talk to Cloudberry to achieve the same interactive user experience.
+
+### Customize front-end requests
+Read more on [documentation](/documentation) about how to write Cloudberry requests for your own front-end applications.
+
+[architecture]: /img/quick-start-architecture.png
+{: width="800px"}
+[docker]: /img/docker.png
+{: width="800px"}
+[ingestion]: /img/ingestion.png
+{: width="800px"}
+[neo]: /img/neo.png
+{: width="800px"}
+[twittermap]: /img/twittermap.png
+{: width="800px"}
+[web]: /img/web.png
+{: width="800px"}

docs/img/docker.png

@@ -43,7 +43,9 @@ public void finalize() {
 
     public void ingest(String record) throws IOException{
         recordCount++;
-        System.err.println("send record: " + recordCount);
+        if (recordCount % 5000 == 0) {
+            System.err.println("send record: " + recordCount);
+        }
         byte[] b = record.replaceAll("\\s+", " ").getBytes();
         try {
             out.write(b);

docs/img/ingestion.png

@@ -0,0 +1,9 @@
+#!/usr/bin/env bash
+#clean up the existing images
+docker stop -f cc nc1
+docker rm -f cc nc1
+docker volume rm dbstore
+# remove the local image to fetch the newest remote version
+docker rmi jianfeng/asterixdb
+
+

docs/img/neo.png

@@ -21,9 +21,11 @@
 set -o nounset                              # Treat unset variables as an error
 
 ncs=${1:-1}          # the number of NCs in local cluster, default is 1 ncs
-NC_JVM_MEM=1024      # the JVM -Xmx2048m memory budget for each NC. the Unit is in meta bytes
-
+NC_JVM_MEM=2048      # the JVM -Xmx2048m memory budget for each NC. the Unit is in meta bytes
 docName=dbstore
+
+./script/dockerClean.sh
+
 docker volume create --driver local --name $docName
 
 echo "build the cc"