Home

Welcome to the stream-counting wiki!

Storm projects

• [Storm GitHub project|https://github.com/nathanmarz/storm]

Storm examples

• [storm-starter GitHub project|https://github.com/nathanmarz/storm-starter]
• [storm-feeds-example GitHub project|https://github.com/pereferrera/storm-feeds-example]
• [Taking the Emmys by Storm|http://storm.twitsprout.com/]

Setting up a Storm cluster in Forge sandbox

Set up a Zookeeper cluster

1. Download Zookeeper (tar.gz) from [http://zookeeper.apache.org/] and extract it in {code}/mnt/hgfs/workspace/zookeeper-3.4.5{code}
2. Update configuration as described in [Installing Single Mode|http://zookeeper.apache.org/doc/r3.3.3/zookeeperStarted.html#sc_InstallingSingleMode].

Install Storm's dependencies

1. Java 6 (already on sandbox)
2. Python 2.6.6 (already on sandbox)
3. unzip (already on sandbox)
4. [ZeroMQ 2.1.7|http://www.zeromq.org/] by running the following commands {code} yum info e2fsprogs-libs yum info e2fsprogs-devel yum install e2fsprogs-devel yum info uuid yum install uuid yum info uuid-devel yum install uuid-devel cd /lib64 ln libuuid.so.1 libuuid.so yum info libtool yum install libtool yum info autoconf yum info automake cd /mnt/hgfs/workspace/ wget http://download.zeromq.org/zeromq-2.1.7.tar.gz tar -xzf zeromq-2.1.7.tar.gz cd zeromq-2.1.7 ./configure make sudo make install {code}
5. Clone [jzmq GitHub project|https://github.com/nathanmarz/jzmq] at "/mnt/hgfs/workspacejzmq" and install it by running the following commands {code} cd /mnt/hgfs/workspace/jzmq/ ./autogen.sh ./configure make sudo make install export LD_LIBRARY_PATH=/usr/local/lib (or add LD_LIBRARY_PATH to ".bash_profile") {code}

Install Storm

1. Download and extract [Storm|http://storm-project.net/downloads.html] {code} cd /mnt/hgfs/workspace/ wget https://github.com/downloads/nathanmarz/storm/storm-0.8.1.zip {code}
2. Update (or add) "storm.yaml" in "/mnt/hgfs/workspace/storm-0.8.1/conf". See details at section [Fill in mandatory configurations into storm.yaml|https://github.com/nathanmarz/storm/wiki/Setting-up-a-Storm-cluster].

Running Storm cluster in Forge sandbox

Run Zookeeper

Start Zookeeper as {code} /mnt/hgfs/workspace/zookeeper-3.4.5/bin/zkServer.sh start {code}

Run Storm

1. Start Nimbus as {code} /mnt/hgfs/workspace/storm-0.8.1/bin/storm nimbus {code}

2. Start Supervisor as {code} /mnt/hgfs/workspace/storm-0.8.1/bin/storm supervisor {code}

3. Start Storm console application (which can be accessed as [http://pal.sandbox.dev.bbc.co.uk:8080/] {code} /mnt/hgfs/workspace/storm-0.8.1/bin/storm ui {code}

Submitting topology

Submit topology using backtype.storm.StormSubmitter

To submit a topology to Storm cluster, use the following command {code} /mnt/hgfs/workspace/storm-0.8.1/bin/storm jar {code} For example, {code} cd /mnt/hgfs/workspace/storm-feeds-example/target/ /mnt/hgfs/workspace/storm-0.8.1/bin/storm jar storm-feeds-example-0.0.1-SNAPSHOT-jar-with-dependencies.jar datasalt.storm.feeds.FeedTopology {code}

Remove topology

To remove an existing topology, use the following command {code} /mnt/hgfs/workspace/storm-0.8.1/bin/storm kill {code}

For example, {code} /mnt/hgfs/workspace/storm-0.8.1/bin/storm kill feedTopology {code}

Shutdown Zookeeper

Stop Zookeeper as {code} /mnt/hgfs/workspace/zookeeper-3.4.5/bin/zkServer.sh stop {code}