Project Blog
Instead of a continuous stream of messages, we decided to send event messages as a report. Each report contains the current priority levels of all sensors. Images are still a separate stream. Once an event is starts, the priority is sampled every second and a report is created until the event ends. Here are some details:
Messages/Sampling:
- When the priority crosses the threshold, an initial report is sent (type == start)
- Every second, a message with current priority levels is sent (type == update)
- When the priority reaches 0 and stays there for 3 seconds, a final report is sent (type == end)
Camera Priority:
- The camera priority increases 100 units/sec (if continuous motion). A calculation is done based on FPS.
- The camera priority decreases at 50 units/sec
Decreasing the Priority:
- A thread runs in 100 ms intervals that decrements the various priorities
- For now, except for the camera (above), each sensor's is decreased by 1.
Server:
- Each priority level is displayed under the image (per node)
- The event duration is displayed when the event ends
We've spent the last few days cleaning up the project code and working on getting a presentation together for April 11th. We met today to discuss our presentation structure and how to setup the demo in class. We're not sure whether we will be implementing the project over the UB wireless infrastructure or if we're going to bring our own wireless router to reduce the possibility of an issue during the demo.
We've finally tied it all together! The visual motion detection and the sensor code has been integrated into the client from their own separate directories. Also, we've added new network event types to handle camera motion detection, which includes passing the frames to the server.
On the server, we've rolled our own dashboard which will display the frames sent from the camera when movement is detected. We're planning on adding other sensor indicators on the server dashboard as well, which will show the current and historical values of the sensors on the phones.
We are currently researching motion detection on Android, including visual motion detection. The goal of the research was to build an application that would track multiple objects on the screen and highlight them as being separate from the background. In addition, the application would need to be able to learn about new objects in the view. For example - if we were monitoring a parking lot -- a car drives into the frame then parks. The application should correctly recognize the initial movement into the frame, but should "learn" about the car and incorporate it into the background if it remains still.
We created a crude motion detection model which performs a diff of two still images taken at intervals using the raw byte images. Applying a difference threshold can tune the performance slightly, but it is still susceptible to false positives. It also gives us no way detecting contiguous objects in the frame or learning about the scene changes over time.
We began to experiment with using OpenCV to do motion detection and we were able to achieve decent results. OpenCV is an open source computer vision library that provides functions for transforming images and video. We were able to detect, isolate, and recognize movement with a real-time video by using the following process:
First we learn the background via the OpenCV BackgroundSubtractionMOG, which will use a gaussian technique with a learning rate to build a model of the static background.
In this case, a pencil has entered the frame.
We convert to grayscale to reduce the number of k values required to compute the foreground mask.
Using the learned background, compute the foreground mask to identify moving objects.
We apply a Gaussian Blur to help define contiguous objects in the following steps.
Next, we identify the contours in the foreground mask.
Here we show the contours overlaid on the original image for reference.
We then create a bounding box around each contour that meets a minimum size threshold that can be tuned for accuracy.
This method works for multiple objects, and will merge them into a single bounding box if they are close enough.
Now that we have highlighted regions moving through the frame, the next step would be implementing some sort of directional motion prediction algorithm to detect where the object is headed. We are researching Kalman filters to provide this functionality.
I forgot to post this earlier, so I’ll combine a couple of notes into one here. Using Android as a platform for unattended sensing to some degree has been like swimming against the current. Android is geared towards providing a rich user experience. In doing so, Android enforces certain rules that have consequences on how we approach certain aspects of our application.
Doing any networking on the UI thread is not permitted. The reason for this is so that apps remain responsive by preventing code that blocks. Similarly, you cannot update the UI from a background thread. This is good design anyways, but it added quite a bit of complication just to get a sample client app up and running. We were able to get past this with a (expanded) callback mechanism between the UI and network modules with a bridge between them. The “bridge” uses the Android method runOnUiThread() to transfer control from the network thread invoking the callback to the UI class handling it.
Memory in an Android device is more limited than in a laptop or PC. Besides a smaller amount of physical memory, Android does not support swapping to disk without some hacking. Because of this, apps are given relatively low maximum heap sizes. For example, by default the max heap size for my phone is 48 MB. This is probably fine for most apps, but we run into some issues with our application. The image processing portion of our project will be reading in very large raw byte arrays from the camera. Storing a few of these can blow out the heap in no time. Also, the max heap size can vary from device to device which we found after our test app started crashing on my phone. This might limit how much we can do on the devices as we are limited, but don’t quite know by how much.
Luckily, there are a couple of things we can do to mitigate this:
- Be efficient. We need to keep memory allocations as small as possible. We need to keep this in mind as we design our detection algorithms.
- The application manifest: Android has an option that can be added to the application manifest to request a larger heap: android:largeHeap="true". This should alleviate the problem on most devices
- Hack Android! In the future, we could modify the OS to give us more memory. Our application is just about the only app that should running and so it should be allowed more memory. Since Android is open source, this is definitely a possibility.
Scott standardized the server by creating generic interfaces and first-order classes that could be extended by different sub-level architectures. Scott created a "lowlevel" network architecture and Rob implemented an "akka" network architecture. We plan on contrasting these different approaches to networking in our report, as they both have different strengths and weaknesses.
Matt continues on his sensor work, which is going really well. He has multiple sensors reading and is researching the use of sound, vibration, and ambient light for detection in addition to the visual motion sensing. The next sensor problem is that the input range is not standardized for each sensor type, and different models of the same type can have different sensitivities. He is also implementing motion detection by using a diff-based approach on images taken with the camera.
Due to other class projects, we held our meeting online in a google hangout. We discussed progress in the network layer -- we've implemented buffering for high-latency connections and we discussed utilizing google protocol buffers in the serialization layer so that we can integrate with libraries written in other languages.
Scott demo'd his basic network server and client. He discussed some of the challenges on the android platform - specifically getting the network events to be recognized on the user thread. His tasks for the next week are to work on a client-side architecture that allows for message passing between the network and user threads.
Rob presented his network architecture which consists of a registration server, management server, and event server. He designed special protocols for each server with the idea that these could eventually be split into different hardware. His tasks for the next week are to take the protocol and translate it into code and modify our server/client to handle the protocols.
Matt demo'd his sensor recognition code and described some of the challenges he's having with reading the camera data. He currently has the code reading other sensors, such as accelerometer data and his tasks for this week are to continue working on capturing the video data.
Over the past week we did some research on the 0mq socket library and in this meeting we discussed the pros and cons of using a 3rd party library. 0mq provides a simple, concurrent, scalable, event-driven library in which full messages are queued for consumption in the application. While this could have a large benefit in a full-scale distribution, we've decide the learning time and setup would be overkill for our application at this stage. Hopefully we’ll have time to come back to it at the end of the semester.
We discussed different server architectures for the application. Part of our objective with this project is to keep the components as decoupled as possible, at both the system-level and application level. Hopefully by taking this approach our work can be expanded upon and we can swap out different components as needed. Under this model, our application server (communication with nodes) will be completely separate from any presentation (web) aside from simple screen output.
Our first two milestones (simple communication/simple motion detection) can be worked on simultaneously as they are independent. Our current tasks are to work towards these first two milestones:
- Matt: Start working on motion detection by experimenting with the camera and sensors on Android
- Scott: Create a basic client- server model between a java server and Android client that can be expanded on.
- Rob: Start working on the communications protocol and message formats
We are currently researching the best way to implement a simple network protocol than can be expanded in the future. We are starting out with a simple threaded server written by Scott. However, the short-comings of this approach are evident already -- we lack basic logging, debugging, and configuration options that will be useful in scaling the server up to production. We are stuck with having to implement all these features on our own and given the complications of threading in java, we will probably end up with alot of debug time.
So, we've spent some time researching industry best practices. There is no shortage of possible solutions, however we needed something that would have:
- Simple Setup
- Logging
- Easy to extend protocol definitions
- Works on Android
- Could be used in high-latency environments
Identification Server:
- Node identifies itself to server
- Can contain authentication later?
- Uses SSL - in tunnel, can provide unique shared-key per node for encryption purposes
- Will assign node to event/management server (load balancing)
- Handles node identification and assignment only, then hands off event/management server
- Node contacts on startup/shutdown only
- TCP/IP
Identification Protocol:
- SSL handshake
- TCP Packet (contains length/checksum)
- TCP Data section contains TID message: ** Message Type ** Message Data
- Node streams events to this server
- Handles events and secondary event processing (AMT handoff, additional CV processing)
- Designed to receive and route events - single function
- Has authenticated node list with shared-key for decryption of messages
- UDP/IP ?
- Handles passing commands to nodes
- Has authenticated node list with shared-key for decryption of messages
- TCP/IP
TID Message Types: 1 AUTH 2 KEY 3 ACK 4 NAK 5 DEAUTH
EXAMPLE FLOW:
Node connects to server via TCP SSL Handshake Inside SSL Tunnel Node - AUTH, options { id: 1829281020192, // node id (serial num?) authkey: l21j21hasjhasdjhasd, // hex auth key }
Server - if successful auth: Server - KEY, options { id: 1829281020192, // repeat node id event-server: xxx.xxx.xxx.xxx, // ip of event server shared-key: 81782878219218332323, // shared key for encryption } Node - ACK, options { id: 1829281020192, // repeat node id }
if unsuccessful auth or not primary auth server:
Server- NAK, options {
id: 1829281020192, // repeat node id
id-server: xxx.xxx.xxx.xxx // ip of id server
}
Node - ACK, options {
id: 1829281020192, // repeat node id
id-server: xxx.xxx.xxx.xxx // ip of id server
}
Eventually, node decides to leave...
Node - DEAUTH, options { id: 1829281020192, // node id authkey: l21j21hasjhasdjhasd, // hex auth key }
Server - ACK, options { id: 1829281020192, // node id }
0mq provides a simple, concurrent, scalable, event-driven library in which full messages are queued for consumption in the application. While this could have a large benefit in a full-scale distribution, we've decide the learning time and setup would be overkill for our application at this stage. It is also designed for low-latency environments, rather than high-latency environments.
Netty is a NIO framework created for the development of network applications such as protocol servers and clients. The primary benefits to this framework is that it is non-blocking and abstracts the lower-level threading. The library is sufficiently flexible that you can write protocols in a byte format or you can use POJO (Plain Old Java Objects) as message envelopes. Some of the cons of this library are...
Met with Murat and spoke about what assumptions we can make and how to structure our spec document.
First Team Meeting -- discussed ideas for the project, mostly about how we wanted to address the problem and what we thought we could accomplish in one semester.