This project was started to compete in the Tech To Protect challenge, specificially for contest #5. We created the idea in order to address what we saw as a large gap in the use of technology by firefighters and first responders. While pursuing our idea, we realized that there is also a viable business idea in using 3D modeling and machine learning in order to play the middle man between home insurance companies, city and county fire departments, and homeowners. Using the data generated by the user and the automated generation of the machine learning models, we will be able to increase the efficiency of the emergency responses to houses within our network, while also providing savings to the homeowners by ensuring that their homeowners insurance company is continually updated with their pre-incident plan and fire safety checklist.
We interviewed multiple firefighters in the Denver area, and were able to compile a large amount of information concerning the current gaps in emergency responses. These included not having any idea of home floor plans, not knowing if residents of the home had medical conditions that could interfere with rescues, and not having any idea of the clutter level or hazards that could be in the home. These problems could be solved using machine learning on point clouds in order to do item detection and floor plan generation.
This will be accomplished by incentivising the homeowners to take multiple pictures and/or videos of each room and using those photos or videos to create a point cloud rendering of the room. This will allow the creation of floor plans using this data and also of image recognition using the images/videos and the point cloud. This would allow us to detect various things about the evironment, included the presence of safety equipment and verification that the homeowner is following the commonly accepted safety regulations for fire protection.
This data can then be sent to the homeowners insurance company along with the local fire department. The insurance company could give the homeowner a premium break for being in compliance with security practices. The local fire department could access the information about the homeowners house in order to make any emergency response much more efficient. The most requested feature from the firefighters we interviewed was information about the residents of the house. We would prompt the homeowner to upload that information, including non-specific health conditions that would allow the firefighters to plan for residents in a wheelchar, for example. Using the 3D models generated from the homeowners pictures and videos, we would also generate a 3D model of the house for the firefighter to explore as they respond to the emergency.
Currently we are iterating through the different aspects of this application. We envision a React Native mobile application, utilizing a Flask back-end with a Postgres database, running on Amazon Web Services (AWS) or Microsoft Azure. This would allow the secure storage of homeowner information, and allow the mobile application to communicate securely with the back-end. The Flask back-end will encompass a couple different machine learning models. One of those models will process the stitched together point cloud images in order to recognize items inside the rooms. Another model will take the pictures and video and provide item detection using computer vision. The third model will perform analysis of the image outputs of the cloud slicing script, which will slice the point cloud file into slices that are then turned into JPEGs. This analysis will determine the floor plan of the house.
The image analysis will also be used to help pre-populate a fire safety checklist. Utilizing a final machine learning model, we will use Q learning to attempt to compute the most efficient escape routes from each room, utilizing the information we have about the floor plan and rest of the house in order to navigate to floor level windows and doors. All of this information will be verified by the homeowner before being finalized.
You can find a Canva presentation at this link. You can find the three different prototypes at these links - the sign up process for new users, the homeowner experience, and the firefighter experience.
In order to explore the React Native prototype we have built so far, you can navigate to here and follow the instructions on the ReadMe. If you would like to use the docker container in order to process a point cloud image and check out the code we are using to build the different models, you can go here and follow the ReadMe.