diff --git a/docs/data_processing.md b/docs/data_processing.md index e51a9ba..fe5f0aa 100644 --- a/docs/data_processing.md +++ b/docs/data_processing.md @@ -2,15 +2,15 @@ The files documents the steps for creating a dataset to use with Fast Change Detection. -First, install the [Astrobee flight software](https://github.com/nasa/astrobee). Additional instructions for using the Astrobee flight software with Docker are [here](https://docs.google.com/document/d/1Wx54si5_24rz0kJie31X54PIk_k_owT6qzlziGnAWYc/edit?usp=sharing). Clone this repository into the astrobee flight software repository: +First, install the [Astrobee flight software](https://github.com/nasa/astrobee) in the '$HOME' directory. Additional instructions for using the Astrobee flight software with Docker are [here](https://docs.google.com/document/d/1Wx54si5_24rz0kJie31X54PIk_k_owT6qzlziGnAWYc/edit?usp=sharing). Next, clone this repository: ```bash cd $HOME && git clone https://github.com/hollydinkel/astrobee_change_detection --recurse-submodules ``` -Download a dated raw dataset of `.bag` files from [here](https://docs.google.com/document/d/1Wx54si5_24rz0kJie31X54PIk_k_owT6qzlziGnAWYc/edit?usp=sharing). Unzip the dataset into a `astrobee_change_detection/data` directory (**HD 20231113: This is not the correct link, and the data on GDrive is no longer in the format to support this step anyway. Needs re-testing**). Note that the survey number (e.g., 1, 2, 3), the date (e.g., 20230419), and the robot name (e.g., bsharp) must be specified in each step. The first four steps can be performed in a docker container where the running container is mounted to a local directory. The provided `run.sh` script runs these four steps. The last step should be performed locally if the FastCD workspace is built outside of the container. +Follow the steps in the provided `process_data.sh` file to download raw `.bag` data and process it into a change detection-compatible dataset. Note that the home directory, the change detection catkin workspac, the survey number (e.g., 1, 2, 3), the date (e.g., 20230419), and the robot name (e.g., bsharp) must be specified in the `process_data.sh` script. If the `astrobee` catkin workspace does not build locally, it is possible to do these steps in a docker container where the running container is mounted to a local directory. The provided `process_data.sh` script runs these four steps. -1. The first step of data processing is extraction of images and poses from bag data to folder, processing of sequential images to remove images where frames did not move much between images, and processing of sequential poses so that final poses are close in time (timestamps are close) to the image timestamps. First, start a running Astrobee docker container with +1. The first step of data processing is extraction of images and poses from bag data to folder, processing of sequential images to remove images where frames did not move much between images, and processing of sequential poses so that pose timestamps are close in time to image timestamps. First, start a running Astrobee docker container with ```bash export ASTROBEE_WS=$HOME/astrobee @@ -20,8 +20,7 @@ cd $ASTROBEE_WS && sudo ./scripts/docker/run.sh -m bash Inside the docker container, run ```bash -export DATA=/src/astrobee/src/astrobee_change_detection -cd $DATA && ./run.sh +cd $DATA && ./process_data.sh ``` After processing the data in the the docker container, return to a local terminal to create a `cameras.xml` file for the dataset to use in FastCD. Make sure to change the `fastcd_data_directory` variable in line 21 of `scripts/create_cameras_xml.py` before running this. Also make sure the `DATE` and `ROBOT` arguments match those used in `run.sh`. diff --git a/run.sh b/process_data.sh old mode 100644 new mode 100755 similarity index 60% rename from run.sh rename to process_data.sh index bb5dd8e..02e9428 --- a/run.sh +++ b/process_data.sh @@ -1,12 +1,14 @@ #!/bin/bash export HOME=/home/hdinkel -export ASTROBEE_WS=$HOME/astrobee/src/ -export ASTROBEE_DEVEL=$HOME/astrobee/devel -export ASTROBEE_CHANGE_DETECTION=$HOME/astrobee_change_detection +export CATKIN_WS=$HOME/test_ws/src +export ASTROBEE=$HOME/astrobee +export ASTROBEE_CHANGE_DETECTION=$CATKIN_WS/astrobee_change_detection export DATA=$ASTROBEE_CHANGE_DETECTION/data export DATE=20230419 export ROBOT=bsharp -source $ASTROBEE_DEVEL/setup.bash +source $ASTROBEE/devel/setup.bash + +mkdir $DATA # Install dependencies apt-get update @@ -17,14 +19,14 @@ pip install gdown pyquaternion pandas # Download data cd $DATA && gdown https://drive.google.com/uc?id=1qNCRl9XceINrHLFUp4pBti9I4oJO2zf1 unzip data.zip -rm data.zip +rm $DATA/data.zip for SURVEY_NUMBER in 1 2 3 4 do export SURVEY=$SURVEY_NUMBER cd $DATA && python3 $ASTROBEE_CHANGE_DETECTION/astrobee_data_processing_scripts/poses_to_file.py $SURVEY $DATE $ROBOT - rosrun sparse_mapping process_sequential_images.py $DATA/data/$DATE/$ROBOT/bayer/survey$SURVEY $ASTROBEE_WS/src/astrobee/config + rosrun sparse_mapping process_sequential_images.py $DATA/$DATE/$ROBOT/bayer/survey$SURVEY $ASTROBEE/src/astrobee/config cd $DATA && python3 $ASTROBEE_CHANGE_DETECTION/astrobee_data_processing_scripts/process_sequential_poses.py $SURVEY $DATE $ROBOT - cd $DATA/data/$DATE/$ROBOT/bayer/survey$SURVEY - # ls -v | nl -v 0 | while read n f; do mv -n "$f" "Image$n.JPG"; done + cd $DATA/$DATE/$ROBOT/bayer/survey$SURVEY + ls -v | nl -v 0 | while read n f; do mv -n "$f" "Image$n.JPG"; done done \ No newline at end of file