fix deployment of dsvm

README.md

@@ -4,7 +4,7 @@ Good news that you do not have to label all images  (draw bounding boxes) from s
 Please refer to this blog post that describes Active Learning and semi-automated flow: 
   [Active Learning for Object Detection in Partnership with Conservation Metrics](https://www.microsoft.com/developerblog/2018/11/06/active-learning-for-object-detection/)
 We will use Transfer Learning and Active Learning as core Machine Learning  components of the pipeline.
- -- Transfer Learning: use powerful pre-trained on big dataset (COCO) model as a startining point for fine-tuning foe needed classes.
+ -- Transfer Learning: use powerful pre-trained on big dataset (COCO) model as a starting point for fine-tuning foe needed classes.
  -- Active Learning: human annotator labels small set of images (set1), trains Object Detection Model  (model1) on this set1 and then uses model1 to predict bounding boxes on images (thus pre-labeling those). Human annotator reviews mode1's predictions where the model was less confident -- and thus comes up with new set of images -- set2. Next phase will be to train more powerful model2 on bigger train set that includes set1 and set2 and use model2 prediction results as draft of labeled set3…
 The plan is to have 2 versions of pipeline set-up.
 
@@ -17,7 +17,7 @@ This one (ideally) includes minimum setup. The core components here are:
 It will also be used to save "progress" logs of labeling activities
 2) "Tagger" machine(s) 
 This is computer(s) that human annotator(s) is using as environment for labeling portion of images -- for example [VOTT](https://github.com/Microsoft/VoTT).  
-Here example of labeling flow in VOTT: I've labled wood "knots" (round shapes) and "defect" (pretty much  non-round shaped type of defect):
+Here example of labeling flow in VOTT: I've labeled wood "knots" (round shapes) and "defect" (pretty much  non-round shaped type of defect):
 
 ![Labeling](images/VOTT_knot_defect.PNG)
 
@@ -40,12 +40,12 @@ The flow below assumes the following:
 1) We use Tensorflow Object Detection API (Faster RCNN with Resnet 50 as default option)  to fine tune object detection. 
 2) Tensorflow Object Detection API is setup on Linux box (Azure DSVM is an option) that you can ssh to. See docs for Tensorflow Object Detection API regarding its general config.
 3) Data(images) is in Azure blob storage
-4) Human annotators use [VOTT](https://github.com/Microsoft/VoTT)  to label\revise images.  To support another tagging tool it's output (boudin boxes) need to be converted to csv form -- pull requests are welcomed!
+4) Human annotators use [VOTT](https://github.com/Microsoft/VoTT)  to label\revise images.  To support another tagging tool it's output (bounding boxes) need to be converted to csv form -- pull requests are welcomed!
 
 Here is general flow has 2 steps:
 1) Environments setup
-2) Active Learnining cycle: labeling data and running scipts to update model and feed back results for human annotator to review.  
-The whole flow is currenly automated with **4 scrips** user needs to run.
+2) Active Learning cycle: labeling data and running scripts to update model and feed back results for human annotator to review.  
+The whole flow is currently automated with **4 scrips** user needs to run.
 
 
 ### General  prep
@@ -54,30 +54,21 @@ The whole flow is currenly automated with **4 scrips** user needs to run.
 
 
 ### On Linux box aka Model (re)training env
-1) Setup Tensorflow Object Detection API if you have not already.
-This will include cloning of https://github.com/tensorflow/models. (On my machine I have it cloned to `/home/olgali/repos/models`).
- Run `research/object_detection/object_detection_tutorial.ipynb` to make sure Tensorflow Object Detection API is functioning.
-2) Clone this repo to the machine (for example: `/home/olgali/repos/models/research/active-learning-detect/`)
-3)  Update _config.ini_: 
-  - set values for _AZURE_STORAGE_ACCOUNT_ and _AZURE_STORAGE_KEY_
-  - set (update if needed) values for _# Data Information_ section
- -  set values for _# Training Machine_ and _# Tensorflow_ sections of the config.ini  
- _"python_file_directory"_ config value should point to the _"train"_ scripts from this project.
-Example:  
-`python_file_directory=/home/olgali/repos/models/research/active-learning-detect/train`
-3) pip install azure-blob packages: azure.storage.blob
+Run the devops/dsvm/deploy_dsvm.sh script to create a VM for this process. follow the instructions [here]()
 
 ### Tagger machine(s) (could be same as Linux box or separate boxes\vms)
-1) Have Python 3.6 up and running.
-2) Pip install azure-blob packages: azure.storage.blob
+1) Have Python 3.6+ up and running 
+TODO: add section for installing python 
+If you do not have python 3.6+ download [anaconda python](https://www.anaconda.com/distribution/#download-section) 
+```
+python --version
+
+python -m pip install azure.storage.blob
+```
 3) Clone this repo, copy  updated config.ini from Model re-training box (as it has Azure Blob Storage and other generic info already).
-4) Update  _config.ini_ values for _# Tagger Machine_ section:    
+4) Update  _config.ini_ values for _# Tagger Machine_ section:    This is a temporary directory, and the process will delete all the 
+   files every time you label so do not use an existing dir that you care about
         `tagging_location=D:\temp\NewTag`
-
-# Label data, run the scripts!
-Overview: you will run **4 scripts* in total: 
-- two scipts on the machine where model (re)training happens and 
-- two scripts where human annotators label images (or review images pre-labeled by the model).
 
 ### On Linux box aka Model (re)training env
 Run bash script to Init pipeline  
@@ -87,28 +78,14 @@ This step will:
 - Create totag_xyz.csv on the blob storage ( "activelearninglabels" container by default).  
 This is the snapshot of images file names that need tagging (labeling).  As human annotators make progress on labeling data the list will get smaller and smaller.
 
-### Tagger machine(s)
-1) Make sure that the tagging_location is empty.
-2) Start each "phase" with downloading images to label (or to review pre-labeled images).  
-Sample cmd below requests 40 images for tagging:  
-`D:\repo\active-learning-detect\tag>python download_vott_json.py 40 ..\config.ini`  
-This step will create new version of totag_xyz.csv on blob storage that will have 40 images excluded from the list.  
-File tagging_abc.csv will hold list of 40 images being tagged.
-3) Start [VOTT](https://github.com/Microsoft/VoTT) , load the folder for labeling\review (in my case it will be `D:\temp\NewTag\images`)
-4) Once done with labeling push results back to central storage:  
- `D:\repo\active-learning-detect\tag>python upload_vott_json.py ..\config.ini`
-This step will push tagged_123.csv to blob storage: this file contains actual bounding boxes coordinates for every image.  
-Tagging_abc.csv will contain list of files that are "work in progress" -- the ones to be tagged soon.
-
-
-Now model can be trained.
+### Label Now model can be trained.
 
 ### Model(re)training on Linux box
 Before your first time running the model, and at any later time if you would like to repartition the test set, run:
 
 `~/repos/models/research/active-learning-detect/train$ . ./repartition_test_set_script.sh  ../config.ini`
 
-This script will take all the tagged data and split some of it into a test set, which will not be trained/validated on and will then be use by evalution code to return mAP values.
+This script will take all the tagged data and split some of it into a test set, which will not be trained/validated on and will then be use by evaluation code to return mAP values.
 
 Run bash script:  
 `~/repos/models/research/active-learning-detect/train$ . ./active_learning_train.sh  ../config.ini`
@@ -127,36 +104,35 @@ Human annotator(s) deletes any leftovers from previous predictions (csv files in
 
 Training cycle can now be repeated on bigger training set and dataset with higher quality of pre-labeled bounding boxes could be obtained. 
 
+# Running prediciton on a new batch of data (existing model)
 
-# Using Custom Vision service for training
+Send the config to the remote machine (make sure it is updated with the model that you want to use).
+```
+scp "config_usgs19_pred20191021.ini" [email protected]:/home/cmi/active-learning-detect
+```
 
-The Custom Vision service can be used instead of Tensorflow in case you do not have access to an Azure Data Science VM or other GPU-enabled machine. The steps for Custom Vision are pretty similar to those for Tensorflow, although the training step is slightly different:
+SSH into the machine
 
-### Model (re)training on Custom Vision
-If you would like to repartition the test set, run:
+```
+[email protected]
+```
 
-`~/repos/models/research/active-learning-detect/train$ . ./repartition_test_set_script.sh  ../config.ini`
-
-This script will take all the tagged data and split some of it into a test set, which will not be trained/validated on and will then be use by evalution code to return mAP values.
+Remove the old training directory to avoid running prediction on the last batch of images you uploaded.
 
-To train the model:  
-python cv_train.py ../config.ini
+```
+rm -rd data
+```
 
-This python script will train a custom vision model based on available labeled data.  
+change directory to "train" 
 
-Model will evaluated on test set and perf numbers will be saved in blob storage (performance.csv).
-
-Latest totag.csv will have predictions for all available images made of the newly trained model -- bounding box locations that could be used by human annotator as a starter.
+```
+cd active-learning-detect/train
+```
 
+Run the prediction script with the config that you just uploaded
 
-# Sample dataset
-I'm using wood knots dataset mentioned in this [blog](http://blog.revolutionanalytics.com/2017/09/wood-knots.html) 
-Here is [link](https://olgaliakrepo.blob.core.windows.net/woodknots/board_images_png.zip) to the dataset: zip file with 800+ board png images.
-
-# Custom Vision HttpOperationError 'Bad Request'
+active_learning_predict_no_train.sh uses the info in the config file to download the images from blob storage, download the model file from blob storage, and then run prediction on these images.  
 
-The current custom vision SDK is in preview mode, and one of the limitations is that an error while training does not return an error message, just a generic 'Bad Request' response. Common reasons for this error include:
-1) Having a tag with less than 15 images. Custom Vision requires a minimum of 15 images per tag and will throw an error if it finds any tag with less than that many.
-2) Having a tag out of bounds. If for some reason you attempt to add a tag through the API which is out of bounds, it will accept the request but will throw an error while training.
-3) No new images since last training session. If you try to train without adding additional images Custom Vision will return a bad request exception.
-The best way to debug these is to go into the Custom Vision website (customvision.ai) and click the train button, which should then tell you what the error was.
+```
+sh active_learning_predict_no_train.sh ../config_usgs19_pred20191021.ini
+```

config.ini

@@ -3,6 +3,10 @@ AZURE_STORAGE_ACCOUNT=<STORAGE ACCOUNT NAME>
 AZURE_STORAGE_KEY=<STORAGE ACCOUNT KEY>
 image_container_name=activelearningimages
 label_container_name=activelearninglabels
+pred_model_name=None
+pred_dir=None
+min_tile_size=250
+max_tile_size=500
 # IMAGE INFORMATION
 user_folders=True
 classes=knots,defect

config_devtest.ini

@@ -0,0 +1,58 @@
+# AZURE STORAGE ACCOUNT INFORMATION
+AZURE_STORAGE_ACCOUNT=usgsaerialimages
+AZURE_STORAGE_KEY=NEED-KEY-HERE
+image_container_name=devtest-images
+label_container_name=devtest-labels
+pred_model_name=model_1593140460.pb
+pred_dir=None 
+min_tile_size=600
+max_tile_size=1024
+# IMAGE INFORMATION
+user_folders=True
+classes=dark_bird,marine_mammal,dark_bird_f,light_bird_f,light_bird,trash
+filetype=*.JPG
+# TRAINING MACHINE
+# Locationss
+python_file_directory=/home/cmi/active-learning-detect/train
+data_dir=~/data/usgs_2019pred
+train_dir=~/data/usgs_2019pred/training
+inference_output_dir=~/data/usgs_2019pred/training/usgs_2019_inference_graphs
+tf_models_location=/home/cmi/repos/models/research
+download_location=/home/cmi/downloads
+# Training
+train_iterations=20000
+eval_iterations=100
+min_confidence=.01
+test_percentage=.15
+model_name=faster_rcnn_inception_v2_coco_2018_01_28
+optional_pipeline_url=https://raw.githubusercontent.com/tensorflow/models/master/research/object_detection/samples/configs/faster_rcnn_inception_v2_coco.config
+#Init Predictions
+init_model_name=faster_rcnn_resnet101_coco_2018_01_28
+# Config File Details
+old_label_path=PATH_TO_BE_CONFIGURED/mscoco_label_map.pbtxt
+old_train_path=PATH_TO_BE_CONFIGURED/mscoco_train.record-?????-of-00100
+old_val_path=PATH_TO_BE_CONFIGURED/mscoco_val.record-?????-of-00010
+old_checkpoint_path=PATH_TO_BE_CONFIGURED/model.ckpt
+num_examples_marker=num_examples:
+num_steps_marker=num_steps:
+num_classes_marker=num_classes:
+# Calculated
+num_classes="$(awk -F ',' '{print NF}' <<< ${classes})"
+image_dir=${data_dir}/AllImages
+untagged_output=${data_dir}/untagged.csv
+tagged_output=${data_dir}/tagged.csv
+tagged_predictions=${data_dir}/tagged_preds.csv
+test_output=${data_dir}/test.csv
+validation_output=${data_dir}/val.csv
+tf_location=${tf_models_location}/object_detection
+tf_location_legacy=${tf_models_location}/object_detection/legacy
+PYTHONPATH=$PYTHONPATH:${tf_models_location}:${tf_models_location}/slim/
+label_map_path=${data_dir}/pascal_label_map.pbtxt
+tf_record_location=${data_dir}/stamps.record
+tf_train_record=${tf_record_location%.*}_train.${tf_record_location##*.}
+tf_val_record=${tf_record_location%.*}_val.${tf_record_location##*.}
+tf_url=http://download.tensorflow.org/models/object_detection/${model_name}.tar.gz
+pipeline_file=${download_location}/${model_name}/pipeline.config
+fine_tune_checkpoint=${download_location}/${model_name}/model.ckpt
+init_pred_tf_url=http://download.tensorflow.org/models/object_detection/${init_model_name}.tar.gz
+init_model_graph=${download_location}/${init_model_name}/frozen_inference_graph.pb

config_oikonos_mega.ini

@@ -0,0 +1,70 @@
+# AZURE STORAGE ACCOUNT INFORMATION
+AZURE_STORAGE_ACCOUNT=oikonos
+AZURE_STORAGE_KEY="put key here"
+image_container_name=oikonos-mega-images
+label_container_name=oikonos-mega-labels
+# IMAGE INFORMATION
+user_folders=True
+classes=cat,dog,rabbit,rat,shearwater,Unknown,zorzal,petrel,kestrel,pigeon,human,goat,mouse,coati,cow,horse,pig,owl
+# Provide preferred distribution of images-review ratio.
+# Last value corresponds to images were no object were detected.
+# In the example below: 60% of images that use will be reviewing have at least one  bbox with objct class1 (knot),
+#   30%  images that have  bboxes for class  (defects)
+#   and 10% of images get class "NULL" -- were neither knots nor defects were detected by the model
+ideal_class_balance=0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
+filetype=*.JPG
+# TAGGING MACHINE
+tagging_location=C:\Users\ConservationMetrics\Desktop\oikonos_tagging
+pick_max=False
+max_tags_per_pixel=1
+#
+# CUSTOM VISION
+# Uncomment lines below if using Azure Custom Vision Service training_key=<CUSTOM VISION TRAINING KEY>
+# prediction_key=<CUSTOM VISION PREDICTION KEY>
+# project_id=<CUSTOM VISION PROJECT ID>
+#
+# TRAINING MACHINE
+# Locations
+python_file_directory=/home/cmi/repos/models/research/active-learning-detect/train
+data_dir=/home/cmi/data/oikonosmega
+train_dir=/home/cmi/data/oikonosmega/training
+inference_output_dir=/home/cmi/data/oikonosmega/training/oikonosmega_inference_graphs
+tf_models_location=/home/cmi/repos/models/research
+download_location=/home/cmi/downloads
+# Training
+train_iterations=100
+eval_iterations=100
+min_confidence=.1
+test_percentage=.15
+model_name=megadetector_v3
+optional_pipeline_url=https://lilablobssc.blob.core.windows.net/models/camera_traps/megadetector/megadetector_v3.config
+#Init Predictions
+init_model_name=faster_rcnn_resnet101_coco_2018_01_28
+# Config File Details
+old_label_path=/megadetectorv3/PyCharm/CameraTraps/detection/experiments/megadetector_v3/label_map.pbtxt
+old_train_path=/disk/megadetectorv3_tfrecords/???????~train-?????-of-?????
+old_val_path=/disk/megadetectorv3_tfrecords/???????~val__-?????-of-?????
+old_checkpoint_path=PATH_TO_BE_CONFIGURED/model.ckpt
+num_examples_marker=num_examples:
+num_steps_marker=num_steps:
+num_classes_marker=num_classes:
+# Calculated
+num_classes="$(awk -F ',' '{print NF}' <<< ${classes})"
+image_dir=${data_dir}/AllImages
+untagged_output=${data_dir}/untagged.csv
+tagged_output=${data_dir}/tagged.csv
+tagged_predictions=${data_dir}/tagged_preds.csv
+test_output=${data_dir}/test.csv
+validation_output=${data_dir}/val.csv
+tf_location=${tf_models_location}/object_detection
+tf_location_legacy=${tf_models_location}/object_detection/legacy
+PYTHONPATH=$PYTHONPATH:${tf_models_location}:${tf_models_location}/slim/
+label_map_path=${data_dir}/pascal_label_map.pbtxt
+tf_record_location=${data_dir}/stamps.record
+tf_train_record=${tf_record_location%.*}_train.${tf_record_location##*.}
+tf_val_record=${tf_record_location%.*}_val.${tf_record_location##*.}
+tf_url=https://lilablobssc.blob.core.windows.net/models/camera_traps/megadetector/${model_name}_checkpoint.zip
+pipeline_file=${download_location}/${model_name}/pipeline.config
+fine_tune_checkpoint=${download_location}/${model_name}/${model_name}_checkpoint/model.ckpt
+init_pred_tf_url=http://download.tensorflow.org/models/object_detection/${init_model_name}.tar.gz
+init_model_graph=${download_location}/${init_model_name}/frozen_inference_graph.pb