You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
[](https://colab.research.google.com/github/ericsujw/InstColorization/blob/master/InstColorization.ipynb)
Image colorization is inherently an ill-posed problem with multi-modal uncertainty. Previous methods leverage the deep neural network to map input grayscale images to plausible color outputs directly. Although these learning-based methods have shown impressive performance, they usually fail on the input images that contain multiple objects. The leading cause is that existing models perform learning and colorization on the entire image. In the absence of a clear figure-ground separation, these models cannot effectively locate and learn meaningful object-level semantics. In this paper, we propose a method for achieving instance-aware colorization. Our network architecture leverages an off-the-shelf object detector to obtain cropped object images and uses an instance colorization network to extract object-level features. We use a similar network to extract the full-image features and apply a fusion module to full object-level and image-level features to predict the final colors. Both colorization networks and fusion modules are learned from a large-scale dataset. Experimental results show that our work outperforms existing methods on different quality metrics and achieves state-of-the-art performance on image colorization.
11
+
12
+
13
+
**Instance-aware Image Colorization**
14
+
<br/>
15
+
[Jheng-Wei Su](https://github.com/ericsujw),
16
+
[Hung-Kuo Chu](https://cgv.cs.nthu.edu.tw/hkchu/), and
2. Now the COCOStuff train set would place in [train_data](train_data).
57
+
58
+
### Your own Dataset
59
+
1. If you want to train on your dataset, you should change the dataset path in [scripts/prepare_train_box.sh's L1](scripts/prepare_train_box.sh#L1) and in [scripts/train.sh's L1](scripts/train.sh#L1).
60
+
61
+
## Pretrained Model
62
+
1. Download it from [google drive](https://drive.google.com/open?id=1Xb-DKAA9ibCVLqm8teKd1MWk6imjwTBh).
63
+
```sh
64
+
sh scripts/download_model.sh
65
+
```
66
+
2. Now the pretrained models would place in [checkpoints](checkpoints).
67
+
68
+
## Instance Prediction
69
+
Please follow the command below to predict all the bounding boxes fo the images in `${DATASET_DIR}` folder.
70
+
```sh
71
+
sh scripts/prepare_train_box.sh
72
+
```
73
+
All the prediction results would save in `${DATASET_DIR}_bbox` folder.
74
+
75
+
## Training the Instance-aware Image Colorization model
76
+
Simply run the following command, then the training pipeline would get start.
77
+
```sh
78
+
sh scripts/train.sh
79
+
```
80
+
To view training results and loss plots, run `visdom -port 8098` and click the URL http://localhost:8098.
81
+
82
+
This is a 3 stage training process.
83
+
1. We would start to train our full image colorization branch based on the [siggraph_retrained's pretrained weight](https://github.com/richzhang/colorization-pytorch).
84
+
2. We would use the full image colorization branch's weight as our instance colorization branch's pretrained weight.
85
+
3. Finally, we would train the fusion module.
86
+
87
+
## Testing the Instance-aware Image Colorization model
88
+
1. Our model's weight would place in [checkpoints/coco_mask](checkpoints/coco_mask).
89
+
2. Change the checkpoint's path in [test_fusion.py's L38](test_fusion.py#L38) from `coco_finetuned_mask_256_ffs` to `coco_mask`
90
+
3. Please follow the command below to colorize all the images in `example` foler based on the weight placed in `coco_mask`.
parser.add_argument('--model', type=str, default='train', help='only train_model need to be used')
10
+
parser.add_argument('--name', type=str, default='coco_mask', help='name of the experiment. It decides where to store samples and models')
7
11
parser.add_argument('--display_freq', type=int, default=2000, help='frequency of showing training results on screen')
8
12
parser.add_argument('--display_ncols', type=int, default=5, help='if positive, display all images in a single visdom web panel with certain number of images per row.')
9
13
parser.add_argument('--update_html_freq', type=int, default=10000, help='frequency of saving training results to html')
0 commit comments