Skip to content

Commit 45c8fa4

Browse files
tomas-gajarskytomas-gajarsky
authored
Merge pull request #37 from tomas-gajarsky/feature/action_unit
Feature/action unit
2 parents fe3587d + af6640c commit 45c8fa4

34 files changed

+2415
-1347
lines changed

‎CHANGELOG.md

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,17 @@
11
# Change Log
22

3+
4+
## 0.3.0
5+
6+
Released on February 9, 2023.
7+
8+
### Added
9+
* predictor for facial action unit detection - OpenGraphAU from Luo et al.
10+
11+
### Changed
12+
* facetorch version retrieval error handling to not fail the run
13+
14+
315
## 0.2.4
416

517
Released on February 4, 2023.

‎README.md

Lines changed: 14 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -78,6 +78,7 @@ analyzer
7878
├── embed
7979
├── verify
8080
├── fer
81+
├── au
8182
├── deepfake
8283
└── align
8384
└── utilizer
@@ -150,6 +151,16 @@ analyzer
150151
* B0 [![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/facial-expression-and-attributes-recognition/facial-expression-recognition-on-affectnet)](https://paperswithcode.com/sota/facial-expression-recognition-on-affectnet?p=facial-expression-and-attributes-recognition)
151152
* B0 [![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/facial-expression-and-attributes-recognition/facial-expression-recognition-on-acted-facial)](https://paperswithcode.com/sota/facial-expression-recognition-on-acted-facial?p=facial-expression-and-attributes-recognition)
152153

154+
#### Facial Action Unit Detection (au)
155+
156+
| model | source | params | license | version |
157+
| ------------------- | --------- | ------- | ------------------ | ------- |
158+
| OpenGraph Swin Base | CVI-SZU | 94M | MIT License | 1 |
159+
160+
1. CVI-SZU
161+
* code: [ME-GraphAU](https://github.com/CVI-SZU/ME-GraphAU)
162+
* paper: [Luo et al. - Learning Multi-dimensional Edge Feature-based AU Relation Graph for Facial Action Unit Recognition](https://arxiv.org/abs/2205.01782)
163+
* [![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/learning-multi-dimensional-edge-feature-based/facial-action-unit-detection-on-bp4d)](https://paperswithcode.com/sota/facial-action-unit-detection-on-bp4d?p=learning-multi-dimensional-edge-feature-based)
153164

154165
#### Deepfake Detection (deepfake)
155166

@@ -185,18 +196,19 @@ You can also download the models manually from a [public Google Drive folder](ht
185196

186197
### Execution time
187198

188-
Image test.jpg (4 faces) is analyzed (including drawing boxes and landmarks, but not saving) in about 465ms and test3.jpg (25 faces) in about 1480ms (batch_size=8) on NVIDIA Tesla T4 GPU once the default configuration (*conf/config.yaml*) of models is initialized and pre heated to the initial image size 1080x1080 by the first run. One can monitor the execution times in logs using the DEBUG level.
199+
Image test.jpg (4 faces) is analyzed (including drawing boxes and landmarks, but not saving) in about 482ms and test3.jpg (25 faces) in about 1844ms (batch_size=8) on NVIDIA Tesla T4 GPU once the default configuration (*conf/config.yaml*) of models is initialized and pre heated to the initial image size 1080x1080 by the first run. One can monitor the execution times in logs using the DEBUG level.
189200

190201
Detailed test.jpg execution times:
191202
```
192203
analyzer
193204
├── reader: 27 ms
194-
├── detector: 230 ms
205+
├── detector: 193 ms
195206
├── unifier: 1 ms
196207
└── predictor
197208
├── embed: 8 ms
198209
├── verify: 58 ms
199210
├── fer: 28 ms
211+
├── au: 57 ms
200212
├── deepfake: 117 ms
201213
└── align: 5 ms
202214
└── utilizer

‎conf/analyzer/predictor/au/open_graph_swin_base.yaml

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
_target_: facetorch.analyzer.predictor.FacePredictor
2+
3+
downloader:
4+
_target_: facetorch.downloader.DownloaderGDrive
5+
file_id: 1uoVX9suSA5JVWTms3hEtJKzwO-CUR_jV
6+
path_local: /opt/facetorch/models/torchscript/predictor/au/1/model.pt # str
7+
8+
device:
9+
_target_: torch.device
10+
type: ${analyzer.device} # str
11+
12+
preprocessor:
13+
_target_: facetorch.analyzer.predictor.pre.PredictorPreProcessor
14+
transform:
15+
_target_: torchvision.transforms.Compose
16+
transforms:
17+
- _target_: torchvision.transforms.Resize
18+
size: [224, 224] # List[int]
19+
- _target_: torchvision.transforms.Normalize
20+
mean: [0.485, 0.456, 0.406] # List[float]
21+
std: [0.229, 0.224, 0.225] # List[float]
22+
device:
23+
_target_: torch.device
24+
type: ${analyzer.predictor.au.device.type}
25+
optimize_transform: ${analyzer.optimize_transforms}
26+
reverse_colors: False # bool
27+
28+
postprocessor:
29+
_target_: facetorch.analyzer.predictor.post.PostMultiLabel
30+
transform: None
31+
device:
32+
_target_: torch.device
33+
type: ${analyzer.predictor.au.device.type}
34+
optimize_transform: ${analyzer.optimize_transforms}
35+
dim: 1
36+
threshold: 0.5
37+
labels: ["inner_brow_raiser", "outer_brow_raiser", "brow_lowerer", "upper_lid_raiser", "cheek_raiser", "lid_tightener", "nose_wrinkler", "upper_lip_raiser", "nasolabial_deepener", "lip_corner_puller", "sharp_lip_puller", "dimpler", "lip_corner_depressor", "lower_lip_depressor", "chin_raiser", "lip_pucker", "tongue_show", "lip_stretcher", "lip_funneler", "lip_tightener", "lip_pressor", "lips_part", "jaw_drop", "mouth_stretch", "lip_bite", "nostril_dilator", "nostril_compressor", "left_inner_brow_raiser", "right_inner_brow_raiser", "left_outer_brow_raiser", "right_outer_brow_raiser", "left_brow_lowerer", "right_brow_lowerer", "left_cheek_raiser", "right_cheek_raiser", "left_upper_lip_raiser", "right_upper_lip_raiser", "left_nasolabial_deepener", "right_nasolabial_deepener", "left_dimpler", "right_dimpler"] # List
38+

‎conf/config.yaml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ defaults:
66
- analyzer/predictor/embed: r50_vggface_1m.yaml
77
- analyzer/predictor/verify: adaface_ir101_webface12m.yaml
88
- analyzer/predictor/fer: efficientnet_b2_8.yaml
9+
- analyzer/predictor/au: open_graph_swin_base.yaml
910
- analyzer/predictor/deepfake: efficientnet_b7.yaml
1011
- analyzer/predictor/align: synergy_mobilenet_v2.yaml
1112
- analyzer/utilizer/align: lmk3d_mesh_pose.yaml

‎conf/merged/gpu.merged.config.yaml

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -228,6 +228,81 @@ analyzer:
228228
- Neutral
229229
- Sadness
230230
- Surprise
231+
au:
232+
_target_: facetorch.analyzer.predictor.FacePredictor
233+
downloader:
234+
_target_: facetorch.downloader.DownloaderGDrive
235+
file_id: 1uoVX9suSA5JVWTms3hEtJKzwO-CUR_jV
236+
path_local: /opt/facetorch/models/torchscript/predictor/au/1/model.pt # str
237+
device:
238+
_target_: torch.device
239+
type: ${analyzer.device} # str
240+
preprocessor:
241+
_target_: facetorch.analyzer.predictor.pre.PredictorPreProcessor
242+
transform:
243+
_target_: torchvision.transforms.Compose
244+
transforms:
245+
- _target_: torchvision.transforms.Resize
246+
size: [224, 224] # List[int]
247+
- _target_: torchvision.transforms.Normalize
248+
mean: [0.485, 0.456, 0.406] # List[float]
249+
std: [0.229, 0.224, 0.225] # List[float]
250+
device:
251+
_target_: torch.device
252+
type: ${analyzer.predictor.au.device.type}
253+
optimize_transform: ${analyzer.optimize_transforms}
254+
reverse_colors: False # bool
255+
postprocessor:
256+
_target_: facetorch.analyzer.predictor.post.PostMultiLabel
257+
transform: None
258+
device:
259+
_target_: torch.device
260+
type: ${analyzer.predictor.au.device.type}
261+
optimize_transform: ${analyzer.optimize_transforms}
262+
dim: 1
263+
threshold: 0.5
264+
labels:
265+
- inner_brow_raiser
266+
- outer_brow_raiser
267+
- brow_lowerer
268+
- upper_lid_raiser
269+
- cheek_raiser
270+
- lid_tightener
271+
- nose_wrinkler
272+
- upper_lip_raiser
273+
- nasolabial_deepener
274+
- lip_corner_puller
275+
- sharp_lip_puller
276+
- dimpler
277+
- lip_corner_depressor
278+
- lower_lip_depressor
279+
- chin_raiser
280+
- lip_pucker
281+
- tongue_show
282+
- lip_stretcher
283+
- lip_funneler
284+
- lip_tightener
285+
- lip_pressor
286+
- lips_part
287+
- jaw_drop
288+
- mouth_stretch
289+
- lip_bite
290+
- nostril_dilator
291+
- nostril_compressor
292+
- left_inner_brow_raiser
293+
- right_inner_brow_raiser
294+
- left_outer_brow_raiser
295+
- right_outer_brow_raiser
296+
- left_brow_lowerer
297+
- right_brow_lowerer
298+
- left_cheek_raiser
299+
- right_cheek_raiser
300+
- left_upper_lip_raiser
301+
- right_upper_lip_raiser
302+
- left_nasolabial_deepener
303+
- right_nasolabial_deepener
304+
- left_dimpler
305+
- right_dimpler
231306
deepfake:
232307
_target_: facetorch.analyzer.predictor.FacePredictor
233308
downloader:

‎conf/merged/merged.config.yaml

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -228,6 +228,81 @@ analyzer:
228228
- Neutral
229229
- Sadness
230230
- Surprise
231+
au:
232+
_target_: facetorch.analyzer.predictor.FacePredictor
233+
downloader:
234+
_target_: facetorch.downloader.DownloaderGDrive
235+
file_id: 1uoVX9suSA5JVWTms3hEtJKzwO-CUR_jV
236+
path_local: /opt/facetorch/models/torchscript/predictor/au/1/model.pt # str
237+
device:
238+
_target_: torch.device
239+
type: ${analyzer.device} # str
240+
preprocessor:
241+
_target_: facetorch.analyzer.predictor.pre.PredictorPreProcessor
242+
transform:
243+
_target_: torchvision.transforms.Compose
244+
transforms:
245+
- _target_: torchvision.transforms.Resize
246+
size: [224, 224] # List[int]
247+
- _target_: torchvision.transforms.Normalize
248+
mean: [0.485, 0.456, 0.406] # List[float]
249+
std: [0.229, 0.224, 0.225] # List[float]
250+
device:
251+
_target_: torch.device
252+
type: ${analyzer.predictor.au.device.type}
253+
optimize_transform: ${analyzer.optimize_transforms}
254+
reverse_colors: False # bool
255+
postprocessor:
256+
_target_: facetorch.analyzer.predictor.post.PostMultiLabel
257+
transform: None
258+
device:
259+
_target_: torch.device
260+
type: ${analyzer.predictor.au.device.type}
261+
optimize_transform: ${analyzer.optimize_transforms}
262+
dim: 1
263+
threshold: 0.5
264+
labels:
265+
- inner_brow_raiser
266+
- outer_brow_raiser
267+
- brow_lowerer
268+
- upper_lid_raiser
269+
- cheek_raiser
270+
- lid_tightener
271+
- nose_wrinkler
272+
- upper_lip_raiser
273+
- nasolabial_deepener
274+
- lip_corner_puller
275+
- sharp_lip_puller
276+
- dimpler
277+
- lip_corner_depressor
278+
- lower_lip_depressor
279+
- chin_raiser
280+
- lip_pucker
281+
- tongue_show
282+
- lip_stretcher
283+
- lip_funneler
284+
- lip_tightener
285+
- lip_pressor
286+
- lips_part
287+
- jaw_drop
288+
- mouth_stretch
289+
- lip_bite
290+
- nostril_dilator
291+
- nostril_compressor
292+
- left_inner_brow_raiser
293+
- right_inner_brow_raiser
294+
- left_outer_brow_raiser
295+
- right_outer_brow_raiser
296+
- left_brow_lowerer
297+
- right_brow_lowerer
298+
- left_cheek_raiser
299+
- right_cheek_raiser
300+
- left_upper_lip_raiser
301+
- right_upper_lip_raiser
302+
- left_nasolabial_deepener
303+
- right_nasolabial_deepener
304+
- left_dimpler
305+
- right_dimpler
231306
deepfake:
232307
_target_: facetorch.analyzer.predictor.FacePredictor
233308
downloader:

‎conf/tests.config.1.yaml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ defaults:
66
- analyzer/predictor/embed: r50_vggface_1m.yaml
77
- analyzer/predictor/verify: adaface_ir101_webface12m.yaml
88
- analyzer/predictor/fer: efficientnet_b2_8.yaml
9+
- analyzer/predictor/au: open_graph_swin_base.yaml
910
- analyzer/predictor/deepfake: efficientnet_b7.yaml
1011
- analyzer/predictor/align: synergy_mobilenet_v2.yaml
1112
- analyzer/utilizer/align: lmk3d_mesh_pose.yaml

‎conf/tests.config.2.yaml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ defaults:
66
- analyzer/predictor/embed: r50_vggface_1m.yaml
77
- analyzer/predictor/verify: r100_magface_unpg.yaml
88
- analyzer/predictor/fer: efficientnet_b0_7.yaml
9+
- analyzer/predictor/au: open_graph_swin_base.yaml
910
- analyzer/predictor/deepfake: efficientnet_b7.yaml
1011
- analyzer/predictor/align: synergy_mobilenet_v2.yaml
1112
- analyzer/utilizer/align: lmk3d_mesh_pose.yaml

‎conf/tests.config.3.yaml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ defaults:
66
- analyzer/predictor/embed: r50_vggface_1m.yaml
77
- analyzer/predictor/verify: adaface_ir101_webface12m.yaml
88
- analyzer/predictor/fer: efficientnet_b2_8.yaml
9+
- analyzer/predictor/au: open_graph_swin_base.yaml
910
- analyzer/predictor/deepfake: efficientnet_b7.yaml
1011
- analyzer/predictor/align: synergy_mobilenet_v2.yaml
1112
- analyzer/utilizer/align: lmk3d_mesh_pose.yaml

‎conf/tests.config.4.yaml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ defaults:
66
- analyzer/predictor/embed: r50_vggface_1m.yaml
77
- analyzer/predictor/verify: adaface_ir101_webface12m.yaml
88
- analyzer/predictor/fer: efficientnet_b2_8.yaml
9+
- analyzer/predictor/au: open_graph_swin_base.yaml
910
- analyzer/predictor/deepfake: efficientnet_b7.yaml
1011
- analyzer/predictor/align: synergy_mobilenet_v2.yaml
1112
- analyzer/utilizer/align: lmk3d_mesh_pose.yaml

0 commit comments

Comments
 (0)