Feat/batch predict age and gender

[NatLee]

Tickets

#441
#678
#1069
#1101

What has been done

With this PR, new predicts function to support batch predictions.

How to test

Use this class, user can load model and predict in batch.

class AgeGenderModel():
    """
    Age and gender model
    """
    def __init__(self):
        self.target_size = (224, 224)
        self.age_model, self.gender_model = self.load()

    def load(self) -> Tuple:
        age_model = modeling.build_model(task="facial_attribute", model_name="Age")
        gender_model = modeling.build_model(task="facial_attribute", model_name="Gender")
        return age_model, gender_model

    def process_data(self, data: np.ndarray) -> np.ndarray:
        """
        Process input image data
        """
        img_content = data[:, :, ::-1] # rgb to bgr
        img_content = preprocessing.resize_image(
            img=img_content,
            target_size=self.target_size
        )
        return img_content

    def predict(self, data: np.ndarray) -> Tuple[np.ndarray, np.ndarray]:
        """
        Predict age and gender for one input image
        """
        img_content = self.process_data(data)
        age = self.age_model.predict(img_content)
        gender = self.gender_model.predict(img_content)
        return gender, age

    def predicts(self, data: List[np.ndarray]) -> Tuple[List, List]:
        """
        Predict age and gender for batch input images
        """
        img_contents = [self.process_data(img_content) for img_content in data]
        ages = self.age_model.predicts(img_contents)
        genders = self.gender_model.predicts(img_contents)
        return genders, ages 

Time Costs for 10 Images:

Original for-loop prediction: 1.91108 seconds

Loading split Age and Gender models: 0.00002 seconds
Batch prediction: 0.58243 seconds
Single prediction: 0.03749 seconds

Here's my test script:

from deepface import DeepFace
import cv2
import time
# Load one face
img = cv2.imread("./test.png")

# Make it as a list
imgs = [img] * 10

# For-loop predict
start = time.time()
print("=====For-loop Predict=====")
for img in imgs:
    objs = DeepFace.analyze(
    img_path=img,
    actions=['age', 'gender'],
    )
print(f"Time: {time.time() - start:.5f}s")

# Load models
print("=====Load Split Models=====")
start = time.time()
model = AgeGenderModel()
print(f"Time: {time.time() - start:.5f}s")

# Batch Predict
start = time.time()
genders, ages = model.predicts(imgs)
print("=====Batch Predict=====")
print(genders, ages)
print(f"Time: {time.time() - start:.5f}s")

# Single Predict
start = time.time()
gender, age = model.predict(img)
print("=====Single Predict=====")
print(gender, age)
print(f"Time: {time.time() - start:.5f}s")

predicts is placed in age and gender clients to keep the DeepFace.analyze function logic.

Thank you for taking time to go through this feedback. :)

[h-alice]

Bump
Really needs this feature.

[serengil]

I don't support having another predicts function. Instead, you can add that logic under predict.

1- predict accepts both single image and list of images as

img: Union[np.ndarray, List[np.ndarray]]

2- in predict function, you can check the type of img, and redirect it to your logic if it is list as

if isinstance(img, np.ndarray):
   # put old predict logic here
elif isinstance(img, np.ndarray):
   # put your batch processing logic here

3- this new logic is worth to have its own unit tests. possibly, you can add some unit tests here.

4- return type of predict should be Union[np.float64, np.ndarray]

5- You should also update the interface in DeepFace.py

[serengil]

cast img to img itself please, in that way we will not need else condition

[serengil]

you squeezed imgs and then expand its dimension, instead can we decide the input is single or many images when predict called?

if isinstance(img, list):
   # then it is many images
elif len(img.shape) == 3:
   # then it is single image
elif len(img.shape) == 4 and img.shape[0] == 1:
   # then it is single image
elif len(img.shape) == 4 and img.shape[0] > 1:
   # then it is many images
else:
   raise error

[serengil]

Actions failed because of linting - link

************* Module deepface.models.demography.Age
pylint: Command line or configuration file:1: UserWarning: 'Exception' is not a proper value for the 'overgeneral-exceptions' option. Use fully qualified name (maybe 'builtins.Exception' ?) instead. This will cease to be checked at runtime in 3.1.0.
deepface/models/demography/Age.py:70:0: C0303: Trailing whitespace (trailing-whitespace)
************* Module deepface.models.demography.Emotion
deepface/models/demography/Emotion.py:88:0: C0303: Trailing whitespace (trailing-whitespace)

[serengil]

model.predict causes memory issue when it is called in a for loop, that is why we call it as self.model(img, training=False).numpy()[0, :]

in your design, if this is called in a for loop, still it will cause memory problem.

IMO, if it is single image, we should call self.model(img, training=False).numpy()[0, :], it is many faces then call self.model.predict_on_batch

[h-alice]

Thank you for sharing your perspective on this matter.

We found the issue you mentioned is also mentioned in this page: tensorflow/tensorflow#44711. We believe it’s being resolved.

Furthermore, if we can utilize the batch prediction method provided in this PR, we may be able to avoid repeatedly calling the predict function within a loop of unrolled batch images, which is the root cause of the memory issue you described.

We recommend that you consider retaining our batch prediction method.

[serengil]

hey, even though this is sorted in newer tf versions, many users using old tf versions raise tickets about this problem. so, we should consider the people using older tf version. that is why, i recommend to use self.model(img, training=False).numpy()[0, :] for single images, and self.model.predict_on_batch for batches.

[serengil]

please do not create resp_objects in advance, instead create resp_object in the loop and append it to resp_objects similar to previous design

[serengil]

i really don't like to set resp_objects with idx, recommend something in the code

[NatLee]

i simplified it with zip and update in 52a38ba
it looks more clean and pythonic now :)

[serengil]

please set resp_objects to an empty list here as

resp_objects = []

[NatLee]

resolved in #1396 (comment)

[serengil]

seems this is not resovled.

before for loop, you should create a resp_objects as resp_objects = []

in for loop, you should create resp_object as resp_object = {} and set its features.

in summary, instead resp_objects[idx]["emotion"] = {}, we should have something like resp_object["emotion"] = {}

[serengil]

set resp_object instead of resp_objects as

resp_object["age"] = int(age)

do this for each attribute please

[NatLee]

are you referring to whether the attribute mentioned refers to the percentage of emotion or gender?

[serengil]

i don't want you to use idx at all.

[NatLee]

i initial the resp_objects renamed as results with face region and confidence.
https://github.com/NatLee/deepface/blob/52a38ba21a9b7edb331e2a6f25e68115ca1a663c/deepface/modules/demography.py#L155-L157

and use update function to write result of models like this
https://github.com/NatLee/deepface/blob/52a38ba21a9b7edb331e2a6f25e68115ca1a663c/deepface/modules/demography.py#L180-L193

the newest version didn't use any idx in the code.

[NatLee]

ahh the newest version is in the other branch which hasnt been merged yet

[serengil]

if you use it like this, you don't have to have if and else blocks.

img = alpha_embedding = np.asarray(img)

[NatLee]

i think @h-alice 's concern is likely about the img variable being compromised. 🤔

[h-alice]

@serengil We appreciate your advice.
We will implement your recommended approach later.