How to improve Accuracy

[Manishsinghrajput98]

can you share the percentages of accuracy. and how much images needed for training and also how much class we can use ?

[yong2khoo-lm]

can you share the percentages of accuracy. and how much images needed for training and also how much class we can use ?

I do have the similar question. In my cases, if there are around 8 persons (each around 5-10 samples), the 'probabilities' using SVM is fine ( > 0.7+). But if it exceeds 20 persons, the probability declines to around (0.2 to 0.3). Probably i lack of something. Can you share yours too?

[Manishsinghrajput98]

@kyygit i think this repository is not useful for production. this repository is best for colleges projects. i have tried with 25 classes but accuracy is very low. if you fined any solution for this so please let me know.

thanks

[yong2khoo-lm]

hm... not sure we are talking about the same thing.
do you mean 'low probability' = 'low accuracy'? or your case is totally 'wrong matching'?

[ldulcic]

@Manishsinghrajput98 It really depends on your use-case and your data.
In Ars Futura, we use this lib for face recognition smart lock, we have 26 employees with ~30-40 images for every employee. We managed to get very high accuracy of >95%, but I did tweak the dataset couple of times. Lot of things matter, head pose matters a lot, lighting matters, size of face in image matters. That's why your use-case is important, you won't use the same dataset if you are implementing face recognition smart lock or if you are trying to recognise faces from some public street camera.
What are your use-cases @Manishsinghrajput98 @kyygit ?

[yong2khoo-lm]

Hi @ldulcic , thanks for your sharing.

we have 26 employees with ~30-40 images for every employee

30-40 images. That really opens my eyes~

In my case, it is an attendance system for around hundreds of workers.
Probably i need to clarify my words. (to avoid the confusion of 'probability' and 'accuracy' in my case)
So far, as observed, with around 20 persons (with about 10 images each), by using SVM, the 'probability' is somewhere around 0.2 to 0.4. However, the identified face is still 'mostly correct'.
In short, it means that, the results are 'correct', despite with low 'confidence'. (As i assume, a high confidence would have a probability of maybe beyond 0.7+)
Still, I got to add more persons to further verify these numbers.

@ldulcic Allow me to ask, when you get an accuracy of >95%, is it, the softmax classifier is 'confident' of the matches? What is the 'probability' of the result classified?

Thanks in advance.

[ldulcic]

@kyygit I had the same problem when I first started to work on face recognition smart lock. Low confidences but mostly correct labels.
First thing I did was to add more images of people, that did help with confidences (probabilities) but not enough to get high confidences. After debugging and looking at the bad frames from camera I realised I was solving the wrong problem. I was trying to classify people in all kinds of head poses, all kinds of lighting conditions and at a far away. I googled quickly and found 2 examples of bad images for face recognition.

Head poses in these images are not very good for face recognition, you see only the side of the face, in lot of situations there are not enough data on the image where person is turned on one side. First image is a good example, you only see one half of the face at bad lighting. If lighting was little bit worse, you as a human could sometimes mistaken this person for someone else when you look at it from the side.

Side head pose is just an example, model performance is deteriorated mostly by side head poses, bad lighting positions and people faces which are far away in image. This gives model too much variation in data which is why it doesn't produce high confidences. When I started I didn't realise this so my dataset was filled with these "bad images" because I thought "more data is always better".

After realising this I restricted my problem to front-facing head poses in reasonable lighting conditions which are relatively close to camera, so I "refactored" my dataset to include images like this. From this point on I was getting pretty good confidences. Camera didn't catch you half-way across the hallway but when you came close to the door and looked at the camera, the door would open within a second or two.

To answer you question, when I wrote accuracy > 95%, I was referring to accuracy on my test set that I compiled, that's softmax classifier score (probability).

Sorry for the late answer, I will write a blog post about this where I'll explain in more detail what I did. I just didn't have enough time to do it lately, hopefully it will be soon 🙂

PS. If anyone else is reading this...This lib saves you the hassle of manually writing the code for classifying people, you can just plug your data and get working system out. But it's all about the data, your face recognition system will only be as good as your data is.

Cheers!

[yong2khoo-lm]

To answer you question, when I wrote accuracy > 95%, I was referring to accuracy on my test set that I compiled, that's softmax classifier score (probability).

Thank you. It looks promising!

Sorry for the late answer

Not at all! Your experience is Gold to me :) I am grateful for your patience and kindness to share your work.
The attendance system that I am going to build, is mostly similar to yours.

your face recognition system will only be as good as your data is

You have highlighted the main point that i have overlooked - 'data'. With your detailed advice on the 'frame quality', i would be more careful and picky on the 'face enrolment' part now.

[yong2khoo-lm]

not sure @Manishsinghrajput98 has any concern?

[marcolyh]

Hi, i am working on an attendance system as well. To provide context:

Facenet was used as an embedding generator to generate facial embeddings

As for the classifier, i am not using any, instead, i am comparing the cosine similarity between embeddings to find out the highest confident between two embeddings.

Reason i am not using a classifier is that i am required to add new users (facial embeddings) to the system occasionally, thus it seems that using a classifier is not ideal as you would require to retrain it again and again.

But i am getting a lot wrong labels predicted for each user, ie. The person was wrongly recognized with high confidence.

How should i tackle my situation where new users may be added while maintaining correct labels with high confidence?

[Manishsinghrajput98]

You can added multiple checks for removing false cases. Also you can add strong face detection models. For face recognition you can add more data for each position. Like beard etc. Added one more checks for removing false positives histogram matching and face recognition library.

…

On Fri 18 Aug, 2023, 5:05 PM marcolyh, ***@***.***> wrote: Hi, i am working on an attendance system as well. To provide context: Facenet was used as an embedding generator to generate facial embeddings As for the classifier, i am not using any, instead, i am comparing the cosine similarity between embeddings to find out the highest confident between two embeddings. Reason i am not using a classifier is that i am required to add new users (facial embeddings) to the system occasionally, thus it seems that using a classifier is not ideal as you would require to retrain it again and again. But i am getting a lot wrong labels predicted for each user, ie. The person was wrongly recognized with high confidence. How should i tackle my situation where new users may be added while maintaining correct labels with high confidence? — Reply to this email directly, view it on GitHub <#22 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AI74CZANUUIEEOCVXB7N54TXV5HQJANCNFSM4S666SDQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[marcolyh]

@Manishsinghrajput98 Thanks for replying. Can you explain more on adding multiple checks?

Sorry but i forgot to mention that i am using MTCNN as my facial detector. I am now testing out on images instead of video(frames)

my biggest restrain is still on how can i add new users (facial embeddings), without retraining the whole classifier SVM. Or should i just use the method i described above? - Using cosine similarity to compare embeddings. Is there a more efficient and productive way?

[ldulcic]

@marcolyh you have a database of embeddings where each person has multiple embeddings? When you try to classify person in image you compare its embedding to the whole database?

Retraining of classifier is usually very fast. I had a dataset with ~1000 embeddings (images), retraining lasted a second or two. Why is this a bottleneck for you?