Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Evaluation protocol IV get the different result from paper #15

Open
gangeqian opened this issue Jul 14, 2021 · 1 comment
Open

Evaluation protocol IV get the different result from paper #15

gangeqian opened this issue Jul 14, 2021 · 1 comment

Comments

@gangeqian
Copy link

gangeqian commented Jul 14, 2021

Hi
I evaluation the four testing protocols in paper. I don't confirm my method and result is correct.
there is 4 file list txt。[0] gallery_black_glass.txt [1]gallery_no_glass.txt [2]probe_black_glass.txt [3]probe_no_glass.txt
4 test protocols:
I) [1] vs [3]
II) [0] vs [2]
III) [1] vs [2]
IV) [0][1] vs [2][3] (Gallery images contain both eyeglass images and non eyeglass images, so as probe image.)
the result from paper show that III) has the better result than [IV)。
for ResNet-22-A RPR@FAR=10e-4 III): 88.13 IV):78.17 Rank1 III): 95.61 IV):92.31

in my experiment test, I use LightCNN29 model to evaluate, ang get the following result:
for LightCNN29 RPR@FAR=10e-4 III): 87.49 IV):87.03 Rank1 III): 93.39 IV):98.54
as showed, III) has litter better result than IV) and Rank1 smaller than IV).

I mixture [0][1] as the new Gallery, and mixture [2][3] as the new probe in protocol IV). which should has less degree of difficulty than protocol III), because all of pairs in III) contains non-eyeglass and eyeglass, otherwise,IV)contains some simple pairs of non-eyeglass.

So I don't think IV) could get better than III) from my consider and test

where is wrong for my test? could you tell me

@xsacha
Copy link

xsacha commented Aug 12, 2021

Would it depend on which combinations were taken?
Was there any given .txt file provided with the pairs list for any of the protocols?

Could you provide the pair lists you have used yourself and I could compare my results.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants