Skip to content

[Identity] Best frame detector for document capture #5861

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
Pearapps wants to merge 12 commits into
base: master
Choose a base branch
from
Open

[Identity] Best frame detector for document capture #5861

Pearapps wants to merge 12 commits into from

Conversation

@Pearapps
Copy link
Collaborator

@Pearapps Pearapps commented Nov 25, 2025 •
edited
Loading

This PR adds in a 1.0 second sampling period after we find a valid document capture frame, instead of picking that first frame, to see if we can capture a better quality frame during that window. If we do, we use that frame instead of the first frame.

This allows us to squeak out a slightly better quality image which should increase the likelihood of a successful flow.

Testing:

I added logs to confirm the best frame within the second was chosen. See these two runs:

[BestFrame][CANDIDATE] side=front classScore=0.82373047 iou=Optional(0.97187847) rawScore=0.8978045
[BestFrame][CANDIDATE] side=front classScore=0.82177734 iou=Optional(0.96825606) rawScore=0.8950167
[BestFrame][CANDIDATE] side=front classScore=0.8208008 iou=Optional(0.9831338) rawScore=0.9019673
[BestFrame][CANDIDATE] side=front classScore=0.8261719 iou=Optional(0.964613) rawScore=0.8953924
[BestFrame][CANDIDATE] side=front classScore=0.8251953 iou=Optional(0.9963604) rawScore=0.91077787
[BestFrame][CANDIDATE] side=front classScore=0.82177734 iou=Optional(0.96472746) rawScore=0.8932524
[BestFrame][CANDIDATE] side=front classScore=0.83154297 iou=Optional(0.98709565) rawScore=0.9093193
[BestFrame][CANDIDATE] side=front classScore=0.83740234 iou=Optional(0.9950805) rawScore=0.9162414
[BestFrame][CANDIDATE] side=front classScore=0.8251953 iou=Optional(0.9823598) rawScore=0.9037776
[BestFrame][CANDIDATE] side=front classScore=0.8198242 iou=Optional(0.9579156) rawScore=0.8888699
[BestFrame][CANDIDATE] side=front classScore=0.8154297 iou=Optional(0.98828363) rawScore=0.90185666
[BestFrame][CANDIDATE] side=front classScore=0.8178711 iou=Optional(0.9523371) rawScore=0.88510406
[BestFrame][CANDIDATE] side=front classScore=0.8144531 iou=Optional(0.9586113) rawScore=0.8865322
[BestFrame][CANDIDATE] side=front classScore=0.8149414 iou=Optional(0.97049904) rawScore=0.8927202
[BestFrame][CANDIDATE] side=front classScore=0.81396484 iou=Optional(0.9867102) rawScore=0.9003375
[BestFrame][CANDIDATE] side=front classScore=0.8125 iou=Optional(0.98574716) rawScore=0.89912355
[BestFrame][CANDIDATE] side=front classScore=0.81640625 iou=Optional(0.9698961) rawScore=0.89315116
[BestFrame][CANDIDATE] side=front classScore=0.8154297 iou=Optional(0.9877475) rawScore=0.90158856
[BestFrame][CANDIDATE] side=front classScore=0.82177734 iou=Optional(0.98230475) rawScore=0.9020411
[BestFrame][CANDIDATE] side=front classScore=0.8203125 iou=Optional(0.99534494) rawScore=0.9078287
[BestFrame][CANDIDATE] side=front classScore=0.81396484 iou=Optional(0.9838868) rawScore=0.8989258
[BestFrame][CANDIDATE] side=front classScore=0.8120117 iou=Optional(0.98820335) rawScore=0.9001075
[BestFrame][CANDIDATE] side=front classScore=0.8173828 iou=Optional(0.95267916) rawScore=0.885031
[BestFrame][CANDIDATE] side=front classScore=0.8154297 iou=Optional(0.9734545) rawScore=0.8944421
[BestFrame][CANDIDATE] side=front classScore=0.8222656 iou=Optional(0.98201835) rawScore=0.902142
[BestFrame][CANDIDATE] side=front classScore=0.8276367 iou=Optional(0.982417) rawScore=0.90502685
[BestFrame][CANDIDATE] side=front classScore=0.82958984 iou=Optional(0.98838174) rawScore=0.9089858
[BestFrame][CANDIDATE] side=front classScore=0.8339844 iou=Optional(0.9883256) rawScore=0.911155
[BestFrame][CANDIDATE] side=front classScore=0.8339844 iou=Optional(0.99016505) rawScore=0.9120747
[BestFrame][CANDIDATE] side=front classScore=0.82666016 iou=Optional(0.98627084) rawScore=0.90646553
[BestFrame][CANDIDATE] side=front classScore=0.8300781 iou=Optional(0.99137044) rawScore=0.9107243
[BestFrame][PICKED] side=front classScore=0.83740234 variance=2.0 iou=Optional(0.9950805)  finalScore=0.9162414



----




[BestFrame][CANDIDATE] side=back classScore=0.80126953 iou=Optional(0.97421) rawScore=0.8877398
[BestFrame][CANDIDATE] side=back classScore=0.79345703 iou=Optional(0.96649027) rawScore=0.87997365
[BestFrame][CANDIDATE] side=back classScore=0.7919922 iou=Optional(0.9616109) rawScore=0.87680155
[BestFrame][CANDIDATE] side=back classScore=0.7841797 iou=Optional(0.9574268) rawScore=0.87080324
[BestFrame][CANDIDATE] side=back classScore=0.7910156 iou=Optional(0.9740467) rawScore=0.88253117
[BestFrame][CANDIDATE] side=back classScore=0.8071289 iou=Optional(0.95257145) rawScore=0.87985015
[BestFrame][CANDIDATE] side=back classScore=0.81347656 iou=Optional(0.9688502) rawScore=0.89116335
[BestFrame][CANDIDATE] side=back classScore=0.8017578 iou=Optional(0.98167175) rawScore=0.8917148
[BestFrame][CANDIDATE] side=back classScore=0.8198242 iou=Optional(0.9803921) rawScore=0.90010816
[BestFrame][CANDIDATE] side=back classScore=0.82714844 iou=Optional(0.97893614) rawScore=0.9030423
[BestFrame][CANDIDATE] side=back classScore=0.8203125 iou=Optional(0.9886427) rawScore=0.9044776
[BestFrame][CANDIDATE] side=back classScore=0.82470703 iou=Optional(0.9930783) rawScore=0.90889263
[BestFrame][CANDIDATE] side=back classScore=0.8286133 iou=Optional(0.98770297) rawScore=0.9081581
[BestFrame][CANDIDATE] side=back classScore=0.83984375 iou=Optional(0.97802275) rawScore=0.9089333
[BestFrame][CANDIDATE] side=back classScore=0.8388672 iou=Optional(0.99363077) rawScore=0.916249
[BestFrame][CANDIDATE] side=back classScore=0.8334961 iou=Optional(0.9872076) rawScore=0.9103519
[BestFrame][CANDIDATE] side=back classScore=0.83740234 iou=Optional(0.9821808) rawScore=0.9097916
[BestFrame][CANDIDATE] side=back classScore=0.83203125 iou=Optional(0.98403317) rawScore=0.9080322
[BestFrame][CANDIDATE] side=back classScore=0.8339844 iou=Optional(0.97468317) rawScore=0.9043338
[BestFrame][CANDIDATE] side=back classScore=0.828125 iou=Optional(0.9903303) rawScore=0.9092276
[BestFrame][CANDIDATE] side=back classScore=0.8222656 iou=Optional(0.9938987) rawScore=0.9080821
[BestFrame][CANDIDATE] side=back classScore=0.8261719 iou=Optional(0.98660165) rawScore=0.90638673
[BestFrame][CANDIDATE] side=back classScore=0.8339844 iou=Optional(0.9806582) rawScore=0.9073213

[BestFrame][PICKED] side=back classScore=0.8388672 variance=2.0 iou=Optional(0.99363077) finalScore=0.916249

@Pearapps Pearapps marked this pull request as ready for review November 26, 2025 14:32
@Pearapps Pearapps requested review from a team as code owners November 26, 2025 14:32
@Pearapps Pearapps changed the title [Draft] [Identity] Best frame detector for document capture [Identity] Best frame detector for document capture Nov 26, 2025
@cjmisenas-stripe
Copy link

cjmisenas-stripe commented Nov 26, 2025

Thanks for adding the test logs @Pearapps . Had a couple of questions /comments

cjmisenas-stripe
cjmisenas-stripe reviewed Nov 26, 2025
View reviewed changes
...y/StripeIdentity/Source/NativeComponents/ViewControllers/DocumentCaptureViewController.swift
exifMetadata: best.exif,
method: .autoCapture
)
sheetController?.analyticsClient.updateBlurScore(blurResult.variance, for: documentSide)
Copy link

@cjmisenas-stripe cjmisenas-stripe Nov 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could be in a later PR but it might also be useful to add:

  • Number of frames we looked at
  • Individual scores
  • Final overall best score

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@luisv-stripe luisv-stripe Awaiting requested review from luisv-stripe

@cjmisenas-stripe cjmisenas-stripe Awaiting requested review from cjmisenas-stripe

Requested changes must be addressed to merge this pull request.

Assignees

No one assigned

Labels

Identity

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@Pearapps @cjmisenas-stripe