Here is a sample video feed from the bigscreen beyond eye cams (had to zip it to get a bigger 25mb attachment limit):
WIN_20260805_22_06_02_Pro.zip
(This is just a quick clip I took while playing and should be enough to test cam splitting and improve pupil detection, but if you need more comprehensive examples with more eye angles I can record more)
And here's a frame from the video just so you can see it here:
-
The bigscreen beyond eye cam feed is a double wide video stream with the left eye on the left and right eye on the right, in my capture it's 800px wide and 400px tall, so a simple / generic auto detection method for merged cams could be something like: If the feed width is 2x the height, then split it into left and right. You could add an option to disable auto splitting just in case someone has a weird camera and needs to have it treated as 1 feed, but it seems pretty safe to have this logic as a default to make it work out of the box for feeds like this.
-
The second issue is the eye angle. Because of the shallow angle of the bigscreen eye cams, the top of the video feed is looking up past the eye and into unlit areas of the face like the eyebrows, or even past the eyebrows into the dark void. On the flip side, the shallow angle causes the bottom of the feed to be a very close up view of the area under the eye, which is close to the IR LEDs and much brighter. After playing with thresholds a bunch (I hacked in basic cam splitting pre-stereo support so I could play around with it) it seems like any brightness/darkness threshold for classifying the pupil will also classify the top of the feed, and if you were to try to classify the area around the pupil, the bright skin near the cam would also confuse that since for me that area appears to be even brighter than even the eye whites. This confuses the pupil detection a lot as it tries to draw a pupil oval around the entire top of the feed. In short, any brightness based threshold will start with a lot of junk data.
I'm not sure the best way to solve the pupil detection, but I have a few ideas:
- If I remember correctly, you run several thresholds and pick the best one. You could possibly improve it to handle BSB feeds by rejecting blobs that touch the edge of the feed.
- You could maybe use an edge detection algorithm that doesn't rely on a static threshold so that there's no need to find a threshold value that splits the darkness of the pupil and the lightness of the area around the pupil, since edge detection should be able to outline edges simultaneously in dark/black and grey/white in the same image. This might help to give better starting data since the pupil edge is a lot sharper than the gradual fade into black at the top of the feed.
- Another idea I had was to try to detect the edges of the eyelid and constrain the pupil search to within the intersection of the 2 circles made by the upper and lower eyelid. One way could be to find all arcs that longer than ~1/3 of the width of the feed and then check for pairs that face each other and have ends that are close, then fit an ellipse to each and form a pupil search area within those 2 arcs, perhaps offsetting each arc away from each other by a certain amount so that a partially occluded pupil isn't rejected by the search area if that's necessary.
Not sure how helpful those thoughts are, but hopefully with this example some solution can be found. If the eyelids get tracked, it would be super cool to get squint data out as well, because other tracking solutions like Bigscreen's eye tracking utility as well as Baballonia (both neural based) can be hit or miss for many people, and even when it works, it can be inaccurate or jittery and labor intensive to train and calibrate.
Here is a sample video feed from the bigscreen beyond eye cams (had to zip it to get a bigger 25mb attachment limit):
WIN_20260805_22_06_02_Pro.zip
(This is just a quick clip I took while playing and should be enough to test cam splitting and improve pupil detection, but if you need more comprehensive examples with more eye angles I can record more)
And here's a frame from the video just so you can see it here:
The bigscreen beyond eye cam feed is a double wide video stream with the left eye on the left and right eye on the right, in my capture it's 800px wide and 400px tall, so a simple / generic auto detection method for merged cams could be something like: If the feed width is 2x the height, then split it into left and right. You could add an option to disable auto splitting just in case someone has a weird camera and needs to have it treated as 1 feed, but it seems pretty safe to have this logic as a default to make it work out of the box for feeds like this.
The second issue is the eye angle. Because of the shallow angle of the bigscreen eye cams, the top of the video feed is looking up past the eye and into unlit areas of the face like the eyebrows, or even past the eyebrows into the dark void. On the flip side, the shallow angle causes the bottom of the feed to be a very close up view of the area under the eye, which is close to the IR LEDs and much brighter. After playing with thresholds a bunch (I hacked in basic cam splitting pre-stereo support so I could play around with it) it seems like any brightness/darkness threshold for classifying the pupil will also classify the top of the feed, and if you were to try to classify the area around the pupil, the bright skin near the cam would also confuse that since for me that area appears to be even brighter than even the eye whites. This confuses the pupil detection a lot as it tries to draw a pupil oval around the entire top of the feed. In short, any brightness based threshold will start with a lot of junk data.
I'm not sure the best way to solve the pupil detection, but I have a few ideas:
Not sure how helpful those thoughts are, but hopefully with this example some solution can be found. If the eyelids get tracked, it would be super cool to get squint data out as well, because other tracking solutions like Bigscreen's eye tracking utility as well as Baballonia (both neural based) can be hit or miss for many people, and even when it works, it can be inaccurate or jittery and labor intensive to train and calibrate.