jeoresearch.github.io/eyetracking.html at main · JEOresearch/jeoresearch.github.io · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8" />
  <link rel="icon" href="images/favicon.ico" type="image/x-icon" />
  <meta name="viewport" content="width=device-width, initial-scale=1.0" />

  <!-- Google tag (gtag.js) -->
  <script async src="https://www.googletagmanager.com/gtag/js?id=G-WZ7XJMTZF5"></script>
  <script>
    window.dataLayer = window.dataLayer || [];
    function gtag(){dataLayer.push(arguments);}
    gtag('js', new Date());
    gtag('config', 'G-WZ7XJMTZF5');
  </script>
  <script src="analytics.js" defer></script>

  <title>DIY Eye Tracking with a Front Camera | JEO Research</title>

  <meta
    name="description"
    content="Front camera eye tracking tutorial from JEO Research, including resources for webcam-based gaze tracking and accessible eye tracking projects."
  />

  <link rel="canonical" href="https://www.jeoresearch.com/eyetracking.html" />
  <link rel="stylesheet" href="style.css" />
  <script type="application/ld+json">
  {
    "@context": "https://schema.org",
    "@graph": [
      {
        "@type": "WebPage",
        "@id": "https://www.jeoresearch.com/eyetracking.html#webpage",
        "url": "https://www.jeoresearch.com/eyetracking.html",
        "name": "DIY Eye Tracking with a Front Camera",
        "isPartOf": {
          "@id": "https://www.jeoresearch.com/#website"
        },
        "author": {
          "@id": "https://www.jeoresearch.com/#person"
        },
        "publisher": {
          "@id": "https://www.jeoresearch.com/#organization"
        },
        "breadcrumb": {
          "@id": "https://www.jeoresearch.com/eyetracking.html#breadcrumb"
        },
        "mainEntity": {
          "@id": "https://www.jeoresearch.com/eyetracking.html#article"
        }
      },
      {
        "@type": "TechArticle",
        "@id": "https://www.jeoresearch.com/eyetracking.html#article",
        "headline": "DIY Eye Tracking with a Front Camera",
        "author": {
          "@id": "https://www.jeoresearch.com/#person"
        },
        "publisher": {
          "@id": "https://www.jeoresearch.com/#organization"
        },
        "mainEntityOfPage": {
          "@id": "https://www.jeoresearch.com/eyetracking.html#webpage"
        },
        "about": {
          "@id": "https://www.jeoresearch.com/eyetracking.html#software"
        }
      },
      {
        "@type": "SoftwareSourceCode",
        "@id": "https://www.jeoresearch.com/eyetracking.html#software",
        "name": "FrontCameraTracker",
        "codeRepository": "https://github.com/JEOresearch/EyeTracker/tree/main/FrontCameraTracker",
        "programmingLanguage": "Python",
        "author": {
          "@id": "https://www.jeoresearch.com/#person"
        },
        "publisher": {
          "@id": "https://www.jeoresearch.com/#organization"
        }
      },
      {
        "@type": "BreadcrumbList",
        "@id": "https://www.jeoresearch.com/eyetracking.html#breadcrumb",
        "itemListElement": [
          {
            "@type": "ListItem",
            "position": 1,
            "name": "Home",
            "item": "https://www.jeoresearch.com/"
          },
          {
            "@type": "ListItem",
            "position": 2,
            "name": "Front-Camera Eye Tracking",
            "item": "https://www.jeoresearch.com/eyetracking.html"
          }
        ]
      }
    ]
  }
  </script>
</head>

<body>
  <header class="site-header">
    <div class="header-inner">
      <a class="brand" href="index.html"><span class="brand-jeo">JEO</span><span class="brand-research">research</span></a>

      <nav class="nav">
        <div class="nav-dropdown">
          <button class="nav-trigger" type="button">Home▼</button>
          <div class="nav-menu">
            <a href="index.html">Home Page</a>
            <a href="aboutme.html">About Me</a>
            <a href="news.html">Appearances</a>
          </div>
        </div>
        <div class="nav-dropdown">
          <button class="nav-trigger" type="button">EyeTracking▼</button>
          <div class="nav-menu">
            <a href="eyecameras.html">Camera Comparison</a>
            <a href="eyetracking.html">DIY Eye Tracker (Glasses)</a>
            <a href="vrtracker.html">DIY Eye Tracker (VR)</a>
            <a href="headgaze.html">DIY Head Tracker (Webcam)</a>
            <a href="irsafety.html">Infrared Safety</a>
          </div>
        </div>
        <div class="nav-dropdown">
          <button class="nav-trigger" type="button">ARVRLab▼</button>
          <div class="nav-menu">
            <a href="ccslab.html">Lab Page</a>
            <a href="ccslab.html#apply">Apply</a>
            <a href="ccslab/team.html">Members</a>
            <a href="students/timelines.html">Timelines</a>
            <a href="ccslab/xr.html">XR Research</a>
            <a href="ccslab/intelligent.html">Intelligent Agents</a>
          </div>
        </div>
        <a href="research.html">Publications</a>
        <a href="https://github.com/JEOresearch">GitHub</a>
      </nav>
    </div>
  </header>

  <main class="page">
    <section class="hero tutorial-hero">
      <div class="hero-text">
        <h1>DIY Eye Tracking with a Front Camera</h1>
        <p class="eyebrow">By <a href="aboutme.html">Jason Orlosky, PhD</a></p>

        <p class="intro">
          This tutorial shows how to build a simple front-camera eye tracker
          with Python. The tracker estimates a 3D gaze origin and direction
          that can be used for screen control, interaction experiments, and
          gaze visualization.
        </p>

        <div class="profile-links" aria-label="Tutorial links">
          <a href="https://youtu.be/Zed2jPgks9s">Video Summary</a>
          <span>|</span>
          <a href="https://youtu.be/Gh8LS9erugE">The Math behind the Eye Tracking</a>
          <span>|</span>
          <a href="https://github.com/JEOresearch/EyeTracker/tree/main/FrontCameraTracker">Python Script on GitHub</a>
        </div>
      </div>

      <div class="hero-image">
        <a href="https://amzn.to/41x8p2W" target="_blank" rel="noopener" data-product="GC0308 Eye Tracking Camera" data-category="Camera">
          <img src="images/frontcamera.jpg" alt="Front camera eye tracking tutorial" />
        </a>
      </div>
      <p class="channel-support">
          To help support this software and other open-source projects, please consider
          <a href="https://www.youtube.com/channel/UCS9oAlDpRkq14yxmpmw2jpA?sub_confirmation=1">subscribing</a>
          to or <a href="https://www.youtube.com/@jeoresearch/join">joining</a> my YouTube channel.
        </p>
    </section>

    <section class="section tutorial-steps">
      <h2>Building the Tracker</h2>

      <article class="tutorial-step">
        <h3>Step 1: Gather the parts</h3>

        <p>
          You need a small <a href="https://amzn.to/41x8p2W" data-product="GC0308 Eye Tracking Camera" data-category="Camera">eye tracking camera</a> (Fig. 1),
          a low cost <a href="https://amzn.to/444eXIa" data-product="Board Camera" data-category="Camera">board camera</a> (Fig. 2),
          <a href="https://amzn.to/43SznVl" data-product="USB Extension Cables" data-category="Cable">USB extension cables</a> if you
          need more cable length, a pair of
          <a href="https://amzn.to/42KBLN3" data-product="Sunglasses Frame" data-category="Mount">cheap sunglasses</a> with the
          lenses removed (Fig. 3), <a href="https://amzn.to/4pi6wBn" data-product="Soft Wire Cabling" data-category="Mount">soft wire cabling</a>
          (Fig. 4)
          or another flexible material for mounting, and
          <a href="https://amzn.to/44ef97I" data-product="Electrical Tape" data-category="Accessory">electrical tape</a> for securing
          loose cable sections. A low-cost GC0308 eye tracking camera works for
          this build, and inexpensive dollar-store glasses are enough because
          the frame is only used to hold the camera near the eye.
          Review <a href="irsafety.html">IR safety</a> and evaluate the camera's
          infrared output before using it close to the eye.
        </p>

        <div class="step-images">
          <figure>
            <a href="https://amzn.to/41x8p2W" target="_blank" rel="noopener" data-product="GC0308 Eye Tracking Camera" data-category="Camera">
              <img src="images/frontcamera/gc0308.jpg" alt="GC0308 eye tracking camera" loading="lazy" />
            </a>
            <figcaption><a href="https://amzn.to/41x8p2W" data-product="GC0308 Eye Tracking Camera" data-category="Camera">Fig. 1. GC0308 IR camera</a></figcaption>
          </figure>
          <figure>
            <a href="https://amzn.to/444eXIa" target="_blank" rel="noopener" data-product="Board Camera" data-category="Camera">
              <img src="images/frontcamera/boardcamera.jpg" alt="Low-cost board camera for DIY eye tracking" loading="lazy" />
            </a>
            <figcaption><a href="https://amzn.to/444eXIa" data-product="Board Camera" data-category="Camera">Fig. 2. Board (environment) camera</a></figcaption>
          </figure>
          <figure>
            <a href="https://amzn.to/42KBLN3" target="_blank" rel="noopener" data-product="Sunglasses Frame" data-category="Mount">
              <img src="images/frontcamera/nolens.jpg" alt="Glasses frame with lenses removed" loading="lazy" />
            </a>
            <figcaption><a href="https://amzn.to/42KBLN3" data-product="Sunglasses Frame" data-category="Mount">Fig. 3. Glasses frame with no lenses</a></figcaption>
          </figure>
          <figure>
            <a href="https://amzn.to/4pi6wBn" target="_blank" rel="noopener" data-product="Soft Wire Cabling" data-category="Mount">
              <img src="images/frontcamera/wire.jpg" alt="Soft wire for mounting the camera" loading="lazy" />
            </a>
            <figcaption><a href="https://amzn.to/4pi6wBn" data-product="Soft Wire Cabling" data-category="Mount">Fig. 4. Soft wire</a></figcaption>
          </figure>
        </div>
      </article>

      <article class="tutorial-step">
        <h3>Step 2: Attach the camera to the glasses</h3>

        <p>
          Wrap the <a href="https://amzn.to/4pi6wBn" data-product="Soft Wire Cabling" data-category="Mount">soft wire</a>
          around the <a href="https://amzn.to/41x8p2W" data-product="GC0308 Eye Tracking Camera" data-category="Camera">camera</a>
          and cable, then attach it to
          the <a href="https://amzn.to/42KBLN3" data-product="Sunglasses Frame" data-category="Mount">glasses frame</a>
          to create the completed camera mount (Fig. 5).
          Use <a href="https://amzn.to/44ef97I" data-product="Electrical Tape" data-category="Accessory">electrical tape</a> if you need to
          secure the camera cable or keep the mount from sliding. The goal is
          to make a firm but adjustable mount:
          the camera should stay in place after you reposition it, but you
          should still be able to bend the mount until the eye is fully visible
          in the camera feed (Fig. 6).
        </p>

        <div class="step-images">
          <figure>
            <a href="https://amzn.to/41x8p2W" target="_blank" rel="noopener" data-product="GC0308 Eye Tracking Camera" data-category="Camera">
              <img src="images/frontcamera/camerasmounted.jpg" alt="Eye tracking cameras mounted to glasses" loading="lazy" />
            </a>
            <figcaption><a href="https://amzn.to/41x8p2W" data-product="GC0308 Eye Tracking Camera" data-category="Camera">Fig. 5. Both cameras mounted</a></figcaption>
          </figure>
          <figure>
            <a href="https://amzn.to/41x8p2W" target="_blank" rel="noopener" data-product="GC0308 Eye Tracking Camera" data-category="Camera">
              <img src="images/frontcamera/eyeinview.jpg" alt="Eye fully visible in the mounted camera view" loading="lazy" />
            </a>
            <figcaption><a href="https://amzn.to/41x8p2W" data-product="GC0308 Eye Tracking Camera" data-category="Camera">Fig. 6. Adjusted mount with the eye fully visible</a></figcaption>
          </figure>
        </div>
      </article>

      <article class="tutorial-step">
        <h3>Step 3: Install the Python dependencies</h3>

        <p>
          Download <code>Orlosky3DEyeTracker.py</code> and install the required
          Python packages. The tracker uses OpenCV and NumPy for image
          processing. Tkinter is used for the startup window and is included
          with many Python installations. The optional <code>gl_sphere.py</code>
          file enables the OpenGL sphere visualization, but the tracker will
          still run without it.
        </p>

        <pre><code>pip install opencv-python numpy</code></pre>

      </article>

      <article class="tutorial-step">
        <h3>Step 4: Run the tracker</h3>

        <p>
          Open a terminal in the folder containing the script and run it with
          Python. On startup, the tracker opens a small window titled
          <code>Select Input Source</code>. Choose your camera from the dropdown
          and click <code>Start Camera</code>, as shown in Fig. 7. You can also use
          <code>Browse Video</code> to test the tracker on an existing
          <code>.mp4</code> or <code>.avi</code> file.
        </p>

        <pre><code>python Orlosky3DEyeTracker.py</code></pre>

        <p>
          If you found this software/tutorial useful, please consider
          <a href="https://www.youtube.com/channel/UCS9oAlDpRkq14yxmpmw2jpA/?sub_confirmation=1">subscribing</a> or
          <a href="https://www.youtube.com/channel/UCS9oAlDpRkq14yxmpmw2jpA/join">joining</a>
          my YouTube channel to support this project!
        </p>

        <div class="step-images">
          <figure>
            <a href="https://github.com/JEOresearch/EyeTracker/tree/main/3DTracker" target="_blank" rel="noopener">
              <img src="images/frontcamera/camerabutton.jpg" alt="Input source window with Start Camera button" loading="lazy" />
            </a>
            <figcaption><a href="https://github.com/JEOresearch/EyeTracker/tree/main/3DTracker">Fig. 7. Input-source window with the camera dropdown and Start Camera button.</a></figcaption>
          </figure>
        </div>
      </article>

      <article class="tutorial-step">
        <h3>Step 5: Position the eye in the camera view</h3>

        <p>
          After clicking <code>Start Camera</code>, adjust the camera until the
          eye is clearly visible in the OpenCV window (Fig. 8). The script crops
          the frame, finds the darkest pupil region, and thresholds the image at
          several levels (Fig. 9). It then fits an ellipse to the pupil,
          estimates the eye center from recent pupil rays, and displays the
          tracking overlay (Fig. 10). In normal use, the tracker self-calibrates
          to the observed eye position as you move.
        </p>

        <div class="step-images">
          <figure>
            <a href="https://amzn.to/41x8p2W" target="_blank" rel="noopener" data-product="GC0308 Eye Tracking Camera" data-category="Camera">
              <img src="images/frontcamera/eyevisible.jpg" alt="Eye clearly visible in the OpenCV camera window" loading="lazy" />
            </a>
            <figcaption><a href="https://amzn.to/41x8p2W" data-product="GC0308 Eye Tracking Camera" data-category="Camera">Fig. 8. Eye centered and clearly visible in the OpenCV camera window.</a></figcaption>
          </figure>
          <figure>
            <a href="https://amzn.to/41x8p2W" target="_blank" rel="noopener" data-product="GC0308 Eye Tracking Camera" data-category="Camera">
              <img src="images/frontcamera/thresholding.jpg" alt="Pupil region isolated by thresholding" loading="lazy" />
            </a>
            <figcaption><a href="https://amzn.to/41x8p2W" data-product="GC0308 Eye Tracking Camera" data-category="Camera">Fig. 9. Dark pupil region isolated by thresholding.</a></figcaption>
          </figure>
          <figure>
            <a href="https://amzn.to/41x8p2W" target="_blank" rel="noopener" data-product="GC0308 Eye Tracking Camera" data-category="Camera">
              <img src="images/frontcamera/gazeray.jpg" alt="Pupil ellipse eye center and gaze line overlay" loading="lazy" />
            </a>
            <figcaption><a href="https://amzn.to/41x8p2W" data-product="GC0308 Eye Tracking Camera" data-category="Camera">Fig. 10. Pupil ellipse, estimated eye center, and gaze line overlay.</a></figcaption>
          </figure>
        </div>
      </article>

      <article class="tutorial-step">
        <h3>Step 6: Read the gaze output</h3>

        <p>
          The OpenCV output window displays the current 3D gaze origin and gaze
          direction near the bottom of the frame. The script also writes these
          six values to <code>gaze_vector.txt</code> on each frame: the three
          origin coordinates followed by the three normalized gaze direction
          values. Use <code>q</code> to quit, the spacebar to pause, <code>e</code>
          to toggle ellipse capture, and <code>c</code> to clear captured
          ellipses.
        </p>

        <pre><code>x_origin,y_origin,z_origin,x_direction,y_direction,z_direction</code></pre>

      </article>

      <article class="tutorial-step">
        <h3>Step 7: Use the vector in your own project</h3>

        <p>
          Once the tracker is running, you can read <code>gaze_vector.txt</code>
          from another program or modify the Python script to send the gaze
          vector directly to your application. This first setup gives you a raw
          3D gaze direction. From there, you can calibrate it to a monitor, VR
          headset, or another display surface for interaction experiments.
        </p>

        <p>
          For the VR installation process, see this
          <a href="https://youtu.be/l95eJe8MV5g">video</a> or this
          <a href="vrtracker.html">tutorial</a>.
        </p>

      </article>

      <article class="tutorial-step">
        <h3>Step 8: Use AI to build your own gaze application</h3>

        <p>
          You can use an AI coding assistant or large language model to turn
          this tracker into your own application. Start by giving the model the
          Python script, explain what the gaze vector represents, and describe
          the interaction you want to build. Ask for small changes first, test
          each version, and paste any error messages or unexpected behavior
          back into the chat so the model can help you debug.
        </p>

        <p>
          These sample prompts can help you get started:
        </p>

        <pre><code>I have a Python eye tracking script that writes gaze data to gaze_vector.txt as:
x_origin,y_origin,z_origin,x_direction,y_direction,z_direction

Explain how this data could be used to control a simple application.</code></pre>

        <pre><code>Modify this Python eye tracking project so another Python program can read
gaze_vector.txt and move a cursor or marker on screen based on the gaze direction.
Keep the first version simple and explain where I should paste each code change.</code></pre>

        <pre><code>Create a small demo app that reads gaze_vector.txt in real time and displays
a dot moving on a 2D window. Include smoothing so the dot does not jitter too much.</code></pre>

        <pre><code>Help me add a calibration step to this gaze tracker. I want the user to look at
points on the screen, save the measured gaze vectors, and map future gaze vectors
to screen coordinates.</code></pre>

        <pre><code>I want to use this eye tracker for a custom interaction project. Ask me the
minimum questions needed, then propose a simple first prototype using the existing
gaze_vector.txt output.</code></pre>

        <pre><code>Here is the error message I get when running the eye tracking code. Explain what
is probably wrong and give me the smallest code or setup change to try first.</code></pre>

      </article>
    </section>
  </main>

  <footer class="footer">
    <p>JEOresearch LLC</p>
    <p>As an Amazon Associate, I earn from qualifying purchases at no extra cost to you.</p>
    <p>This website provides educational information about eye tracking and related technologies. Any hardware modifications, software, or procedures described are performed at the user's own risk. Always follow applicable manufacturer instructions and safety guidelines. All open-source software is released under the MIT License.</p>
  </footer>
</body>
</html>