-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathvrtracker.html
More file actions
585 lines (525 loc) · 28 KB
/
Copy pathvrtracker.html
File metadata and controls
585 lines (525 loc) · 28 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<link rel="icon" href="images/favicon.ico" type="image/x-icon" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<!-- Google tag (gtag.js) -->
<script async src="https://www.googletagmanager.com/gtag/js?id=G-WZ7XJMTZF5"></script>
<script>
window.dataLayer = window.dataLayer || [];
function gtag(){dataLayer.push(arguments);}
gtag('js', new Date());
gtag('config', 'G-WZ7XJMTZF5');
</script>
<script src="analytics.js" defer></script>
<title>DIY Eye Tracking for VR | JEO Research</title>
<meta
name="description"
content="VR headset eye tracking tutorial from JEO Research, including resources for eye-tracked virtual reality and gaze-based interaction projects."
/>
<link rel="canonical" href="https://www.jeoresearch.com/vrtracker.html" />
<link rel="stylesheet" href="style.css" />
<script type="application/ld+json">
{
"@context": "https://schema.org",
"@graph": [
{
"@type": "WebPage",
"@id": "https://www.jeoresearch.com/vrtracker.html#webpage",
"url": "https://www.jeoresearch.com/vrtracker.html",
"name": "DIY Eye Tracking for VR",
"isPartOf": {
"@id": "https://www.jeoresearch.com/#website"
},
"author": {
"@id": "https://www.jeoresearch.com/#person"
},
"publisher": {
"@id": "https://www.jeoresearch.com/#organization"
},
"breadcrumb": {
"@id": "https://www.jeoresearch.com/vrtracker.html#breadcrumb"
},
"mainEntity": {
"@id": "https://www.jeoresearch.com/vrtracker.html#article"
}
},
{
"@type": "TechArticle",
"@id": "https://www.jeoresearch.com/vrtracker.html#article",
"headline": "Eye Tracking with a VR Headset",
"author": {
"@id": "https://www.jeoresearch.com/#person"
},
"publisher": {
"@id": "https://www.jeoresearch.com/#organization"
},
"mainEntityOfPage": {
"@id": "https://www.jeoresearch.com/vrtracker.html#webpage"
},
"about": {
"@id": "https://www.jeoresearch.com/vrtracker.html#software"
}
},
{
"@type": "SoftwareSourceCode",
"@id": "https://www.jeoresearch.com/vrtracker.html#software",
"name": "VR Eye Tracker",
"codeRepository": "https://github.com/JEOresearch/EyeTracker/tree/main/VREyeTracker",
"programmingLanguage": [
"Python",
"C#"
],
"author": {
"@id": "https://www.jeoresearch.com/#person"
},
"publisher": {
"@id": "https://www.jeoresearch.com/#organization"
}
},
{
"@type": "BreadcrumbList",
"@id": "https://www.jeoresearch.com/vrtracker.html#breadcrumb",
"itemListElement": [
{
"@type": "ListItem",
"position": 1,
"name": "Home",
"item": "https://www.jeoresearch.com/"
},
{
"@type": "ListItem",
"position": 2,
"name": "VR Eye Tracking",
"item": "https://www.jeoresearch.com/vrtracker.html"
}
]
}
]
}
</script>
</head>
<body>
<header class="site-header">
<div class="header-inner">
<a class="brand" href="index.html"><span class="brand-jeo">JEO</span><span class="brand-research">research</span></a>
<nav class="nav">
<div class="nav-dropdown">
<button class="nav-trigger" type="button">Home▼</button>
<div class="nav-menu">
<a href="index.html">Home Page</a>
<a href="aboutme.html">About Me</a>
<a href="news.html">Appearances</a>
</div>
</div>
<div class="nav-dropdown">
<button class="nav-trigger" type="button">EyeTracking▼</button>
<div class="nav-menu">
<a href="eyecameras.html">Camera Comparison</a>
<a href="eyetracking.html">DIY Eye Tracker (Glasses)</a>
<a href="vrtracker.html">DIY Eye Tracker (VR)</a>
<a href="headgaze.html">DIY Head Tracker (Webcam)</a>
<a href="irsafety.html">Infrared Safety</a>
</div>
</div>
<div class="nav-dropdown">
<button class="nav-trigger" type="button">ARVRLab▼</button>
<div class="nav-menu">
<a href="ccslab.html">Lab Page</a>
<a href="ccslab.html#apply">Apply</a>
<a href="ccslab/team.html">Members</a>
<a href="students/timelines.html">Timelines</a>
<a href="ccslab/xr.html">XR Research</a>
<a href="ccslab/intelligent.html">Intelligent Agents</a>
</div>
</div>
<a href="research.html">Publications</a>
<a href="https://github.com/JEOresearch">GitHub</a>
</nav>
</div>
</header>
<main class="page">
<section class="hero tutorial-hero">
<div class="hero-text">
<h1>Eye Tracking with a VR Headset</h1>
<p class="eyebrow">By <a href="aboutme.html">Jason Orlosky, PhD</a></p>
<p class="intro">
This tutorial shows how to add low-cost eye tracking to a VR or AR
headset with a small infrared camera. The tracker writes a gaze vector
from Python, then a Unity scene calibrates that vector into a 3D gaze
ray for analysis, interaction, communication, or custom VR projects.
</p>
<div class="profile-links" aria-label="Tutorial links">
<a href="https://youtu.be/l95eJe8MV5g">Video Summary</a>
<span>|</span>
<a href="https://github.com/JEOresearch/EyeTracker/blob/main/3DTracker/Orlosky3DEyeTracker.py">Python Tracker Code</a>
<span>|</span>
<a href="https://github.com/JEOresearch/EyeTracker/tree/main/VREyeTracker">VR Calibration Script for Unity</a>
</div>
</div>
<div class="hero-image">
<a href="https://amzn.to/41x8p2W" target="_blank" rel="noopener" data-product="GC0308 Eye Tracking Camera" data-category="Camera">
<img src="images/vrgaze.jpg" alt="VR headset eye tracking tutorial" />
</a>
</div>
<p class="channel-support">
To help support this software and other open-source projects, please consider
<a href="https://www.youtube.com/channel/UCS9oAlDpRkq14yxmpmw2jpA?sub_confirmation=1">subscribing</a>
to or <a href="https://www.youtube.com/@jeoresearch/join">joining</a> my YouTube channel.
</p>
</section>
<section class="section tutorial-steps">
<h2>Building the VR Tracker</h2>
<article class="tutorial-step">
<h3>Step 1: Gather the parts</h3>
<p>
You need a VR or AR headset, a small
<a href="https://amzn.to/41x8p2W" data-product="GC0308 Eye Tracking Camera" data-category="Camera">GC0308 eye tracking camera</a> (Fig. 1),
<a href="https://amzn.to/43SznVl" data-product="USB Extension Cables" data-category="Cable">USB extension cables</a> if you
need more cable length, and
<a href="https://amzn.to/4pi6wBn" data-product="Soft Wire Cabling" data-category="Mount">soft wire cabling</a> or another
flexible mounting material. This build uses an
<a href="https://amzn.to/49OOrW0" data-product="HTC Vive Pro Headset" data-category="Headset">HTC Vive Pro headset</a> (Fig. 2), but the
same idea can be adapted to other headsets. An
<a href="https://amzn.to/4foKDOU" data-product="HTC Vive Kit" data-category="Headset">older Vive kit</a> (Fig. 3) can also work if
you already have compatible trackers or controllers.
A <a href="https://amzn.to/4vp6pqd" data-product="Quest 3 Headset" data-category="Headset">Quest 3 headset</a> (Fig. 4)
is another example of a headset that can be adapted.
Review <a href="irsafety.html">IR safety</a> and evaluate the camera's
infrared output before mounting it close to the eye.
</p>
<p>
The important requirement is that the camera can sit near one eye
without blocking the display or being fully enclosed. The GC0308 can
get warm during use, so leave it exposed enough for heat to escape.
</p>
<p>
<strong>For Stereo Tracking Setup:</strong> Use two eye-tracking
cameras, one for each eye, and provide a separate USB connection for
each camera. Both cameras should meet the same focus, heat, and
infrared-safety requirements described above. If the computer does
not provide enough ports near the headset, plan for two extension
cables or a suitable powered USB hub.
</p>
<div class="step-images">
<figure>
<a href="https://amzn.to/41x8p2W" target="_blank" rel="noopener" data-product="GC0308 Eye Tracking Camera" data-category="Camera">
<img src="images/frontcamera/gc0308.jpg" alt="GC0308 eye tracking camera" loading="lazy" />
</a>
<figcaption><a href="https://amzn.to/41x8p2W" data-product="GC0308 Eye Tracking Camera" data-category="Camera">Fig. 1. GC0308 IR camera</a></figcaption>
</figure>
<figure>
<a href="https://amzn.to/49OOrW0" target="_blank" rel="noopener" data-product="HTC Vive Pro Headset" data-category="Headset">
<img src="images/vrsetup/vivepro.jpg" alt="HTC Vive Pro headset for VR eye tracking" loading="lazy" />
</a>
<figcaption><a href="https://amzn.to/49OOrW0" data-product="HTC Vive Pro Headset" data-category="Headset">Fig. 2. HTC Vive Pro headset</a></figcaption>
</figure>
<figure>
<a href="https://amzn.to/4foKDOU" target="_blank" rel="noopener" data-product="HTC Vive Kit" data-category="Headset">
<img src="images/vrsetup/vive.jpg" alt="Older HTC Vive kit for VR eye tracking" loading="lazy" />
</a>
<figcaption><a href="https://amzn.to/4foKDOU" data-product="HTC Vive Kit" data-category="Headset">Fig. 3. Older Vive headset kit</a></figcaption>
</figure>
<figure>
<a href="https://amzn.to/4vp6pqd" target="_blank" rel="noopener" data-product="Quest 3 Headset" data-category="Headset">
<img src="images/quest3.jpg" alt="Quest 3 headset for VR eye tracking" loading="lazy" style="height: 200px;" />
</a>
<figcaption><a href="https://amzn.to/4vp6pqd" data-product="Quest 3 Headset" data-category="Headset">Fig. 4. Quest 3 headset</a></figcaption>
</figure>
</div>
</article>
<article class="tutorial-step">
<h3>Step 2: Set the camera focus</h3>
<p>
Before mounting the <a href="https://amzn.to/41x8p2W" data-product="GC0308 Eye Tracking Camera" data-category="Camera">camera</a>,
slightly unscrew the camera lens (Fig. 5) so the
focal distance is roughly 3 centimeters. This puts the eye in focus
when the camera is mounted close to the face inside the headset.
</p>
<p>
Connect the <a href="https://amzn.to/41x8p2W" data-product="GC0308 Eye Tracking Camera" data-category="Camera">camera</a>
to your computer and check the stream in any basic
camera viewer. If the image is blurry at eye distance, keep making
small lens adjustments until the pupil and eyelids are clearly visible
(Fig. 6).
</p>
<p>
<strong>For Stereo Tracking Setup:</strong> Adjust and test each
camera independently at the distance it will sit from its
corresponding eye. Match the sharpness of the two views as closely as
possible, and confirm that both pupils and eyelids remain visible
before installing the cameras in the headset.
</p>
<div class="step-images">
<figure>
<a href="https://amzn.to/41x8p2W" target="_blank" rel="noopener" data-product="GC0308 Eye Tracking Camera" data-category="Camera">
<img src="images/vrsetup/camerafocus.jpg" alt="Adjusting the camera lens focus for VR eye tracking" loading="lazy" />
</a>
<figcaption><a href="https://amzn.to/41x8p2W" data-product="GC0308 Eye Tracking Camera" data-category="Camera">Fig. 5. Camera lens adjusted for eye distance</a></figcaption>
</figure>
<figure>
<a href="https://amzn.to/41x8p2W" target="_blank" rel="noopener" data-product="GC0308 Eye Tracking Camera" data-category="Camera">
<img src="images/frontcamera/eyevisible.jpg" alt="Eye clearly visible in the camera view" loading="lazy" />
</a>
<figcaption><a href="https://amzn.to/41x8p2W" data-product="GC0308 Eye Tracking Camera" data-category="Camera">Fig. 6. Eye clearly visible in the camera view</a></figcaption>
</figure>
</div>
</article>
<article class="tutorial-step">
<h3>Step 3: Mount the camera inside the headset</h3>
<p>
Remove the portion of the display padding on the side where you want
to place the camera (Fig. 7). Use <a href="https://amzn.to/4pi6wBn" data-product="Soft Wire Cabling" data-category="Mount">soft wire</a>
to hold the <a href="https://amzn.to/41x8p2W" data-product="GC0308 Eye Tracking Camera" data-category="Camera">camera</a>
near the lens area,
then bend the wire until the camera has a clear view of your eye.
The exact position and orientation will vary by headset, so treat the
first mount as adjustable rather than permanent.
</p>
<p>
To verify the view while wearing the headset, open the desktop view
in SteamVR and display the camera feed with a camera viewer (Fig. 8). Adjust
the wire until the full eye is visible and the camera does not block
the headset display.
</p>
<p>
<strong>For Stereo Tracking Setup:</strong> Remove or reposition
padding on both sides and mount one camera near each headset lens.
Route and secure both cables so they do not pull the cameras out of
alignment. Verify that the left camera has an unobstructed view of the
left eye and the right camera has an unobstructed view of the right
eye, while neither camera blocks the display.
</p>
<div class="step-images">
<figure>
<a href="https://amzn.to/4foKDOU" target="_blank" rel="noopener" data-product="HTC Vive Kit" data-category="Headset">
<img src="images/vrsetup/vrcamerainsert.jpg" alt="Camera inserted inside a VR headset after removing padding" loading="lazy" />
</a>
<figcaption><a href="https://amzn.to/4foKDOU" data-product="HTC Vive Kit" data-category="Headset">Fig. 7. Padding removed and camera inserted</a></figcaption>
</figure>
<figure>
<a href="https://amzn.to/4foKDOU" target="_blank" rel="noopener" data-product="HTC Vive Kit" data-category="Headset">
<img src="images/vrsetup/vrcamera.jpg" alt="Verifying the camera angle inside the VR headset" loading="lazy" />
</a>
<figcaption><a href="https://amzn.to/4foKDOU" data-product="HTC Vive Kit" data-category="Headset">Fig. 8. Camera angle checked inside the headset</a></figcaption>
</figure>
</div>
</article>
<article class="tutorial-step">
<h3>Step 4: Download the tracker code</h3>
<p>
Download the <a href="https://github.com/JEOresearch/EyeTracker/blob/main/3DTracker/Orlosky3DEyeTracker.py">Python eye tracking script</a>
and the <a href="https://github.com/JEOresearch/EyeTracker/tree/main/VREyeTracker">Unity VR calibration
project</a> from GitHub. The Python script detects the pupil and writes a
gaze vector to a text file. The Unity script reads that file, shows
the uncalibrated gaze point, and guides the headset calibration.
</p>
<pre><code>git clone https://github.com/JEOresearch/EyeTracker.git</code></pre>
<p>
The main Python script is
<a href="https://github.com/JEOresearch/EyeTracker/blob/main/3DTracker/Orlosky3DEyeTracker.py"><code>3DTracker/Orlosky3DEyeTracker.py</code></a>.
The Unity files are in
<a href="https://github.com/JEOresearch/EyeTracker/tree/main/VREyeTracker"><code>VREyeTracker</code></a>.
</p>
<p>
<strong>For Stereo Tracking Setup:</strong> Use
<code>Orlosky3DEyeTrackerStereo.py</code> instead of the single-camera
Python tracker and use <code>EyeTrackerStereo.cs</code> for the Unity
component. Keep the two stereo files together with the rest of the
downloaded tracker project so their supporting modules and
dependencies remain available.
</p>
<p>
If you found this software/tutorial useful, please consider
<a href="https://www.youtube.com/channel/UCS9oAlDpRkq14yxmpmw2jpA/?sub_confirmation=1">subscribing</a> or
<a href="https://www.youtube.com/channel/UCS9oAlDpRkq14yxmpmw2jpA/join">joining</a>
my YouTube channel to support this project!
</p>
</article>
<article class="tutorial-step">
<h3>Step 5: Prepare the Unity scene</h3>
<p>
In Unity, import the VR rig or SDK for your headset. For a Vive Pro,
the SteamVR assets can be used. Add the downloaded
<code>EyeTracker.cs</code> script to the headset object in your scene
(Fig. 9).
This should be the object that represents the headset transform,
because the gaze sphere is positioned relative to the headset in 3D
space.
</p>
<p>
Set the <code>GazeFilePath</code> value in the Unity script to match
the write path used by the Python tracker. The example script reads
<code>C:\users\jason\gaze_vector.txt</code>, but you should change
that path if your Python script writes <code>gaze_vector.txt</code>
somewhere else. The Unity script expects six values: the three origin
coordinates followed by the three direction coordinates.
The initial uncalibrated gaze marker is shown in Fig. 10.
</p>
<p>
<strong>For Stereo Tracking Setup:</strong> Add
the <code>EyeTrackerStereo.cs</code> script to the headset
object instead of <code>EyeTracker.cs</code>. Then change its
<code>GazeFilePath</code> constant to the exact
<code>gaze_vector.txt</code> produced by the stereo Python script.
This version expects 12 comma-separated values: origin and direction
for the left eye, followed by origin and direction for the right eye.
The script creates separate cyan and magenta eye markers and, after
calibration, a white combined gaze marker. The default eye-origin
offset is 0.032 meters to either side of the headset center; adjust
<code>EyeOriginHorizontalOffset</code> in the script if your headset
or application requires a different value.
</p>
<div class="step-images">
<figure>
<a href="https://github.com/JEOresearch/EyeTracker/tree/main/VREyeTracker" target="_blank" rel="noopener">
<img src="images/vrsetup/unityscene.jpg" alt="Unity scene set up for VR eye tracking calibration" loading="lazy" />
</a>
<figcaption><a href="https://github.com/JEOresearch/EyeTracker/tree/main/VREyeTracker">Fig. 9. Unity scene with the headset tracker script</a></figcaption>
</figure>
<figure>
<a href="https://github.com/JEOresearch/EyeTracker/tree/main/VREyeTracker" target="_blank" rel="noopener">
<img src="images/vrsetup/headsetview.jpg" alt="Uncalibrated headset view with gaze marker" loading="lazy" />
</a>
<figcaption><a href="https://github.com/JEOresearch/EyeTracker/tree/main/VREyeTracker">Fig. 10. Uncalibrated gaze marker in the headset view</a></figcaption>
</figure>
</div>
</article>
<article class="tutorial-step">
<h3>Step 6: Run the Python tracker first</h3>
<p>
Start the Python script before pressing play in Unity. Select the
eye-tracking camera, then let the tracker self-center on your eye.
If the camera image is mirrored or upside down for your headset
mounting position, use the <code>CVflip</code> function in the Python
script to flip the camera image as needed.
</p>
<pre><code>python Orlosky3DEyeTracker.py</code></pre>
<p>
Once the pupil is being tracked and <code>gaze_vector.txt</code> is
updating, return to Unity and press play. In the Unity scene view,
you should see a moving gray sphere that represents the uncalibrated
gaze estimate.
</p>
<p>
<strong>For Stereo Tracking Setup:</strong> Run
<code>python Orlosky3DEyeTrackerStereo.py</code>. In the input window,
choose the correct camera index under <code>Left Eye Camera</code>
and <code>Right Eye Camera</code>, then start both cameras. Do not
assign the same physical camera to both eyes. Use each eye's
<code>flip image</code> checkbox if its mounted view is inverted.
Wait until both eye models are stable and confirm that
<code>gaze_vector.txt</code> is updating with 12 values before
starting Unity.
</p>
</article>
<article class="tutorial-step">
<h3>Step 7: Calibrate the gaze ray in Unity</h3>
<p>
With the Unity game window selected, press <code>c</code> to begin
calibration. The script shows three red calibration spheres (Fig. 11): center,
up, and down. Look directly at the center sphere and press
<code>c</code>, then look at the up sphere and press <code>c</code>,
then look at the down sphere and press <code>c</code>. The default
up and down targets are 10 degrees above and below center.
</p>
<p>
After calibration, the gray sphere should track with your eye. The
script uses the center target to rotate the raw Python direction
forward, then uses the up and down targets to scale vertical gaze
motion. Press <code>b</code> if you want to leave a frozen copy of the
current gaze sphere in the scene for debugging.
</p>
<p>
<strong>For Stereo Tracking Setup:</strong> Keep both eyes open and
look at each target with both eyes. Press <code>c</code> once to show
the first target, then follow the five-point sequence shown by
<code>EyeTrackerStereo.cs</code>: up, right, down, left, and center.
Press <code>c</code> while looking directly at each target to record
both eyes at the same time. When all five samples are accepted, the
cyan and magenta eye markers become translucent and the white
combined gaze marker appears. Press <code>g</code> to show or hide
the optional 5-by-5 accuracy grid; with the grid visible, press
<code>t</code> at each highlighted target to record an accuracy test
(Fig. 12).
</p>
<div class="step-images">
<figure>
<a href="https://github.com/JEOresearch/EyeTracker/tree/main/VREyeTracker" target="_blank" rel="noopener">
<img src="images/vrsetup/calibrationsphere.jpg" alt="Red calibration sphere shown in the VR headset" loading="lazy" />
</a>
<figcaption><a href="https://github.com/JEOresearch/EyeTracker/tree/main/VREyeTracker">Fig. 11. Red calibration sphere<br />(headset view)</a></figcaption>
</figure>
<figure>
<img src="images/vrsetup/accuracy.jpg" alt="Accuracy testing grid for stereo eye tracker calibration" loading="lazy" />
<figcaption><a href="https://github.com/JEOresearch/EyeTracker/blob/main/VREyeTracker/EyeTrackerStereo.cs">Fig. 12. Accuracy testing grid</a></figcaption>
</figure>
</div>
</article>
<article class="tutorial-step">
<h3>Step 8: Use the gaze ray in your own project</h3>
<p>
The calibrated gaze ray (Fig. 13) can be used for gaze analysis, VR interface
selection, attention-aware games, assistive communication, or custom
interaction experiments. In Unity, use the headset origin and the
direction from the headset to the gray gaze sphere as a ray, then
test what it intersects in your scene.
</p>
<pre><code>Vector3 gazeDirection = (gazeSphere.transform.position - transform.position).normalized;
Ray gazeRay = new Ray(transform.position, gazeDirection);</code></pre>
<p>
For a first prototype, start by drawing the ray and logging the object
it hits. After the ray is stable, add smoothing, dwell selection, or
application-specific behavior.
</p>
<p>
<strong>For Stereo Tracking Setup:</strong> Use the white combined
gaze sphere as the binocular target point after calibration. Build
the application ray from the headset transform toward that sphere.
Keep the individual cyan and magenta eye markers available when you
need to inspect disagreement between the eyes or develop
eye-specific interactions.
</p>
<div class="step-images">
<figure>
<a href="https://github.com/JEOresearch/EyeTracker/tree/main/VREyeTracker" target="_blank" rel="noopener">
<img src="images/vrsetup/vrgazevector.jpg" alt="Final calibrated VR gaze ray in Unity" loading="lazy" />
</a>
<figcaption><a href="https://github.com/JEOresearch/EyeTracker/tree/main/VREyeTracker">Fig. 13. Final calibrated gaze ray in Unity (green ray)</a></figcaption>
</figure>
</div>
</article>
<article class="tutorial-step">
<h3>Step 9: Use AI to build a VR gaze application</h3>
<p>
You can use an AI coding assistant or large language model to turn
the calibrated gaze ray into a custom VR interaction. Give the model
the Python tracker, the Unity script, and a short description of what
the gaze ray represents. Ask for small testable changes, then paste
any Unity console errors or unexpected behavior back into the chat.
</p>
<p>
These sample prompts can help you get started:
</p>
<pre><code>I have a Unity VR eye tracker that reads gaze_vector.txt from a Python script
and calibrates it into a 3D gaze ray from the headset. Explain how I can use
this ray to select objects in a VR scene.</code></pre>
<pre><code>Modify my Unity script so the gaze ray performs a Physics.Raycast every frame
and highlights the object the user is looking at. Keep the first version simple.</code></pre>
<pre><code>Add dwell selection to this VR gaze tracker. If the user looks at the same
interactable object for 1 second, trigger that object's selection event.</code></pre>
<pre><code>Help me smooth this VR gaze ray without adding too much latency. Explain where
to add the smoothing code in the Unity script.</code></pre>
<pre><code>Here is the Unity console error I get when reading gaze_vector.txt. Explain what
is probably wrong and give me the smallest setup or code change to try first.</code></pre>
</article>
</section>
</main>
<footer class="footer">
<p>JEOresearch LLC</p>
<p>As an Amazon Associate, I earn from qualifying purchases at no extra cost to you.</p>
<p>This website provides educational information about eye tracking and related technologies. Any hardware modifications, software, or procedures described are performed at the user's own risk. Always follow applicable manufacturer instructions and safety guidelines. All open-source software is released under the MIT License.</p>
</footer>
</body>
</html>