The ZED open capture stack is functional on macOS / Apple Silicon with the following capabilities:
- AVFoundation video backend captures raw YUYV stereo frames from the connected ZED 2i
- Sensor capture (IMU, magnetometer, barometer, temperature) works via hidapi, same as Linux
- Timestamp-fallback video/sensor sync is wired, validated on hardware (median offset ~2.5 ms)
- Calibration auto-loads from
~/zed/settings/SN<serial>.conf - Stereo rectification works via OpenCV remap
- MLX disparity and depth run on Apple Silicon GPU (~8 ms compute at 720p)
- Live preview pipeline uses shared-memory handoff from C++ capture to Python/MLX viewer (~14.5 FPS with cv2 display)
- Point cloud export produces rectified colored PLY files
- Live stereo tuner persists presets to
~/zed/settings/mlx_live_stereo.json - All upstream examples (video, sensors, control, rectify, sync, depth, tune-stereo) launch on macOS through Terminal-hosted wrappers
| Surface | Status | Notes |
|---|---|---|
| Raw video capture | Native parity | AVFoundation, validated on ZED 2i |
| Sensor capture | Native parity | hidapi, same as Linux |
| Video/sensor sync | macOS replacement | Timestamp fallback, no hardware gate |
| Calibration | Native parity | Same file I/O path |
| Rectification | Native parity | OpenCV remap |
| Video example | Native parity | AVFoundation + OpenCV display |
| Multi-camera example | macOS replacement | Needs multi-device validation |
| Camera control example | Capability-gated partial | Mode toggle only (platform limitation) |
| Depth example | macOS replacement | Routes to MLX disparity |
| Tune-stereo example | macOS replacement | MLX live tuner with persistent config |
| Point cloud export | Native parity | MLX-computed PLY |
| Evaluation harness | New capability | MLX eval with JSON/Markdown summary |
| ROS2 wrapper | Deferred | Pending base stack stability |
- macOS 13+ on Apple Silicon (M1 or later)
- Python 3.12+
- uv
- Xcode command line tools
- Homebrew:
brew install hidapi opencv cmake
make sync
make configure-zed-open-capture
make build-zed-open-capture
brew install opencv
make build-examplesmacOS requires explicit camera permission for the host process. Use the Terminal-hosted targets to trigger the normal macOS prompt:
make request-camera-access-terminalGrant camera permission to Terminal in: System Settings > Privacy & Security > Camera
Inside automated environments (Codex, CI), tccd attributes camera requests to the host process, which may lack the camera entitlement. The *-terminal make targets work around this by running through com.apple.Terminal.
make probe-camerasPrints all AVFoundation cameras, unique IDs, and advertised formats.
# Basic capture validation
make smoke-zed-terminal
# Dump a raw frame and decode to PNG previews
make dump-zed-frame-terminal
make inspect-frame
# Sensor validation
make smoke-sensors
# Video/sensor sync validation
make smoke-sync-terminal
# Capability report
make report-capabilities# One-shot MLX smoke test on dumped frame
make mlx-stereo-smoke
# Benchmark (5 iterations)
make mlx-stereo-benchmark
# Full evaluation report (previews + JSON/Markdown summary)
make mlx-stereo-eval
# Point cloud export from dumped frame
make mlx-point-cloud
# Live MLX disparity viewer (shared-memory pipeline)
make live-mlx-disparity-terminal
# Live evaluation (captures fresh frame, then runs eval)
make live-mlx-eval-terminalAll upstream examples launch through Terminal-hosted wrappers:
make run-video-example-terminal
make run-control-example-terminal
make run-rectify-example-terminal
make run-sync-example-terminal
make run-depth-example-terminal # routes to MLX live depth on macOS
make run-tune-stereo-example-terminal # routes to MLX live tuner on macOS
make run-multicam-video-example-terminalMeasured on ZED 2i at 1280x720 with default live preset (pyramid_factor=2, refinement_radius=0, window_size=5):
| Metric | Value |
|---|---|
| MLX compute per frame | ~8 ms |
| Live loop (with cv2 display) | ~69 ms (~14.5 FPS) |
Offline eval, fast preset (pyr2, consistency=none) |
~160 ms (~6.25 FPS) |
Offline eval, full resolution (pyr1, consistency=none) |
~1054 ms (~0.93 FPS) |
Offline eval, strict (pyr1, consistency=left_right) |
~1790 ms (~0.56 FPS) |
The live display bottleneck is Python/OpenCV overhead, not MLX compute. For faster preview, set LIVE_PYRAMID_FACTOR=4.
The MLX live tuner replaces the upstream OpenCV SGBM tune-stereo tool:
make run-tune-stereo-example-terminalInteractive trackbars control pyramid_factor, max_disparity, window_size, refinement_radius, consistency_mode, processing_scale, and display options. Keybindings:
s-- save current preset to~/zed/settings/mlx_live_stereo.jsonl-- load saved presetr-- reset to defaultsp-- export point cloudq/ESC-- quit
The depth demo (make run-depth-example-terminal) auto-loads the saved preset on startup.
Camera controls on macOS are limited to AVFoundation auto/locked mode toggling. The macOS kernel UVC driver blocks sideband USB control transfers -- there is no UVCIOC_CTRL_QUERY equivalent. This affects:
- Manual exposure duration and ISO
- Specific white balance gains or color temperature
- Exposure compensation/bias
- Numeric gain control
- ROI-based metering
- LED control
- Vendor extension unit (XU) commands
The full XU transport code (IOKit-based discovery, vendor control protocol, I2C sensor register bridge) is implemented and preserved in src/uvc_controls_macos.mm for future use if Apple adds a UVC control query API.
macOS uses a timestamp-based fallback instead of the Linux hardware sync path. The fallback is wired into enableSensorSync and validated through make smoke-sync-terminal. Typical offset: median ~2.5 ms, P95 ~98 ms.
The CPU/OpenCV SGBM depth path works on macOS but is not the primary path. The UMat/T-API teardown is unstable on current macOS builds. Use the MLX targets for production depth work.
The multi-camera example builds but has not been validated with multiple ZED cameras on macOS hardware.
zed-ros2-wrapper-mlx is intentionally untouched until the base capture stack stabilizes.