Skip to content

Latest commit

 

History

History
192 lines (136 loc) · 6.58 KB

macos.md

File metadata and controls

192 lines (136 loc) · 6.58 KB

macOS Bring-Up

Current state

The ZED open capture stack is functional on macOS / Apple Silicon with the following capabilities:

  • AVFoundation video backend captures raw YUYV stereo frames from the connected ZED 2i
  • Sensor capture (IMU, magnetometer, barometer, temperature) works via hidapi, same as Linux
  • Timestamp-fallback video/sensor sync is wired, validated on hardware (median offset ~2.5 ms)
  • Calibration auto-loads from ~/zed/settings/SN<serial>.conf
  • Stereo rectification works via OpenCV remap
  • MLX disparity and depth run on Apple Silicon GPU (~8 ms compute at 720p)
  • Live preview pipeline uses shared-memory handoff from C++ capture to Python/MLX viewer (~14.5 FPS with cv2 display)
  • Point cloud export produces rectified colored PLY files
  • Live stereo tuner persists presets to ~/zed/settings/mlx_live_stereo.json
  • All upstream examples (video, sensors, control, rectify, sync, depth, tune-stereo) launch on macOS through Terminal-hosted wrappers

Port status

Surface Status Notes
Raw video capture Native parity AVFoundation, validated on ZED 2i
Sensor capture Native parity hidapi, same as Linux
Video/sensor sync macOS replacement Timestamp fallback, no hardware gate
Calibration Native parity Same file I/O path
Rectification Native parity OpenCV remap
Video example Native parity AVFoundation + OpenCV display
Multi-camera example macOS replacement Needs multi-device validation
Camera control example Capability-gated partial Mode toggle only (platform limitation)
Depth example macOS replacement Routes to MLX disparity
Tune-stereo example macOS replacement MLX live tuner with persistent config
Point cloud export Native parity MLX-computed PLY
Evaluation harness New capability MLX eval with JSON/Markdown summary
ROS2 wrapper Deferred Pending base stack stability

Prerequisites

  • macOS 13+ on Apple Silicon (M1 or later)
  • Python 3.12+
  • uv
  • Xcode command line tools
  • Homebrew: brew install hidapi opencv cmake

First-time setup

make sync
make configure-zed-open-capture
make build-zed-open-capture
brew install opencv
make build-examples

Camera permission

macOS requires explicit camera permission for the host process. Use the Terminal-hosted targets to trigger the normal macOS prompt:

make request-camera-access-terminal

Grant camera permission to Terminal in: System Settings > Privacy & Security > Camera

Inside automated environments (Codex, CI), tccd attributes camera requests to the host process, which may lack the camera entitlement. The *-terminal make targets work around this by running through com.apple.Terminal.

Camera discovery

make probe-cameras

Prints all AVFoundation cameras, unique IDs, and advertised formats.

Smoke tests

# Basic capture validation
make smoke-zed-terminal

# Dump a raw frame and decode to PNG previews
make dump-zed-frame-terminal
make inspect-frame

# Sensor validation
make smoke-sensors

# Video/sensor sync validation
make smoke-sync-terminal

# Capability report
make report-capabilities

MLX depth pipeline

# One-shot MLX smoke test on dumped frame
make mlx-stereo-smoke

# Benchmark (5 iterations)
make mlx-stereo-benchmark

# Full evaluation report (previews + JSON/Markdown summary)
make mlx-stereo-eval

# Point cloud export from dumped frame
make mlx-point-cloud

# Live MLX disparity viewer (shared-memory pipeline)
make live-mlx-disparity-terminal

# Live evaluation (captures fresh frame, then runs eval)
make live-mlx-eval-terminal

Upstream examples on macOS

All upstream examples launch through Terminal-hosted wrappers:

make run-video-example-terminal
make run-control-example-terminal
make run-rectify-example-terminal
make run-sync-example-terminal
make run-depth-example-terminal          # routes to MLX live depth on macOS
make run-tune-stereo-example-terminal    # routes to MLX live tuner on macOS
make run-multicam-video-example-terminal

Performance

Measured on ZED 2i at 1280x720 with default live preset (pyramid_factor=2, refinement_radius=0, window_size=5):

Metric Value
MLX compute per frame ~8 ms
Live loop (with cv2 display) ~69 ms (~14.5 FPS)
Offline eval, fast preset (pyr2, consistency=none) ~160 ms (~6.25 FPS)
Offline eval, full resolution (pyr1, consistency=none) ~1054 ms (~0.93 FPS)
Offline eval, strict (pyr1, consistency=left_right) ~1790 ms (~0.56 FPS)

The live display bottleneck is Python/OpenCV overhead, not MLX compute. For faster preview, set LIVE_PYRAMID_FACTOR=4.

Live tuning

The MLX live tuner replaces the upstream OpenCV SGBM tune-stereo tool:

make run-tune-stereo-example-terminal

Interactive trackbars control pyramid_factor, max_disparity, window_size, refinement_radius, consistency_mode, processing_scale, and display options. Keybindings:

  • s -- save current preset to ~/zed/settings/mlx_live_stereo.json
  • l -- load saved preset
  • r -- reset to defaults
  • p -- export point cloud
  • q / ESC -- quit

The depth demo (make run-depth-example-terminal) auto-loads the saved preset on startup.

Known limitations

Camera controls

Camera controls on macOS are limited to AVFoundation auto/locked mode toggling. The macOS kernel UVC driver blocks sideband USB control transfers -- there is no UVCIOC_CTRL_QUERY equivalent. This affects:

  • Manual exposure duration and ISO
  • Specific white balance gains or color temperature
  • Exposure compensation/bias
  • Numeric gain control
  • ROI-based metering
  • LED control
  • Vendor extension unit (XU) commands

The full XU transport code (IOKit-based discovery, vendor control protocol, I2C sensor register bridge) is implemented and preserved in src/uvc_controls_macos.mm for future use if Apple adds a UVC control query API.

Video/sensor sync

macOS uses a timestamp-based fallback instead of the Linux hardware sync path. The fallback is wired into enableSensorSync and validated through make smoke-sync-terminal. Typical offset: median ~2.5 ms, P95 ~98 ms.

OpenCV depth fallback

The CPU/OpenCV SGBM depth path works on macOS but is not the primary path. The UMat/T-API teardown is unstable on current macOS builds. Use the MLX targets for production depth work.

Multi-camera

The multi-camera example builds but has not been validated with multiple ZED cameras on macOS hardware.

ROS2 wrapper

zed-ros2-wrapper-mlx is intentionally untouched until the base capture stack stabilizes.