-
Notifications
You must be signed in to change notification settings - Fork 4
Open
Description
Summary
Currently, all frame data flows through system memory using Arc<[u8]> byte slices. Frames are copied from GStreamer buffers to CPU memory via buffer.map_readable(). This introduces unnecessary memory copies and CPU overhead, especially for high-resolution or high-framerate video.
Current Implementation
- Frame extraction: GStreamer buffers are mapped to CPU memory and wrapped in
Arc<[u8]> - No dma-buf support: Zero dma-buf references exist in the codebase
- Memory model: All frames pass through system memory regardless of source or destination
Key locations:
src/backends/camera/pipewire/pipeline.rs- Camera capture pipelinesrc/backends/virtual_camera/pipeline.rs- Virtual camera output pipelinesrc/backends/camera/types.rs- CameraFrame structure
Proposed Changes
-
Enable dma-buf in GStreamer pipeline
- Use GStreamer dma-buf elements for zero-copy buffer passing
- Expose file descriptors from dma-buf buffers
-
Update frame data structure
- Replace
Arc<[u8]>with dma-buf handles/file descriptors - Track dma-buf lifecycle (acquire/release)
- Add proper synchronization between camera, GPU, and applications
- Replace
-
GPU direct access
- Enable direct GPU→dma-buf mapping
- Eliminate CPU readback in filter pipeline where possible
- Keep data on GPU until final output
Benefits
- Reduced memory bandwidth: No unnecessary copies between GPU and CPU
- Lower latency: Direct buffer passing between components
- Lower CPU usage: Less memory management overhead
- Better efficiency: Especially important for 4K/high-framerate capture
Considerations
- Requires proper dma-buf lifecycle management
- Need to handle fallback for systems without dma-buf support
- May require Vulkan/NVIDIA extensions for GPU memory access
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels