Skip to content

Use dma-buf copy for all pipelines #53

@FreddyFunk

Description

@FreddyFunk

Summary

Currently, all frame data flows through system memory using Arc<[u8]> byte slices. Frames are copied from GStreamer buffers to CPU memory via buffer.map_readable(). This introduces unnecessary memory copies and CPU overhead, especially for high-resolution or high-framerate video.

Current Implementation

  • Frame extraction: GStreamer buffers are mapped to CPU memory and wrapped in Arc<[u8]>
  • No dma-buf support: Zero dma-buf references exist in the codebase
  • Memory model: All frames pass through system memory regardless of source or destination

Key locations:

  • src/backends/camera/pipewire/pipeline.rs - Camera capture pipeline
  • src/backends/virtual_camera/pipeline.rs - Virtual camera output pipeline
  • src/backends/camera/types.rs - CameraFrame structure

Proposed Changes

  1. Enable dma-buf in GStreamer pipeline

    • Use GStreamer dma-buf elements for zero-copy buffer passing
    • Expose file descriptors from dma-buf buffers
  2. Update frame data structure

    • Replace Arc<[u8]> with dma-buf handles/file descriptors
    • Track dma-buf lifecycle (acquire/release)
    • Add proper synchronization between camera, GPU, and applications
  3. GPU direct access

    • Enable direct GPU→dma-buf mapping
    • Eliminate CPU readback in filter pipeline where possible
    • Keep data on GPU until final output

Benefits

  • Reduced memory bandwidth: No unnecessary copies between GPU and CPU
  • Lower latency: Direct buffer passing between components
  • Lower CPU usage: Less memory management overhead
  • Better efficiency: Especially important for 4K/high-framerate capture

Considerations

  • Requires proper dma-buf lifecycle management
  • Need to handle fallback for systems without dma-buf support
  • May require Vulkan/NVIDIA extensions for GPU memory access

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions