Skip to content

About

This repository replicates the NemoStation/Marlin-2B paradigm natively on Apple Silicon using the MLX framework, achieving ultra-fast temporal video captioning and sub-second visual grounding entirely within Apple's Unified Memory space.

Topics

Resources

Stars

4 stars

Watchers

0 watching

Forks

Latest commit

 

History

2 Commits

Folders and files

Repository files navigation

🎬 Sovereign Video-VLM: Replicating Marlin-2B on Apple Silicon (MLX)

OS Framework Model License

A high-throughput, memory-stable local ecosystem to fine-tune, align, and deploy specialized Small Vision-Language Models (SVLMs) on macOS.

This repository replicates the NemoStation/Marlin-2B paradigm natively on Apple Silicon using the MLX framework, achieving ultra-fast temporal video captioning and sub-second visual grounding entirely within Apple's Unified Memory space.


⚡ Architectural Optimizations

Beyond standard VLM inference, this codebase introduces advanced optimizations specifically engineered for Apple Silicon GPUs:

  • Native-to-Native GPU Ingestion Pipeline: Ingests macOS CVPixelBuffer textures (via AVAssetReader or camera streams) directly. Executes channel swapping (BGRA-to-RGB) and parallelized bilinear resizing entirely on the Metal GPU (mlx_resize_bilinear). Bypasses all CPU-bound operations (PIL.Image, cv2.resize) and host-to-device memory copies.
  • Zero-Cost MLX GPU Vision-Caching: Monkey-patches the Qwen2.5-VL Vision Tower (vision_tower.__call__) to cache visual embeddings in VRAM. This completely eliminates redundant Vision Tower forward passes on repeated temporal queries (such as sequential search or hierarchical grounding zooms), cutting successive latency by >85%!
  • Adaptive KV Cache Quantization: Configures dynamic memory precisions—utilizing 3.5-bit TurboQuant for the fast preset and 8-bit cache for balanced and detail presets—maximizing VRAM bandwidth on Apple Silicon.
  • Hierarchical Spatio-Temporal Grounding: Replicates the core visual selection concepts of the DyToK paper. Runs a two-pass temporal zoom inside marlin.find()—first a coarse uniform scan with 4 frames, followed by a high-precision 8-frame scan centered on the candidate sub-window.

📁 Repository Architecture

qwen-fine-tune/
├── .gitignore                      # Standard exclude patterns (caches, weights, video files)
├── README.md                       # Comprehensive repository documentation
├── Fine-tuning VLM on Mac with MLX.md # Deep research guide on SVLM architecture and training
├── dataset_info.json               # ShareGPT dataset registry for SFT and GRPO
├── train_vlm.py                    # LoRA Supervised Fine-Tuning (SFT) pipeline
├── grpo_align.py                   # Rule-based post-training GRPO alignment pipeline
├── inference.py                    # Public developer API wrapper and performance presets
├── scripts/                        # Auxiliary scripts
│   └── profile_pipeline.py         # Memory and compute profiler
├── tests/                          # Automated verification suites
│   ├── test_cvpixelbuffer_ingestion.py # Direct macOS CVPixelBuffer texture test
│   ├── test_gpu_pipeline.py        # Metal GPU array evaluation validation
│   └── test_mlx_resize.py          # GPU bilinear resampler validator
└── scratch/                        # Temporary developer scratchpads

🚀 Performance Benchmarks

Measured on an Apple Silicon Mac Studio running MLX 0.16.x with a 16-frame portrait video clip (sample_clip.mp4):

Speed Preset Frame Resolution Caption Latency Grounding Latency (marlin.find) Peak VRAM Footprint DyToK Keyframe Optimization
Fast 168x168 (~144 tok/f) 2.41 seconds 5.66 seconds (coarse+fine) 8.18 GB Selected 16 frames from 48 visually active candidates
Balanced 224x224 (~256 tok/f) 8.12 seconds 3.64 seconds (with cache) 8.18 GB Selected 16 frames from 48 visually active candidates
Detail 448x448 (~1024 tok/f) 27.33 seconds 19.23 seconds (high-res zoom) 8.18 GB Selected 16 frames from 48 visually active candidates

💻 Developer API Guide

1. Ingesting macOS CVPixelBuffers Natively (Zero-Copy)

Use marlin.process_textures() to ingest a stream of macOS CVPixelBuffer handles (from AVPlayer, camera sessions, or screen capture) directly into the Metal GPU pipeline, bypassing CPU decoding:

import objc
from Quartz import CoreVideo
from inference import SovereignVideoVLM

# 1. Initialize SovereignVideoVLM (runs natively on Metal)
marlin = SovereignVideoVLM("mlx-community/Qwen2.5-VL-3B-Instruct-4bit")

# 2. Ingest your array of CVPixelBuffer textures
prompt = "Analyze the video and return a structured JSON mapping events with exact timestamps."
raw_json = marlin.process_textures(pixel_buffers, prompt, speed_preset="fast")

print(raw_json)

2. Temporal Captioning and Semantic Grounding

For high-level file-based video processing, use marlin.caption() and marlin.find():

from inference import SovereignVideoVLM

marlin = SovereignVideoVLM("mlx-community/Qwen2.5-VL-3B-Instruct-4bit")

# A. Generate structured JSON caption
caption_data = marlin.caption("clip.mp4", speed_preset="balanced")
print(f"Scene: {caption_data['scene']}")
for event in caption_data["events"]:
    print(f"  [{event['span'][0]}s - {event['span'][1]}s]: {event['event']}")

# B. Locate precise chronological event bounds (Hierarchical Zoom Scan)
start, end = marlin.find("clip.mp4", "striking the soccer ball", speed_preset="fast")
print(f"🎯 Event occurred between {start}s and {end}s")

🏋️ Fine-Tuning & Alignment Orchestration

This codebase supports the complete lifecycle of model optimization locally on Apple Silicon:

Step 1: LoRA Supervised Fine-Tuning (SFT)

Configure SFT with 4-bit quantization and unfrozen multimodal projection layers (LoRA applied to self-attention and MLP layers):

  1. Format your video dataset according to the ShareGPT schema inside video_sft_dataset.json (registered via dataset_info.json):
    [
      {
        "conversations": [
          {
            "from": "user",
            "value": "<video>\nAnalyze the video and return a structured JSON mapping the scene description and event durations."
          },
          {
            "from": "assistant",
            "value": "{\"scene\": \"baking demonstration\", \"events\": [{\"event\": \"slicing fruit\", \"span\": [0.0, 5.2]}]}"
          }
        ],
        "videos": ["/absolute/path/to/clip.mp4"]
      }
    ]
  2. Trigger the training pipeline, supplying your system's unified memory size in GB (e.g., 32 for a 32GB Mac) to automatically scale batch parameters:
    python3 train_vlm.py 32

Step 2: Post-Training Rule-Based GRPO Alignment

To eliminate expensive teacher APIs, align your VLM using Group Relative Policy Optimization (GRPO). The alignment uses four rule-based reward functions to calculate relative advantages within completion groups ($G=4$):

  • JSON Validity Reward: +1.0 for valid JSON matching the exact schema.
  • Chronological Time Flow Reward: +1.0 if event timestamps strictly progress forward; -1.5 if time flows backward.
  • Non-Thinking Prefix Suppression: -1.0 if the model leaks chain-of-thought <think> tags.
  • Boundary Containment Reward: Penalizes outputs predicting timestamps extending beyond the video's actual duration.

Run the alignment pipeline locally:

python3 grpo_align.py

🛠️ Installation & Setup

  1. Clone the Repository:

    git clone https://codingstark-dev/qwen2.5-vl-apple-silicon.git
    cd qwen2.5-vl-apple-silicon
  2. Install Core Requirements: Ensure you have Python 3.10+ and install Apple Silicon Metal-optimized dependencies:

    pip install mlx mlx-vlm opencv-python transformers numpy pyobjc
  3. Verify the Texture Pipeline: Ensure Quartz/CoreVideo bindings work natively by running the automated CVPixelBuffer verification test:

    python3 tests/test_cvpixelbuffer_ingestion.py

About

This repository replicates the NemoStation/Marlin-2B paradigm natively on Apple Silicon using the MLX framework, achieving ultra-fast temporal video captioning and sub-second visual grounding entirely within Apple's Unified Memory space.

Topics

Resources

Stars

4 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages