A high-throughput, memory-stable local ecosystem to fine-tune, align, and deploy specialized Small Vision-Language Models (SVLMs) on macOS.
This repository replicates the NemoStation/Marlin-2B paradigm natively on Apple Silicon using the MLX framework, achieving ultra-fast temporal video captioning and sub-second visual grounding entirely within Apple's Unified Memory space.
Beyond standard VLM inference, this codebase introduces advanced optimizations specifically engineered for Apple Silicon GPUs:
- Native-to-Native GPU Ingestion Pipeline: Ingests macOS
CVPixelBuffertextures (viaAVAssetReaderor camera streams) directly. Executes channel swapping (BGRA-to-RGB) and parallelized bilinear resizing entirely on the Metal GPU (mlx_resize_bilinear). Bypasses all CPU-bound operations (PIL.Image,cv2.resize) and host-to-device memory copies. - Zero-Cost MLX GPU Vision-Caching: Monkey-patches the
Qwen2.5-VLVision Tower (vision_tower.__call__) to cache visual embeddings in VRAM. This completely eliminates redundant Vision Tower forward passes on repeated temporal queries (such as sequential search or hierarchical grounding zooms), cutting successive latency by >85%! - Adaptive KV Cache Quantization: Configures dynamic memory precisions—utilizing 3.5-bit TurboQuant for the
fastpreset and 8-bit cache forbalancedanddetailpresets—maximizing VRAM bandwidth on Apple Silicon. - Hierarchical Spatio-Temporal Grounding: Replicates the core visual selection concepts of the DyToK paper. Runs a two-pass temporal zoom inside
marlin.find()—first a coarse uniform scan with4frames, followed by a high-precision8-frame scan centered on the candidate sub-window.
qwen-fine-tune/
├── .gitignore # Standard exclude patterns (caches, weights, video files)
├── README.md # Comprehensive repository documentation
├── Fine-tuning VLM on Mac with MLX.md # Deep research guide on SVLM architecture and training
├── dataset_info.json # ShareGPT dataset registry for SFT and GRPO
├── train_vlm.py # LoRA Supervised Fine-Tuning (SFT) pipeline
├── grpo_align.py # Rule-based post-training GRPO alignment pipeline
├── inference.py # Public developer API wrapper and performance presets
├── scripts/ # Auxiliary scripts
│ └── profile_pipeline.py # Memory and compute profiler
├── tests/ # Automated verification suites
│ ├── test_cvpixelbuffer_ingestion.py # Direct macOS CVPixelBuffer texture test
│ ├── test_gpu_pipeline.py # Metal GPU array evaluation validation
│ └── test_mlx_resize.py # GPU bilinear resampler validator
└── scratch/ # Temporary developer scratchpads
Measured on an Apple Silicon Mac Studio running MLX 0.16.x with a 16-frame portrait video clip (sample_clip.mp4):
| Speed Preset | Frame Resolution | Caption Latency | Grounding Latency (marlin.find) |
Peak VRAM Footprint | DyToK Keyframe Optimization |
|---|---|---|---|---|---|
| Fast | 168x168 (~144 tok/f) | 2.41 seconds | 5.66 seconds (coarse+fine) | 8.18 GB | Selected 16 frames from 48 visually active candidates |
| Balanced | 224x224 (~256 tok/f) | 8.12 seconds | 3.64 seconds (with cache) | 8.18 GB | Selected 16 frames from 48 visually active candidates |
| Detail | 448x448 (~1024 tok/f) | 27.33 seconds | 19.23 seconds (high-res zoom) | 8.18 GB | Selected 16 frames from 48 visually active candidates |
Use marlin.process_textures() to ingest a stream of macOS CVPixelBuffer handles (from AVPlayer, camera sessions, or screen capture) directly into the Metal GPU pipeline, bypassing CPU decoding:
import objc
from Quartz import CoreVideo
from inference import SovereignVideoVLM
# 1. Initialize SovereignVideoVLM (runs natively on Metal)
marlin = SovereignVideoVLM("mlx-community/Qwen2.5-VL-3B-Instruct-4bit")
# 2. Ingest your array of CVPixelBuffer textures
prompt = "Analyze the video and return a structured JSON mapping events with exact timestamps."
raw_json = marlin.process_textures(pixel_buffers, prompt, speed_preset="fast")
print(raw_json)For high-level file-based video processing, use marlin.caption() and marlin.find():
from inference import SovereignVideoVLM
marlin = SovereignVideoVLM("mlx-community/Qwen2.5-VL-3B-Instruct-4bit")
# A. Generate structured JSON caption
caption_data = marlin.caption("clip.mp4", speed_preset="balanced")
print(f"Scene: {caption_data['scene']}")
for event in caption_data["events"]:
print(f" [{event['span'][0]}s - {event['span'][1]}s]: {event['event']}")
# B. Locate precise chronological event bounds (Hierarchical Zoom Scan)
start, end = marlin.find("clip.mp4", "striking the soccer ball", speed_preset="fast")
print(f"🎯 Event occurred between {start}s and {end}s")This codebase supports the complete lifecycle of model optimization locally on Apple Silicon:
Configure SFT with 4-bit quantization and unfrozen multimodal projection layers (LoRA applied to self-attention and MLP layers):
- Format your video dataset according to the ShareGPT schema inside
video_sft_dataset.json(registered via dataset_info.json):[ { "conversations": [ { "from": "user", "value": "<video>\nAnalyze the video and return a structured JSON mapping the scene description and event durations." }, { "from": "assistant", "value": "{\"scene\": \"baking demonstration\", \"events\": [{\"event\": \"slicing fruit\", \"span\": [0.0, 5.2]}]}" } ], "videos": ["/absolute/path/to/clip.mp4"] } ] - Trigger the training pipeline, supplying your system's unified memory size in GB (e.g., 32 for a 32GB Mac) to automatically scale batch parameters:
python3 train_vlm.py 32
To eliminate expensive teacher APIs, align your VLM using Group Relative Policy Optimization (GRPO). The alignment uses four rule-based reward functions to calculate relative advantages within completion groups (
- JSON Validity Reward: +1.0 for valid JSON matching the exact schema.
- Chronological Time Flow Reward: +1.0 if event timestamps strictly progress forward; -1.5 if time flows backward.
- Non-Thinking Prefix Suppression: -1.0 if the model leaks chain-of-thought
<think>tags. - Boundary Containment Reward: Penalizes outputs predicting timestamps extending beyond the video's actual duration.
Run the alignment pipeline locally:
python3 grpo_align.py-
Clone the Repository:
git clone https://codingstark-dev/qwen2.5-vl-apple-silicon.git cd qwen2.5-vl-apple-silicon -
Install Core Requirements: Ensure you have Python 3.10+ and install Apple Silicon Metal-optimized dependencies:
pip install mlx mlx-vlm opencv-python transformers numpy pyobjc
-
Verify the Texture Pipeline: Ensure Quartz/CoreVideo bindings work natively by running the automated CVPixelBuffer verification test:
python3 tests/test_cvpixelbuffer_ingestion.py