Skip to content

installation.md

Latest commit

 

History

History
116 lines (87 loc) · 3.61 KB

installation.md

File metadata and controls

116 lines (87 loc) · 3.61 KB
Raw

Installation

inferenced runs on Apple Silicon macOS only. The reason is in the charter: it exists to wrap MLX, which is Apple-only. Intel Macs and other platforms are out of scope.

Prerequisites

Component Why Install
macOS 14+ on Apple Silicon MLX requires a recent macOS and an arm64 SoC. (you have it)
Rust 1.75+ To build the daemon. rustup.rs or brew install rust
Python 3.10+ mlx-lm requires it. brew install python@3.12
mlx-lm Apple's reference LLM server. inferenced supervises it. python3.12 -m pip install --user --break-system-packages mlx-lm

Confirm the runtime is reachable:

which mlx_lm.server
# Typically: ~/Library/Python/3.12/bin/mlx_lm.server
# Or:        /opt/homebrew/bin/mlx_lm.server  (if pip-installed via Homebrew Python)

Build

git clone https://github.com/dormlab/inferenced
cd inferenced
cargo build --release --target aarch64-apple-darwin
ls -lh target/aarch64-apple-darwin/release/inferenced
# ~3 MB binary

The release profile uses lto = "fat", opt-level = 3, panic = "abort", and strip = true to keep the binary small.

Run interactively

./target/aarch64-apple-darwin/release/inferenced \
  --mlx-lm-server "$(which mlx_lm.server)" \
  --model mlx-community/Qwen2.5-3B-Instruct-4bit

The first request triggers a model download to ~/.cache/huggingface/ (~2 GB for Qwen2.5-3B-Instruct-4bit). After that, requests run on your GPU at the speed MLX gives you (~22 tok/sec on M4 for the 3B-4bit variant, in our testing).

Install as a launchd LaunchDaemon (boot persistence)

launchd LaunchDaemons run as root, which is what we want — Metal access on Apple Silicon needs root for powermetrics-adjacent reasons, and it's what mlx_lm.server expects when used in production.

1. Place the binary

sudo install -m 755 -o root -g wheel \
  target/aarch64-apple-darwin/release/inferenced \
  /usr/local/sbin/inferenced

2. Place the LaunchDaemon plist

A starting template lives at examples/launchd/dev.dormlab.inferenced.plist. Customize INFERENCED_MODEL and INFERENCED_MLX_LM_SERVER for your host.

sudo install -m 644 -o root -g wheel \
  examples/launchd/dev.dormlab.inferenced.plist \
  /Library/LaunchDaemons/dev.dormlab.inferenced.plist

3. Bootstrap

sudo launchctl bootstrap system /Library/LaunchDaemons/dev.dormlab.inferenced.plist
sudo launchctl print system/dev.dormlab.inferenced | head -20

4. Verify

curl -s http://localhost:11434/healthz
# ok

curl -s http://localhost:11434/v1/models | jq

Logs go to /var/log/inferenced.log (or whatever you set as StandardOutPath / StandardErrorPath in the plist).

Uninstall

sudo launchctl bootout system/dev.dormlab.inferenced
sudo rm /Library/LaunchDaemons/dev.dormlab.inferenced.plist
sudo rm /usr/local/sbin/inferenced

The model cache at ~/.cache/huggingface/ is left intact — delete manually if you want the disk back.

Common gotchas

  • pyexpat ImportError on mlx-lm install: a brew Python upgrade left the cached pyexpat.cpython-3xx-darwin.so linking against an older libexpat. brew reinstall python@3.12 fixes it.
  • Connection refused on first request: the first /v1/chat/completions triggers the model download (~2 GB for Qwen2.5-3B-4bit). Subsequent requests are fast. Watch progress in the logs.
  • source not allowed: your client isn't on a Tailscale CIDR. Either connect via Tailscale, or override with --allow-cidrs 0.0.0.0/0,::/0 for testing only.