MLX Local AI — Apple's ML Framework Powers Your Entire Fleet

Everything in this fleet runs on Apple's MLX framework. LLM inference, image generation, speech-to-text, embeddings — all MLX-native, all optimized for Apple Silicon's unified memory architecture.

The MLX stack

One router. One framework. Four modalities. All local.

Setup

CODEBLOCK0

All tools leverage MLX for Metal-accelerated inference on Apple Silicon's GPU cores.

LLM inference via MLX

Ollama runs models using MLX on Apple Silicon. Unified memory means the entire model stays in one address space — no PCIe bottleneck.

CODEBLOCK1

Image generation via MLX

Both mflux and DiffusionKit are pure MLX implementations — no PyTorch, no CUDA.

CODEBLOCK2

Speech-to-text via MLX

Qwen3-ASR transcribes audio using MLX acceleration.

CODEBLOCK3

Embeddings via MLX

Ollama embedding models run on the MLX backend.

CODEBLOCK4

Why MLX matters for local AI

• - Unified memory — model weights, activations, and KV cache share one memory pool. No CPU-GPU transfer overhead.
• Metal acceleration — MLX compiles to Metal shaders that run on Apple Silicon GPU cores (up to 80 on M3/M4 Ultra).
• Lazy evaluation — MLX only computes what's needed, reducing memory pressure.
• Dynamic shapes — no recompilation when input sizes change (unlike some CUDA frameworks).
• Apple-maintained — MLX is developed by Apple's ML research team, optimized for every chip generation.

Fleet performance on Apple Silicon

Monitor your MLX fleet

CODEBLOCK5

Dashboard at http://localhost:11435/dashboard — see every node, every model, every queue in real time.

Full documentation

• - Agent Setup Guide — all 4 model types
• Image Generation Guide — 3 backends
• API Reference

Contribute

Ollama Herd is open source (MIT) and built on the MLX ecosystem. We welcome contributions:

• - Star on GitHub — helps others discover the project
• Open an issue — bug reports, feature requests, questions
• AI agents welcome — CLAUDE.md provides full architectural context. Fork, branch, PR.
• 444 tests, async Python, runs in under 40 seconds. Hard to break things.

Guardrails

• - No automatic downloads — all model pulls require explicit user confirmation.
• Model deletion requires explicit user confirmation.
• All requests stay local — no data leaves your network.
• Never delete or modify files in ~/.fleet-manager/.