GPU Cluster Manager

You are managing a GPU cluster that combines multiple machines into one inference endpoint for running local LLMs via Ollama. The GPU cluster routes every request to the best available device automatically.

What this GPU cluster solves

Your desktop, laptop, and maybe an old Linux box all have GPUs sitting idle most of the time. You want one GPU cluster URL that uses all of them — without Kubernetes, without Docker, without editing config files. Just point your AI apps at the GPU cluster endpoint and let the cluster figure out which machine should handle each request.

This GPU cluster manager does exactly that. Install it, run two commands, and your GPU cluster machines discover each other automatically. The GPU cluster learns when your devices are free, pauses during video calls, and picks the best GPU cluster node for every request based on real-time conditions.

Getting started with the GPU cluster

CODEBLOCK0

On your main GPU cluster machine (the router):
CODEBLOCK1

On each other GPU cluster machine:
CODEBLOCK2

That's it. The GPU cluster nodes find the router via mDNS. No config files. Your GPU cluster is running.

If mDNS doesn't work on your GPU cluster network: INLINECODE0

GPU Cluster Endpoint

Your GPU cluster runs at http://localhost:11435. Point any AI app at the GPU cluster:

CODEBLOCK3

Works with: LangChain, CrewAI, AutoGen, LlamaIndex, Aider, Cline, Continue.dev, and any OpenAI-compatible client pointing at the GPU cluster.

GPU Cluster Smart Features

• - GPU cluster auto-discovery — machines find each other via mDNS, no config
• 7-signal GPU cluster scoring — picks the best machine based on loaded models, memory, queue depth, latency, and more
• GPU cluster meeting detection — pauses inference when your camera/mic is active (macOS)
• GPU cluster capacity learning — learns your weekly patterns (168-hour behavioral model)
• GPU cluster context protection — prevents models from reloading when apps send different context sizes
• GPU cluster auto-pull — if you request a model that doesn't exist, it downloads to the best GPU cluster node
• GPU cluster auto-retry — if a machine hiccups, retries on the next-best GPU cluster node

Check your GPU cluster

GPU cluster status — all machines

What models are available on the GPU cluster?

What's loaded in GPU cluster memory right now?

How healthy is the GPU cluster?

GPU cluster model recommendations

Returns GPU cluster recommendations based on your hardware — which models fit, which are too big, and the optimal GPU cluster mix.

GPU cluster recent activity

GPU cluster usage stats

GPU cluster settings

Manage GPU cluster models

GPU cluster per-app tracking

Tag your GPU cluster requests to see which apps use the most time:
CODEBLOCK14

GPU Cluster Dashboard

Open http://localhost:11435/dashboard for a visual GPU cluster overview. Eight tabs: Fleet Overview (live GPU cluster node cards), Trends (charts), Model Insights (performance comparison), Apps (per-app usage), Benchmarks, Health (automated GPU cluster checks), Recommendations (what models to run), Settings.

Try the GPU cluster

CODEBLOCK15

GPU Cluster Troubleshooting

Check what's slow in the GPU cluster

See GPU cluster failures

GPU Cluster Guardrails

• - Never restart or stop the GPU cluster without explicit user confirmation.
• Never delete or modify files in ~/.fleet-manager/ (contains all your GPU cluster data and logs).
• Do not pull or delete models on the GPU cluster without user confirmation — downloads can be 10-100+ GB.
• If a GPU cluster machine shows as offline, report it rather than attempting to SSH into it.

GPU Cluster Failure Handling

• - Connection refused → GPU cluster router may not be running, suggest herd or INLINECODE5
• 0 nodes online → suggest starting herd-node on GPU cluster devices
• mDNS discovery fails → use INLINECODE7
• GPU cluster requests hang → check for num_ctx in client requests; context protection handles it
• GPU cluster errors → check INLINECODE9