LocalAI — run any AI model on your own hardware
mudler/LocalAILocalAI is an open-source engine for running LLMs plus vision, voice, image and video models on your own hardware, behind OpenAI-, Anthropic- and ElevenLabs-compatible APIs — no GPU required.
A self-hosted, OpenAI-compatible AI runtime for teams that want to keep data on their own infrastructure. Best if you already build on the OpenAI or Anthropic SDKs and want a drop-in local backend; expect more operational work than a hosted API.
The problem it solves
Hosted AI APIs send your prompts and data to a third party, bill per token, and tie you to one vendor. Running models yourself usually means gluing together separate tools for text, speech and images, each with its own interface.
What is it?
LocalAI is an open-source AI engine written in Go. It keeps a small core and loads backends on demand — each wrapping an engine such as llama.cpp, vLLM, whisper.cpp, stable-diffusion or MLX in its own image — so you install only what a given model needs. A single API surface serves LLM, vision, voice, image and video models, and it mirrors the OpenAI, Anthropic and ElevenLabs API shapes so existing clients can point at it by changing the base URL.
Why it's getting attention
LocalAI sits in the active local-AI and self-hosting space and has roughly 47,000 GitHub stars. Its draw is drop-in API compatibility combined with broad hardware support — NVIDIA, AMD, Intel, Apple Silicon, Vulkan or CPU-only — which lets teams move from a hosted API to local inference without rewriting client code.
Key features
- ✓Drop-in API compatibility with OpenAI, Anthropic and ElevenLabs
- ✓Composable backends (llama.cpp, vLLM, whisper.cpp, stable-diffusion, MLX) pulled on demand
- ✓One API for LLM, vision, voice, image and video models
- ✓Broad hardware support: NVIDIA, AMD, Intel, Apple Silicon, Vulkan or CPU-only
- ✓Multi-user controls: API key auth, user quotas and role-based access
- ✓Built-in AI agents with tool use, RAG and MCP
- ✓Privacy-first: data stays on your own infrastructure
Best use cases
- •Replacing a hosted OpenAI/Anthropic endpoint with a local backend for existing apps
- •Serving LLM, speech-to-text, text-to-speech and image generation from one service
- •Keeping sensitive prompts and documents on-premise for privacy or compliance
- •Sharing hardware across multiple users with API keys and quotas
How to install / try
The README documents a macOS DMG download and Docker/Podman images. CPU-only quickstart: `docker run -ti --name local-ai -p 8080:8080 localai/localai:latest`. GPU images are provided for NVIDIA (CUDA 12/13), AMD (ROCm), Intel (oneAPI) and Vulkan. Full installation guides are at localai.io.
How to use
With the server running, load a model from the gallery or a registry, e.g. `local-ai run llama-3.2-1b-instruct:q4_k_m`, then call the OpenAI-compatible endpoint on port 8080. `local-ai chat --model <name>` opens an interactive terminal session. API details are documented at localai.io.
Strengths
- ✓Drop-in OpenAI/Anthropic/ElevenLabs compatibility means minimal client changes
- ✓On-demand backends keep the install lean — you pull only what you use
- ✓Wide hardware coverage, including CPU-only and Apple Silicon
- ✓MIT-licensed and privacy-first by design
Limitations & risks
- △Self-hosting shifts operational work to you: GPU drivers, model management and updates
- △Local speed and quality depend on your hardware; large models need significant RAM or VRAM
- △The README relies on external docs (localai.io) for many specifics rather than being self-contained
- △The macOS DMG is not signed by Apple and needs a manual quarantine-removal step
Alternatives
Who should try it — and who should skip
Choose LocalAI if you already use the OpenAI or Anthropic SDKs and want to run models on your own hardware for privacy or cost control, and you are comfortable operating containers and GPUs. Skip it if you want a zero-ops hosted API or a polished consumer chat app.
Frequently asked questions
No. The project states it runs on CPU-only as well as NVIDIA, AMD, Intel, Apple Silicon and Vulkan. A GPU improves speed for larger models but is not required.
Yes. LocalAI exposes drop-in OpenAI-compatible endpoints and also mirrors the Anthropic and ElevenLabs APIs, so many existing clients work by changing the base URL.
It runs vision, voice, image and video models through backends such as whisper.cpp and stable-diffusion, all behind the same API.
The repository is MIT licensed.