Foundry Local: On-Device AI SDK for Client Apps
/Foundry-Local
Foundry Local is Microsoft's SDK for running chat and speech-to-text models directly inside a C#, JavaScript, Python, or Rust app — no server, no API key. Reach for it if you're shipping a desktop or mobile app that needs offline inference and can live with a curated, quantized model list. Skip it if you need to serve many concurrent users; that's a job for server-side inference, not a client SDK.
What is Foundry Local?
Foundry Local is an end-to-end local AI runtime and SDK, roughly 20 MB, that runs chat and audio models entirely on the user's device. It bundles a curated model catalog, automatic NPU/GPU/CPU hardware acceleration, and an OpenAI-compatible API, so a device with no network connection can still answer prompts or transcribe audio. The catch: you're limited to whichever models Microsoft has vetted and quantized for that catalog.
Core Capabilities
- ✓Lightweight runtime (~20 MB) built on ONNX Runtime that handles model acquisition, hardware acceleration, and inference in-process — no separate server to run.
- ✓Curated model catalog covering chat completions (GPT OSS, Qwen, DeepSeek, Mistral, Phi) and speech-to-text (Whisper), with every model quantized and compressed before it ships.
- ✓Automatic hardware acceleration — Foundry Local detects the device's NPU, GPU, or CPU and picks the best execution provider without you writing detection logic.
- ✓Smart model management: models download on first use, cache locally for instant next launches, and the best-performing variant for that hardware is selected automatically.
- ✓OpenAI-compatible API, including the OpenAI Responses API format, so an app already built against the OpenAI SDK can point at a Foundry Local endpoint with minimal code changes.
- ✓Optional local web server for cases where you want to share one model across multiple processes or wire it into LangChain via REST, instead of embedding the SDK directly.
Typical Use Cases
- •Desktop or mobile apps that need chat completions to keep working on a plane or in a facility with no reliable network.
- •Embedded AI features in client-side apps where sending user text or audio to a cloud API isn't acceptable for privacy or compliance reasons.
- •Voice note or meeting apps that transcribe audio locally with the bundled Whisper models instead of shipping recordings to a third-party API.
- •Electron or native desktop tools that want offline AI without provisioning an Azure subscription or managing API keys for every install.
Getting Started with the SDK
Install is a single package command per language: `npm install foundry-local-sdk` for JavaScript or `pip install foundry-local-sdk` for Python; C# and Rust bindings are also available through their own samples. There's no separate runtime download, Azure subscription, or API key setup — the local AI SDK pulls in what it needs the first time your app requests a model. A separate CLI (public preview, tagged `cli-preview-0.10.0`) is downloadable from GitHub releases if you want to explore models interactively before wiring up code.
Implementing Chat and Audio Models
In JavaScript, you create a `FoundryLocalManager`, fetch a model from the catalog with `manager.catalog.getModel('qwen2.5-0.5b')`, call `.download()` and `.load()`, then get a chat client with `model.createChatClient()` and call `.completeChat()` with a messages array — the Python SDK follows the same pattern with `catalog.get_model()` and `get_chat_client()`. Audio transcription uses a Whisper model instead: load `whisper-tiny`, create an audio client with `model.createAudioClient()`, set `audioClient.settings.language`, then call `.transcribe()` for a full file or `.transcribeStreaming()` to get text chunks as they're produced. One `FoundryLocalManager` instance can hold both a chat model and an audio model at the same time, and you call `.unload()` on each when you're done with it.
Who Benefits from Foundry Local?
Foundry Local is aimed at developers shipping a client application — desktop, mobile, or a kiosk-style device — who need chat or transcription features to work without a live backend. It fits teams already comfortable with C#, JavaScript, Python, or Rust who don't want to run and pay for inference infrastructure just to add AI to one app. It's a poor fit for a team building a hosted product for many simultaneous users; that's a different problem than embedding a model in one person's copy of an app.
Strengths
- ✓No per-token cost, no API key, and no backend to run — once the model is cached, inference happens entirely offline with zero network latency.
- ✓Automatic hardware acceleration means you don't hand-roll NPU/GPU/CPU detection; Foundry Local picks the execution provider for you.
- ✓An OpenAI-compatible API, including the OpenAI Responses API format, makes it a low-effort swap for apps already built against the OpenAI SDK.
- ✓The package itself is small (~20 MB), so it doesn't bloat an app's install size the way a full local runtime might.
When Not to Use Foundry Local
- △The model catalog is curated, not open — you get GPT OSS, Qwen, DeepSeek, Mistral, Phi, and Whisper variants that Microsoft has quantized, not whatever checkpoint you found elsewhere.
- △It's built for one user per device; the README itself says it isn't designed as a server inference stack and points to vLLM or Triton Inference Server for concurrent multi-user serving.
- △The CLI is still a public preview (`cli-preview-0.10.0`), distributed as a GitHub release download rather than a package-manager install, so it's less polished than the SDK path.
- △License isn't uniform: the SDK is MIT, but the CLI ships under the Microsoft Software License Terms, and each model in the catalog carries its own license — worth checking before you redistribute.
Alternatives for Server-Side Inference
Common Questions
Foundry Local is not primarily a web server or CLI tool — it's an end-to-end local AI SDK that your application embeds directly; the optional web server and CLI exist for development and experimentation, not as the core product.
Foundry Local restricts the catalog to models Microsoft has tested and quantized for consumer hardware and small enough to distribute with an app, trading model variety for predictable on-device behavior in production.
Foundry Local can technically run on server hardware, but it's built for one user per device at a time; for concurrent multi-user serving the README points to dedicated stacks like vLLM or Triton Inference Server instead.
Foundry Local supports Windows, macOS on Apple silicon, and Linux.
The Foundry Local SDK is MIT-licensed; the separate CLI uses the Microsoft Software License Terms, and each model in the catalog carries its own license terms.
Foundry Local's catalog covers chat completion models such as GPT OSS, Qwen, DeepSeek, Mistral, and Phi, plus Whisper models for audio transcription — all quantized for on-device use.
The problem it solves
Adding chat or transcription to a client app usually means standing up a backend, managing API keys, and accepting per-token cost and network latency — or hand-rolling ONNX Runtime integration and hardware detection yourself. Foundry Local packages the model download, quantization, hardware-provider selection, and an OpenAI-compatible client into one SDK. No backend required. A C#, JavaScript, Python, or Rust app can call a local model the same way it would call a cloud one, without owning the plumbing in between.
Related repositories
Curious whether Foundry-Local is right for you?
Let ChatGPT, Claude, or Perplexity look into it — click below and see what AI actually says about Foundry-Local.