SIE Inference Server: Self-Hosted Model Cluster for Agents
SIE (Superlinked Inference Engine) is an open-source inference server that consolidates all the model calls an agent needs into a single cluster. It's worth using if you are tired of juggling a separate service per model and want OpenAI-compatible endpoints backed by a real Kubernetes deployment surface. Skip it if a single-model setup covers your needs and you do not want the infrastructure overhead.
What is SIE?
SIE is a self-hosted inference server that runs every model an agent requires through one OpenAI-compatible API. It replaces separate services for search embeddings, document OCR, text generation, and vision tasks with a single gateway that loads models on demand. Production deployments use Kubernetes with a load-balancing gateway, KEDA autoscaling, and Grafana dashboards.
Core capabilities
- βEmbedding and retrieval models (BAAI/bge-m3, SPLADE, Colbertv2) benchmarked on MTEB, plus cross-encoder rerankers like Qwen3-Reranker
- βDocument-to-markdown pipeline: PDF, Office files, and scans via LightOnOCR, GLM-OCR, MinerU, PaddleOCR-VL, or Docling
- βLLM serving for agent loops and structured output: Qwen3-0.6B through Qwen3.8-27B on SGLang, with JSON schema support via GLiNER2 and NuNER_Zero
- βVision models: Florence-2 for captioning, OWLv2 for object detection, Grounding DINO for localization, and Whisper Large v3 Turbo for transcription
- βOpenAI-compatible API surface: /v1/embeddings, /v1/chat/completions, /v1/completions, /v1/responses for drop-in migration
- βKubernetes deployment via Helm charts with load-balancing gateway, KEDA autoscaling to zero, and Grafana dashboards; Terraform modules for GKE, EKS, AKS, and Alibaba ACK
- βOn-demand model loading with LRU eviction; serves multiple models simultaneously from one cluster
- βNine framework integrations: LangChain, LlamaIndex, Haystack, DSPy, CrewAI, Chroma, Qdrant, Weaviate, and LanceDB
What you can build with SIE
- β’Running all inference for a RAG pipeline: embed a query, retrieve context, rerank results, and generate an answer through a single gateway
- β’Agentic workflows that need structured entity extraction, content safety checks, and tool-calling LLMs from one cluster
- β’Document processing pipelines that convert PDFs to markdown, extract entities, and store results in a vector database
- β’Multi-cloud production deployments with autoscaling, centralized monitoring, and Terraform-managed infrastructure
- β’Replacing an existing OpenAI proxy with an on-premise equivalent that serves open models across all task types
Running SIE locally
Python 3.12 is required. Install the server with pip install sie-server[local] and start with sie-server serve for local development. Docker images are provided for NVIDIA GPU (cuda12) and CPU-only environments, with separate bundles for Transformers 5 OCR models and SGLang vision models. Helm charts deploy the full production cluster to Kubernetes, with values files for GKE, EKS, AKS, and Alibaba ACK.
Calling SIE from Python and TypeScript
After starting the server, install the Python SDK (pip install sie-sdk) or TypeScript SDK (npm install @superlinked/sie-sdk). Call client.encode() to generate embeddings, client.score() to rerank search results, client.extract() for named entity recognition, and client.generate() for text completion. Example: result = client.encode('sentence-transformers/all-MiniLM-L6-v2', Item(text='Hello world')) returns a dense vector of shape (384,).
Why use SIE
- βOne cluster, one gateway, one API for every model type an agent needs, eliminating per-model deployment overhead
- βOpenAI-compatible endpoints mean existing code using the OpenAI SDK migrates without changes
- βModel catalog with 100+ pre-configured open models, each referenced by a stable sie_id in the SDK
- βProduction-ready Kubernetes stack with Helm, KEDA autoscaling, Grafana, and Terraform modules for four major cloud providers
- βOn-demand loading with LRU eviction keeps memory bounded while serving multiple model families simultaneously
Known limitations
- β³GPU is effectively required for any real production workload; CPU-only mode is usable only for development with small models
- β³First inference call on a model downloads weights from HuggingFace, introducing latency on cold starts
- β³The production Kubernetes setup requires significant infrastructure expertise; it is not a simple single-binary deployment
- β³Telemetry is enabled by default (anonymous usage data), which may conflict with air-gapped or privacy-sensitive environments
- β³Some OCR models require specific Docker images (transformers5 or sglang-vision-extract) due to dependency conflicts, complicating the initial setup
How SIE compares to other inference servers
Is SIE right for your project?
Teams building agents that need multiple model types in production and want one API surface, centralized scaling, and a pre-built Kubernetes deployment surface. SIE is a strong fit for RAG pipelines, document processing workflows, and any agentic system that calls embedding, generation, and extraction models together. If you are running a single model in development or prefer to manage each model server independently, the operational overhead is not justified.
Frequently asked questions
Yes. SIE is fully open source under the Apache 2.0 license, and the entire production stack including Helm charts and Terraform modules carries the same license.
SIE targets NVIDIA GPUs with CUDA. Embedding and reranking models work on CPU for development, but document processing, vision tasks, and LLM generation all need a CUDA GPU to be practical in production.
The model catalog lists over 100 pre-configured models across embedding, generation, vision, and OCR families. Each is a YAML config in the repository with a stable sie_id for SDK usage.
Yes. SIE is designed to replace a separate model server per task. The gateway serves multiple models from one cluster, loading each on demand and evicting the least recently used when memory is tight.
SIE has MLX support for text generation via the TypeScript SDK. NVIDIA CUDA remains the primary GPU path for production workloads.
Ollama runs locally on a single machine with a simple CLI. SIE targets production Kubernetes clusters with a load-balancing gateway, autoscaling, Grafana dashboards, and a model catalog spanning embedding, vision, and OCR tasks that Ollama does not cover.
The problem it solves
Agents typically need embedding models for retrieval, LLMs for generation, OCR pipelines for documents, vision models for images, and rerankers for ranking results. The usual approach runs a separate model server per task, each with its own API, scaling logic, and deployment config. SIE consolidates those into one cluster with one gateway.
Related repositories
Curious whether sie is right for you?
Let ChatGPT, Claude, or Perplexity look into it β click below and see what AI actually says about sie.
