TopGit
GitHub Repo Review

SIE Inference Server: Self-Hosted Model Cluster for Agents

superlinked/sie
STopGit review image for superlinked/sie
Review by Topgit.dev for superlinked/sie, with GitHub repository stats and README context.
Quick verdict

SIE (Superlinked Inference Engine) is an open-source inference server that consolidates all the model calls an agent needs into a single cluster. It's worth using if you are tired of juggling a separate service per model and want OpenAI-compatible endpoints backed by a real Kubernetes deployment surface. Skip it if a single-model setup covers your needs and you do not want the infrastructure overhead.

Stars
β˜… 3.5k
Forks
β‘‚ 337
Contributors
πŸ‘₯ 14
Language
Python
License
Apache-2.0
Topic
β€”
Updated
Nov 2026
Homepage

What is SIE?

SIE is a self-hosted inference server that runs every model an agent requires through one OpenAI-compatible API. It replaces separate services for search embeddings, document OCR, text generation, and vision tasks with a single gateway that loads models on demand. Production deployments use Kubernetes with a load-balancing gateway, KEDA autoscaling, and Grafana dashboards.

Core capabilities

  • βœ“Embedding and retrieval models (BAAI/bge-m3, SPLADE, Colbertv2) benchmarked on MTEB, plus cross-encoder rerankers like Qwen3-Reranker
  • βœ“Document-to-markdown pipeline: PDF, Office files, and scans via LightOnOCR, GLM-OCR, MinerU, PaddleOCR-VL, or Docling
  • βœ“LLM serving for agent loops and structured output: Qwen3-0.6B through Qwen3.8-27B on SGLang, with JSON schema support via GLiNER2 and NuNER_Zero
  • βœ“Vision models: Florence-2 for captioning, OWLv2 for object detection, Grounding DINO for localization, and Whisper Large v3 Turbo for transcription
  • βœ“OpenAI-compatible API surface: /v1/embeddings, /v1/chat/completions, /v1/completions, /v1/responses for drop-in migration
  • βœ“Kubernetes deployment via Helm charts with load-balancing gateway, KEDA autoscaling to zero, and Grafana dashboards; Terraform modules for GKE, EKS, AKS, and Alibaba ACK
  • βœ“On-demand model loading with LRU eviction; serves multiple models simultaneously from one cluster
  • βœ“Nine framework integrations: LangChain, LlamaIndex, Haystack, DSPy, CrewAI, Chroma, Qdrant, Weaviate, and LanceDB
How this repository's GitHub stars have grown over time. Source: star-history.com.View the star history β†—

What you can build with SIE

  • β€’Running all inference for a RAG pipeline: embed a query, retrieve context, rerank results, and generate an answer through a single gateway
  • β€’Agentic workflows that need structured entity extraction, content safety checks, and tool-calling LLMs from one cluster
  • β€’Document processing pipelines that convert PDFs to markdown, extract entities, and store results in a vector database
  • β€’Multi-cloud production deployments with autoscaling, centralized monitoring, and Terraform-managed infrastructure
  • β€’Replacing an existing OpenAI proxy with an on-premise equivalent that serves open models across all task types

Running SIE locally

Python 3.12 is required. Install the server with pip install sie-server[local] and start with sie-server serve for local development. Docker images are provided for NVIDIA GPU (cuda12) and CPU-only environments, with separate bundles for Transformers 5 OCR models and SGLang vision models. Helm charts deploy the full production cluster to Kubernetes, with values files for GKE, EKS, AKS, and Alibaba ACK.

Calling SIE from Python and TypeScript

After starting the server, install the Python SDK (pip install sie-sdk) or TypeScript SDK (npm install @superlinked/sie-sdk). Call client.encode() to generate embeddings, client.score() to rerank search results, client.extract() for named entity recognition, and client.generate() for text completion. Example: result = client.encode('sentence-transformers/all-MiniLM-L6-v2', Item(text='Hello world')) returns a dense vector of shape (384,).

Why use SIE

  • βœ“One cluster, one gateway, one API for every model type an agent needs, eliminating per-model deployment overhead
  • βœ“OpenAI-compatible endpoints mean existing code using the OpenAI SDK migrates without changes
  • βœ“Model catalog with 100+ pre-configured open models, each referenced by a stable sie_id in the SDK
  • βœ“Production-ready Kubernetes stack with Helm, KEDA autoscaling, Grafana, and Terraform modules for four major cloud providers
  • βœ“On-demand loading with LRU eviction keeps memory bounded while serving multiple model families simultaneously

Known limitations

  • β–³GPU is effectively required for any real production workload; CPU-only mode is usable only for development with small models
  • β–³First inference call on a model downloads weights from HuggingFace, introducing latency on cold starts
  • β–³The production Kubernetes setup requires significant infrastructure expertise; it is not a simple single-binary deployment
  • β–³Telemetry is enabled by default (anonymous usage data), which may conflict with air-gapped or privacy-sensitive environments
  • β–³Some OCR models require specific Docker images (transformers5 or sglang-vision-extract) due to dependency conflicts, complicating the initial setup
View on GitHub β†—Homepage β†—

How SIE compares to other inference servers

Ollama: runs locally on your machine with a simple CLI, but lacks the Kubernetes production surface, OpenAI-compatible gateway, and multi-model catalog that SIE providesvLLM: optimized for high-throughput LLM serving on NVIDIA GPUs, but focuses on text generation only and does not cover embedding, OCR, or vision modelsText Generation Inference (TGI): HuggingFace's official inference server for text models, comparable to vLLM in scope, with no built-in support for retrieval models or document processing pipelinesRay Serve: a general-purpose distributed serving framework that can host any model but requires you to build the multi-model coordination layer yourselflm Harness: focuses on benchmarking rather than serving; it evaluates models but does not run inference in production

Is SIE right for your project?

Teams building agents that need multiple model types in production and want one API surface, centralized scaling, and a pre-built Kubernetes deployment surface. SIE is a strong fit for RAG pipelines, document processing workflows, and any agentic system that calls embedding, generation, and extraction models together. If you are running a single model in development or prefer to manage each model server independently, the operational overhead is not justified.

Frequently asked questions

Is SIE free and open source?

Yes. SIE is fully open source under the Apache 2.0 license, and the entire production stack including Helm charts and Terraform modules carries the same license.

What GPU does SIE require?

SIE targets NVIDIA GPUs with CUDA. Embedding and reranking models work on CPU for development, but document processing, vision tasks, and LLM generation all need a CUDA GPU to be practical in production.

How many models does SIE support?

The model catalog lists over 100 pre-configured models across embedding, generation, vision, and OCR families. Each is a YAML config in the repository with a stable sie_id for SDK usage.

Can SIE replace multiple separate model servers?

Yes. SIE is designed to replace a separate model server per task. The gateway serves multiple models from one cluster, loading each on demand and evicting the least recently used when memory is tight.

Does SIE support Apple Silicon (MLX)?

SIE has MLX support for text generation via the TypeScript SDK. NVIDIA CUDA remains the primary GPU path for production workloads.

How does SIE compare to Ollama?

Ollama runs locally on a single machine with a simple CLI. SIE targets production Kubernetes clusters with a load-balancing gateway, autoscaling, Grafana dashboards, and a model catalog spanning embedding, vision, and OCR tasks that Ollama does not cover.

The problem it solves

Agents typically need embedding models for retrieval, LLMs for generation, OCR pipelines for documents, vision models for images, and rerankers for ranking results. The usual approach runs a separate model server per task, each with its own API, scaling logic, and deployment config. SIE consolidates those into one cluster with one gateway.

Related repositories

Source & attribution

Source: https://github.com/superlinked/sie

GitHub data Β· last synced Sep 23, 2026Reviewed by Henry
← Back to TopGit

Curious whether sie is right for you?

Let ChatGPT, Claude, or Perplexity look into it β€” click below and see what AI actually says about sie.

GitHub