pytorch/serve là dự án Java với 4.3k sao trong nhóm AI Tools. Serve, optimize and scale PyTorch models in production
Tóm tắt dựng từ metadata GitHub của chính dự án — chưa có bài review TopGit. Trang sẽ tự động cập nhật khi bài review đầy đủ được xuất bản.
VÌ SAO CHƯA CÓ REVIEW
TopGit viết bài đầy đủ cho repo có nhiều sao nhất và được yêu cầu nhiều nhất. Trang này là snapshot trong thời gian chờ — xem README gốc ở tab READ ME.
This project is no longer actively maintained. While existing releases remain available, there are no planned updates, bug fixes, new features, or security patches. Users should be aware that vulnerabilities may not be addressed.
❗ANNOUNCEMENT: Security Changes❗
TorchServe now enforces token authorization enabled and model API control disabled by default. These security features are intended to address the concern of unauthorized API calls and to prevent potential malicious code from being introduced to the model server. Refer the following documentation for more information: Token Authorization, Model API control
TorchServe
TorchServe is a flexible and easy-to-use tool for serving and scaling PyTorch models in production.
# Make sure to install torchserve with pip or conda as described above and login with `huggingface-cli login`
python -m ts.llm_launcher --model_id meta-llama/Llama-3.2-3B-Instruct --disable_token_auth
# Try it out
curl -X POST -d '{"model":"meta-llama/Llama-3.2-3B-Instruct", "prompt":"Hello, my name is", "max_tokens": 200}' --header "Content-Type: application/json" "http://localhost:8080/predictions/model/1.0/v1/completions"
TRT-LLM Engine
# Make sure to install torchserve with python venv as described above and login with `huggingface-cli login`
# pip install -U --use-deprecated=legacy-resolver -r requirements/trt_llm.txt
python -m ts.llm_launcher --model_id meta-llama/Meta-Llama-3.1-8B-Instruct --engine trt_llm --disable_token_auth
# Try it out
curl -X POST -d '{"prompt":"count from 1 to 9 in french ", "max_tokens": 100}' --header "Content-Type: application/json" "http://localhost:8080/predictions/model"
🚢 Quick Start LLM Deployment with Docker
#export token=<HUGGINGFACE_HUB_TOKEN>
docker build --pull . -f docker/Dockerfile.vllm -t ts/vllm
docker run --rm -ti --shm-size 10g --gpus all -e HUGGING_FACE_HUB_TOKEN=$token -p 8080:8080 -v data:/data ts/vllm --model_id meta-llama/Meta-Llama-3-8B-Instruct --disable_token_auth
# Try it out
curl -X POST -d '{"model":"meta-llama/Meta-Llama-3-8B-Instruct", "prompt":"Hello, my name is", "max_tokens": 200}' --header "Content-Type: application/json" "http://localhost:8080/predictions/model/1.0/v1/completions"
Refer to LLM deployment for details and other methods.
⚡ Why TorchServe
Write once, run anywhere, on-prem, on-cloud, supports inference on CPUs, GPUs, AWS Inf1/Inf2/Trn1, Google Cloud TPUs, Nvidia MPS
Model Management API: multi model management with optimized worker to model allocation
Inference API: REST and gRPC support for batched inference
TorchServe Workflows: deploy complex DAGs with multiple interdependent models
Default way to serve PyTorch models in
Sagemaker
Vertex AI
Kubernetes with support for autoscaling, session-affinity, monitoring using Grafana works on-prem, AWS EKS, Google GKE, Azure AKS
Kserve: Supports both v1 and v2 API, autoscaling and canary deployments for A/B testing
Kubeflow
MLflow
Export your model for optimized inference. Torchscript out of the box, PyTorch Compiler preview, ORT and ONNX, IPEX, TensorRT, FasterTransformer, FlashAttention (Better Transformers)
Performance Guide: builtin support to optimize, benchmark, and profile PyTorch and TorchServe performance
Expressive handlers: An expressive handler architecture that makes it trivial to support inferencing for your use case with many supported out of the box
Metrics API: out-of-the-box support for system-level metrics with Prometheus exports, custom metrics,
Large Model Inference Guide: With support for GenAI, LLMs including
SOTA GenAI performance using torch.compile
Fast Kernels with FlashAttention v2, continuous batching and streaming response
Walmart Search: Serving Models at a Scale on TorchServe
🎥 Scaling inference on CPU with TorchServe
🎥 TorchServe C++ backend
Grokking Intel CPU PyTorch performance from first principles: a TorchServe case study
Grokking Intel CPU PyTorch performance from first principles( Part 2): a TorchServe case study
Case Study: Amazon Ads Uses PyTorch and AWS Inferentia to Scale Models for Ads Processing
Optimize your inference jobs using dynamic batch inference with TorchServe on Amazon SageMaker
Using AI to bring children's drawings to life
🎥 Model Serving in PyTorch
Evolution of Cresta's machine learning architecture: Migration to AWS and PyTorch
🎥 Explain Like I’m 5: TorchServe
🎥 How to Serve PyTorch Models with TorchServe
How to deploy PyTorch models on Vertex AI
Quantitative Comparison of Serving Platforms
Efficient Serverless deployment of PyTorch models on Azure
Deploy PyTorch models with TorchServe in Azure Machine Learning online endpoints
Dynaboard moving beyond accuracy to holistic model evaluation in NLP
A MLOps Tale about operationalising MLFlow and PyTorch
Operationalize, Scale and Infuse Trust in AI Models using KFServing
How Wadhwani AI Uses PyTorch To Empower Cotton Farmers
TorchServe Streamlit Integration
Dynabench aims to make AI models more robust through distributed human workers
Announcing TorchServe
💖 All Contributors
Made with contrib.rocks.
⚖️ Disclaimer
This repository is jointly operated and maintained by Amazon, Meta and a number of individual contributors listed in the CONTRIBUTORS file. For questions directed at Meta, please send an email to [email protected]. For questions directed at Amazon, please send an email to [email protected]. For all other questions, please open up an issue in this repository here.
TorchServe acknowledges the Multi Model Server (MMS) project from which it was derived
Trang TopGit này là một snapshot — tab "Readme" hiển thị nguyên văn README của repo (đã bỏ link, giữ ảnh). Repo GitHub ở github.com/pytorch/serve là nguồn chính thức.
pytorch/serve có những chủ đề gì?
GitHub topics của pytorch/serve: "cpu", "deep-learning", "docker", "gpu", "kubernetes", "machine-learning", "metrics", "mlops", "optimization", "pytorch", "serving". TopGit xếp repo vào nhóm AI Tools.
pytorch/serve có phải mã nguồn mở không?
Có — pytorch/serve phát hành theo license Apache-2.0, nghĩa là mã nguồn mở để đọc, fork và (tùy license) tái sử dụng. Mã: github.com/pytorch/serve.
pytorch/serve có trang demo không?
Dự án có trang chủ ở https://pytorch.org/serve/. Tab "Readme" ở trang này thường có ảnh chụp và hướng dẫn bắt đầu nhanh.
pytorch/serve còn đang phát triển không?
Commit gần nhất trên pytorch/serve là 1.3 năm trước (theo timestamp GitHub). Repo có 881 fork — một chỉ báo về mức độ quan tâm của cộng đồng.
pytorch/serve dùng license gì?
pytorch/serve phát hành theo license Apache-2.0. Nên mở file LICENSE trên GitHub để xác nhận — license metadata đôi khi lệch với thực tế dự án.
Đọc đầy đủ README ở tab phía trên.
Vẫn đang phân vân về serve?
Một cú bấm sẽ gửi câu hỏi kèm trang này cho AI — xem AI nói gì về serve.