OpenNMT/CTranslate2 is one of the server-side repositories TopGit tracks, currently at 4.8k stars, written primarily in C++. Fast inference engine for Transformer models
Snapshot summary built from the project's own GitHub metadata — there's no written TopGit review yet. The page will update automatically when a full review is published.
WHY NO REVIEW YET
TopGit writes full reviews for the most-starred, most-requested repositories. This page is a snapshot until then — see the READ ME tab for the original README in full.
CTranslate2 is a C++ and Python library for efficient inference with Transformer models.
The project implements a custom runtime that applies many performance optimization techniques such as weights quantization, layers fusion, batch reordering, etc., to accelerate and reduce the memory usage of Transformer models on CPU and GPU.
The following model types are currently supported:
Text-to-speech (TTS) models: Drama 3 (experimental)
Compatible models should be first converted into an optimized model format. The library includes converters for multiple frameworks:
OpenNMT-py
OpenNMT-tf
Fairseq
Marian
OPUS-MT
Transformers
The project is production-oriented and comes with backward compatibility guarantees, but it also includes experimental features related to model compression and inference acceleration.
Key features
Fast and efficient execution on CPU and GPU The execution is significantly faster and requires less resources than general-purpose deep learning frameworks on supported models and tasks thanks to many advanced optimizations: layer fusion, padding removal, batch reordering, in-place operations, caching mechanism, etc.
Quantization and reduced precision The model serialization and computation support weights with reduced precision: 16-bit floating points (FP16), 16-bit brain floating points (BF16), 16-bit integers (INT16), 8-bit integers (INT8) and AWQ quantization (INT4).
Multiple CPU architectures support The project supports x86-64 and AArch64/ARM64 processors and integrates multiple backends that are optimized for these platforms: Intel MKL, oneDNN, OpenBLAS, Ruy, and Apple Accelerate.
Automatic CPU detection and code dispatch One binary can include multiple backends (e.g. Intel MKL and oneDNN) and instruction set architectures (e.g. AVX, AVX2) that are automatically selected at runtime based on the CPU information.
Parallel and asynchronous execution Multiple batches can be processed in parallel and asynchronously using multiple GPUs or CPU cores.
Dynamic memory usage The memory usage changes dynamically depending on the request size while still meeting performance requirements thanks to caching allocators on both CPU and GPU.
Lightweight on disk Quantization can make the models 4 times smaller on disk with minimal accuracy loss.
Simple integration The project has few dependencies and exposes simple APIs in Python and C++ to cover most integration needs.
Configurable and interactive decoding Advanced decoding features allow autocompleting a partial sequence and returning alternatives at a specific location in the sequence.
Support tensor parallelism for distributed inference Very large model can be split into multiple GPUs. Following this documentation to set up the required environment.
Some of these features are difficult to achieve with standard deep learning frameworks and are the motivation for this project.
Installation and usage
CTranslate2 can be installed with pip:
pip install ctranslate2
The Python module is used to convert models and can translate or generate text with few lines of code:
See the documentation for more information and examples.
If you have an AMD ROCm GPU, we provide specific Python wheels on the releases page.
Apple Metal (experimental). Apple Silicon GPUs can be used with device="mps". The backend is included in Apple Silicon Python wheels and can be enabled in source builds with -DWITH_MPS=ON. It remains an explicit runtime opt-in: device="auto" always selects CPU. See hardware support for the supported features.
Recommended libraries
ctranslate2-web-server: a web server exposing an OpenAI-compatible REST API for CTranslate2 models.
faster-whisper: fast speech-to-text transcription with Whisper and CTranslate2.
faster-tts: text-to-speech with CTranslate2 and experimental Drama 3 support.
Benchmarks
We translate the En->De test set newstest2014 with multiple models:
OpenNMT-tf WMT14: a base Transformer trained with OpenNMT-tf on the WMT14 dataset (4.5M lines)
OpenNMT-py WMT14: a base Transformer trained with OpenNMT-py on the WMT14 dataset (4.5M lines)
OPUS-MT: a base Transformer trained with Marian on all OPUS data available on 2020-02-26 (81.9M lines)
The benchmark reports the number of target tokens generated per second (higher is better). The results are aggregated over multiple runs. See the benchmark scripts for more details and reproduce these numbers.
Please note that the results presented below are only valid for the configuration used during this benchmark: absolute and relative performance may change with different settings.
CPU
Tokens per second
Max. memory
BLEU
OpenNMT-tf WMT14 model
OpenNMT-tf 2.31.0 (with TensorFlow 2.11.0)
209.2
2653MB
26.93
OpenNMT-py WMT14 model
OpenNMT-py 3.0.4 (with PyTorch 1.13.1)
275.8
2012MB
26.77
- int8
323.3
1359MB
26.72
CTranslate2 3.6.0
658.8
849MB
26.77
- int16
733.0
672MB
26.82
- int8
860.2
529MB
26.78
- int8 + vmap
1126.2
598MB
26.64
OPUS-MT model
Transformers 4.26.1 (with PyTorch 1.13.1)
147.3
2332MB
27.90
Marian 1.11.0
344.5
7605MB
27.93
- int16
330.2
5901MB
27.65
- int8
355.8
4763MB
27.27
CTranslate2 3.6.0
525.0
721MB
27.92
- int16
596.1
660MB
27.53
- int8
696.1
516MB
27.65
Executed with 4 threads on a c5.2xlarge Amazon EC2 instance equipped with an Intel(R) Xeon(R) Platinum 8275CL CPU.
GPU
Tokens per second
Max. GPU memory
Max. CPU memory
BLEU
OpenNMT-tf WMT14 model
OpenNMT-tf 2.31.0 (with TensorFlow 2.11.0)
1483.5
3031MB
3122MB
26.94
OpenNMT-py WMT14 model
OpenNMT-py 3.0.4 (with PyTorch 1.13.1)
1795.2
2973MB
3099MB
26.77
FasterTransformer 5.3
6979.0
2402MB
1131MB
26.77
- float16
8592.5
1360MB
1135MB
26.80
CTranslate2 3.6.0
6634.7
1261MB
953MB
26.77
- int8
8567.2
1005MB
807MB
26.85
- float16
10990.7
941MB
807MB
26.77
- int8 + float16
8725.4
813MB
800MB
26.83
OPUS-MT model
Transformers 4.26.1 (with PyTorch 1.13.1)
1022.9
4097MB
2109MB
27.90
Marian 1.11.0
3241.0
3381MB
2156MB
27.92
- float16
3962.4
3239MB
1976MB
27.94
CTranslate2 3.6.0
5876.4
1197MB
754MB
27.92
- int8
7521.9
1005MB
792MB
27.79
- float16
9296.7
909MB
814MB
27.90
- int8 + float16
8362.7
813MB
766MB
27.90
Executed with CUDA 11 on a g5.xlarge Amazon EC2 instance equipped with a NVIDIA A10G GPU (driver version: 510.47.03).
Contributing
CTranslate2 is a community-driven project. We welcome contributions of all kinds:
New Model Support: Help us implement more Transformer architectures.
Performance: Propose optimizations for CPU or GPU kernels.
Bug Reports: Open an issue if you find something not working as expected.
Documentation: Improve our guides or add new examples.
Check out our Contributing Guide to learn how to set up your development environment.
The most recent commit recorded on OpenNMT/CTranslate2 was 4 months ago, based on the GitHub push timestamp. The repository has 559 forks — one of the better signals of community interest.
How many stars does OpenNMT/CTranslate2 have?
OpenNMT/CTranslate2 has 4.8k GitHub stars — refresh the page for the live number, or check github.com/OpenNMT/CTranslate2. TopGit mirrors GitHub's count but does not claim minute-by-minute accuracy.
What language is OpenNMT/CTranslate2 written in?
OpenNMT/CTranslate2 is written primarily in C++. GitHub's language field is based on the largest share of bytes in the default branch.
What license does OpenNMT/CTranslate2 use?
OpenNMT/CTranslate2 is released under the MIT license. Always verify the LICENSE file directly on GitHub for the authoritative terms — license strings can be edited out of sync with a project's actual stance.
What topics is OpenNMT/CTranslate2 associated with?
The project maintains a homepage at https://opennmt.net/CTranslate2. The README tab on this page also usually contains screenshots and a quickstart.
Where do I read more about OpenNMT/CTranslate2?
This TopGit page is a snapshot — the READ ME tab shows the project's own README content (links stripped, images preserved). The GitHub repository at github.com/OpenNMT/CTranslate2 is the definitive source.
Read full README in the tab above.
Still deciding about CTranslate2?
One click hands the question to an AI along with this page — see what it says about CTranslate2.