google/sentencepiece is a open-source project on GitHub, written primarily in C++. It has 12.2k stars. Unsupervised text tokenizer for Neural Network-based text generation.
Snapshot summary built from the project's own GitHub metadata — there's no written TopGit review yet. The page will update automatically when a full review is published.
WHY NO REVIEW YET
TopGit writes full reviews for the most-starred, most-requested repositories. This page is a snapshot until then — see the READ ME tab for the original README in full.
SentencePiece is a fast, lightweight, and unsupervised text tokenizer and detokenizer designed for neural network-based text generation systems (such as Large Language Models) where the vocabulary size is fixed prior to training.
It implements subword units—including Byte-Pair-Encoding (BPE) [Sennrich et al.] and the unigram language model [Kudo.]—with the ability to train directly from raw sentences. By treating input text as a raw sequence of Unicode characters, SentencePiece enables a purely end-to-end, language-independent pipeline that completely eliminates the need for language-specific pre- or post-processing.
This is not an official Google product.
Quick Start (Python)
SentencePiece provides an easy-to-use Python module. Install it via pip:
pip install sentencepiece
Basic Example
Here is how to train a model, encode text into tokens/IDs, and decode them back to the original string:
import sentencepiece as spm
# 1. Train a model directly from a raw text file.
# (No pre-tokenization or language-specific preprocessing required!)
spm.SentencePieceTrainer.train(
input='data/botchan.txt',
model_prefix='m',
vocab_size=1000
)
# 2. Load the trained model.
sp = spm.SentencePieceProcessor(model_file='m.model')
# 3. Encode raw text into subword pieces (strings) or vocabulary IDs (integers).
text = "I saw a girl with a telescope."
pieces = sp.encode(text, return_type=str)
ids = sp.encode(text, return_type=int)
print(f"Pieces: {pieces}")
# Output: ['▁I', '▁saw', '▁a', '▁girl', '▁with', '▁a', '▁', 'te', 'le', 's', 'c', 'o', 'pe', '.']
print(f"IDs: {ids}")
# Output: [9, 459, 11, 939, 44, 11, 4, 142, 82, 8, 28, 21, 132, 6]
# 4. Decode IDs or pieces back into the original text.
# The reconstruction is completely lossless and reversible!
print(sp.decode(ids))
# Output: "I saw a girl with a telescope."
print(sp.decode(pieces))
# Output: "I saw a girl with a telescope."
Why SentencePiece?
1. Reversible & Lossless Tokenization (Whitespace as a Basic Symbol)
Traditional tokenizers drop whitespace information (e.g., treating Tokenize("World.") identically to Tokenize("World .")), making detokenization ambiguous and language-dependent.
SentencePiece treats the input text as a raw sequence of Unicode characters. It escapes whitespaces with a meta-symbol ▁ (U+2581) and includes it in the tokenization. This design ensures that detokenization is a simple, lossless string join operation, entirely independent of the language:
SentencePiece trains tokenization and detokenization models directly from raw sentences. It does not require language-specific pre-tokenizers (such as Moses, MeCab, or KyTea). This makes it highly effective for languages without explicit word boundaries, such as Chinese, Japanese, and Thai.
3. Subword Regularization & BPE-Dropout
To improve the robustness and accuracy of translation and language models, SentencePiece supports on-the-fly subword sampling during training. By sampling different segmentations for the same input text (Subword Regularization for Unigram, BPE-Dropout for BPE), it virtually augments your training data and makes the model more resilient to spelling variations and noise.
# Sample different segmentations on-the-fly
for _ in range(3):
print(sp.encode('New York', return_type=str, enable_sampling=True, alpha=0.1, nbest_size=-1))
# May output:
# ['▁', 'N', 'e', 'w', '▁York']
# ['▁New', '▁York']
# ['▁New', '▁Y', 'o', 'r', 'k']
4. Fast, Lightweight, and Self-Contained
Performance: Written in highly optimized C++. Segmentation speed is around 50,000 sentences per second, with a memory footprint of only ~6MB.
Self-Contained: The generated .model file contains the entire normalization rules, vocabulary mapping, and segmentation model. You are guaranteed to get the exact same tokenization results in any environment (C++, Python, Go, etc.) as long as you use the same model file.
Performance Benchmark (SentencePiece vs. Hugging Face Fast)
Benchmark Setup
Environment: 24-core CPU, Python 3.13.
Dataset: Balanced raw multilingual text from FLORES-200 (parallel sentences in English, Chinese, Japanese, and Thai; 11.29 MB, 60,720 lines). CJK and Thai texts are raw and do not contain artificial space delimiters.
Batch Request Size: The entire dataset (60,720 sentences) is fed as a single batch request (a single Python list[str]) in one call.
Metric: Encoding throughput in MB/s (higher is better).
1. Unigram Model: T5-base (32k vocab)
Tokenizer
1 Thread
2 Threads
4 Threads
8 Threads
16 Threads
24 Threads
SentencePiece
27.41
43.83
71.62
102.08
123.33
127.60
Hugging Face Fast
3.78
7.15
12.45
20.33
27.00
31.49
2. BPE Model: Gemma 3 (256k vocab)
Tokenizer
1 Thread
2 Threads
4 Threads
8 Threads
16 Threads
24 Threads
SentencePiece
7.44
12.82
23.03
36.66
48.65
52.43
Hugging Face Fast
3.66
6.37
10.45
15.54
21.05
20.48
Why performance does not scale linearly:
While the core tokenization (C++ or Rust) runs in parallel, the final step of converting the native results (C++ vector of vectors or Rust vector of vectors) into Python objects (list[list[int]] or list[Encoding]) is sequential and must be done on Python's main thread (GIL-locked). At high thread counts, this single-threaded serialization step becomes the dominant bottleneck, capping the scaling performance.
For the detailed analysis and single-thread reference comparison, see Performance Benchmark Details.
To run these benchmarks yourself, see the reproduction instructions and scripts.
SentencePiece Lite (Zero-Dependency C++20 Runtime)
For mobile apps, edge devices, on-device LLM inference, and high-throughput C++ serving environments, this repository includes SentencePiece Lite (lite/):
Zero Runtime Dependencies: Written in modern C++20 with zero third-party dependencies (no runtime dependency on Protobuf or Abseil).
Zero-Copy Memory-Mapped Startup: Operates directly on FlatBuffers binaries (.spm.fb) via mmap, achieving instantaneous cold-start with only 1.2 KB heap memory overhead regardless of vocabulary size.
Compact Binary Footprint: Core engine is under 2,000 lines of code, compiling to a ~45 KB stripped static library (libsentencepiece_lite.a).
Full Canonical Equivalence: 100% byte-for-byte normalization and token ID equivalence with canonical SentencePiece models (Unigram and BPE).
For details, benchmarks, and C++ integration recipes, see the SentencePiece Lite Documentation.
Documentation & Resources
For detailed guides, API references, and advanced usage, please refer to the following resources:
SentencePiece Lite Runtime Guide
Command Line Interface (CLI) Guide
Building with CMake
Building with Bazel
C++ API Reference
Python API Reference & Python Module Directory
Python Tokenizer Comparison Cheat Sheet
Performance Benchmark Details
Performance Benchmark Code & Reproduction Guide
Training Options Reference
Auto-Character Coverage & Byte-Fallback Guide
Sparse L1 Unigram Pruning Guide
Text Normalization & Custom Rules
Special Symbols & Control Tokens
Vocabulary Piece Constraints
Model Protobuf Schema
Docker Deployment Guide
License
SentencePiece is licensed under the Apache 2.0 License.
How active is development on google/sentencepiece?
The most recent commit recorded on google/sentencepiece was 6 days ago, based on the GitHub push timestamp. The repository has 1.4k forks — one of the better signals of community interest.
How many stars does google/sentencepiece have?
google/sentencepiece has 12.2k GitHub stars — refresh the page for the live number, or check github.com/google/sentencepiece. TopGit mirrors GitHub's count but does not claim minute-by-minute accuracy.
What language is google/sentencepiece written in?
google/sentencepiece is written primarily in C++. GitHub's language field is based on the largest share of bytes in the default branch.
What license does google/sentencepiece use?
google/sentencepiece is released under the Apache-2.0 license. Always verify the LICENSE file directly on GitHub for the authoritative terms — license strings can be edited out of sync with a project's actual stance.
What topics is google/sentencepiece associated with?
GitHub's repository topics for google/sentencepiece: "natural-language-processing", "neural-machine-translation", "word-segmentation". TopGit's editorial category is open-source.
Where can I see google/sentencepiece in action?
The project maintains a homepage at https://google.github.io/sentencepiece/. The README tab on this page also usually contains screenshots and a quickstart.
Where do I read more about google/sentencepiece?
This TopGit page is a snapshot — the READ ME tab shows the project's own README content (links stripped, images preserved). The GitHub repository at github.com/google/sentencepiece is the definitive source.
Read full README in the tab above.
Want a second opinion on sentencepiece?
Ask an AI that can read this page — one click and you get its take on sentencepiece.