Điểm qua huggingface/tokenizers: 11.2k sao trên GitHub, viết chủ yếu bằng Rust, thuộc nhóm AI Tools. 💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
Tóm tắt dựng từ metadata GitHub của chính dự án — chưa có bài review TopGit. Trang sẽ tự động cập nhật khi bài review đầy đủ được xuất bản.
VÌ SAO CHƯA CÓ REVIEW
TopGit viết bài đầy đủ cho repo có nhiều sao nhất và được yêu cầu nhiều nhất. Trang này là snapshot trong thời gian chờ — xem README gốc ở tab READ ME.
The SOTA tokenization library on all languages, all models, all hardwares.
Our goal with tokenizers is to develop and maintain the industry's standard tokenization engine, making it the defacto place for everyone to contribute to the whole ecosystem.
Release candidate: v1.0.0.rc.x
As we are switching from 0.23 to v1.0.0, the current library does not ship all features. If you are worried, check the v1 features section for more details of what we are bringing back.
We will publish a blog about what breaking changes we introduced. We strived to keep them as small as possible.
Installation
pip install --pre tokenizers
For Rust, add the crate to your Cargo.toml:
[dependencies]
tokenizers = "1.0.0"
Training (train) and downloading from the Hub (http) are opt-in features. The crate's
Using the crate section lists every feature and how to
build a smaller binary.
Tokenizer holds the tokenizer.json config and can be edited (e.g. tokenizer.truncation = ...);
compile() builds the CompiledTokenizer that encodes and decodes.
Performances
For more details, checkout https://huggingface-tokenizers-v1.static.hf.space/index.html.
Bindings
status
Rust
✅ reference implementation
Python
✅
Node.js
✅
WebAssembly
✅ tokenizers-wasm on npm
C / C++ / Java / Go/ etc
🚧 planned
Ruby
community, external repo
v1.0.0 roadmap:
As we work toward v1, we are gonna bring back the entire python API that allows transformers' style of interacting with a tokenizer object:
TODOS:
Python Bindings
Bring training back: this should be easy, but we want the perfs gains to come as well.
Batch encoding API with zero-copy views on encoded ids
dlpack API for torch / JAX / numpy interop.
Composition: edit components of the pipeline (tokenizer.normalizer = [...]). The editable
Tokenizer spec and compile() have landed; per-component handles are next.
tokenizer.save to persist a modified or trained tokenizer on disk.
role_to_token override.
Bring back offset output: this should be useful for people training.
Windowing / stride in encode.
Rust core
bitnorm: more performance from optimized normalization, as this will bring breaking changes we want to make sure it gets to v1.
Unroll more regex: we need to cover the most used ones, we might have missed one or two.
Other
C & C++ bindings for loading, encode and decode APIs
After 1.0.0
Improve / rewrite the bitcannon crate
Java, Go, Swift... bindings, supported by the C bindings
tk-devices — exploratory GPU encoding and batch decoding that keeps text and ids on the
device: upload the vocabulary once, compute output positions in parallel, gather the bytes on the
GPU. An optional component aimed at large batches, subject to prototyping and measurement.
Crate details
Crate size
Sub-crates
tk-encode — the inference engine
The models (BPE, Unigram, WordPiece, WordLevel) and the full pipeline: Normalizer,
PreTokenizer, Model, PostProcessor, Decoder.
Why separate: you don't need all features to serve a model in production settings. Splitting it from training
means a serving binary never links a trainer, a corpus reader, legacy converters, etc.
bitcannon — bitstream pre-tokenization
Unicode tag classification plus pre-tokenization as a bitstream program. Follows Interleaved Bitstream Execution for Multi-Pattern Regex Matching on GPUs
(MICRO'25, 10.1145/3725843.3756052) but we added our own algorithm to make sure we don't need a parabix engine.
Characters are first classified into "tags", which use a sparse table generated when compiling bitcannon. This means there are 0 depencencies to the heavy unicode tables.
Ships byte-exact grammars for gpt2 / ByteLevel, cl100k, o200k, tekken, deepseek and kimi-k2.
The classes or tags each have a high and low nibble (encoded on a u8), and carry refinement when needed. Using u8 tags allows us to leverage less bitsream, meaning have a simpler / faster pre tokenization.
Why separate: we just thought it might be useful to some people someday. As far as we know, this implementation is the fastest for every single pretokenizer regex on CPU.
tk-serialize — the reader
from_json_file turns a canonical tokenizer.json into a PipelineTokenizer.
Why separate: this just makes it optional for some usecases!
tk-convert — the upgrade pass
canonicalize_file rewrites legacy a tokenizer.json written by any older version of this library into
the canonical form the reader accepts. A pure JSON→JSON rewrite; it depends on nothing but
std::path and serde_json.
Why separate: every config ever published stays readable without the runtime carrying a decade
of compatibility branches. But, the legacy code amounted to quite a lot of the final crate size. Disabling it allows on device builds to be smaller.
tk-train — the training half
The Trainer trait, every concrete *Trainer, TrainerWrapper and the Trainable extension.
The tokenizers crate re-exports it behind the train feature, which is off by default. It
trains only the models whose feature is on (bpe, unigram, wordpiece, wordlevel). Add
esaxx_fast for the C++ suffix array behind Unigram training.
Why separate: training is a batch job on a workstation; inference is a hot loop in a server.
They have opposite constraints, so they get opposite dependency budgets.
bitmap_gen — dev-only table generator
cargo run -p bitmap_gen regenerates bitcannon's committed classify tables from unicode-properties, emitting one Atom tag per codepoint.
Why separate — this is what buys the zero Unicode dependency.unicode-properties is a
dependency of this crate and of nothing else: the tables it produces are checked into
bitcannon/src/classify/atom_tables.rs as ordinary source, so the Unicode data is resolved once, at
development time, by a crate that is never linked into anything that ships and is never published.
No build script either — bitcannon compiles with no code generation step, and a release binary
contains the tags without containing a Unicode crate to derive them. The release workflow re-runs
the generator and fails if the committed table differs, so "baked" cannot silently mean "stale".
tokenizers — umbrella
A thin re-export so existing tokenizers::… paths keep working. Depend on this one unless you know
you want less.
Hardware
No SIMD is required. Every kernel has a portable path that is always compiled and is the
byte-exact test oracle for its vectorised siblings, so correctness never depends on a kernel being
present — only throughput does.
Which ops are hardware-adapted
op
aarch64
x86_64
wasm32
portable fallback
Unicode atom classify
NEON (baseline)
AVX-512 VBMI → SSE4.1/SSSE3, runtime-detected
SIMD128
classify_scalar
bitstream block build
NEON
SSE/AVX
—
build_block_scalar
literal & added-token scan
NEON
x86_64
—
scalar
vocab bucket nibble match
NEON
—
—
scalar
aarch64 selects at compile time (NEON is baseline). x86_64 dispatches at runtime via
is_x86_feature_detected!, so one binary covers every x86_64 CPU since 2008. wasm32 needs the
simd128 target feature; without it, the scalar walk.
Acknowledgements
This work stands on the shoulders of a lot of open source projects. Thanks to all of them for their great work.
gigatoken,
tiktoken, kitoken,
tokie, fastokens,
wordchipper and
ai-tokenizer each pushed on what a fast tokenizer can
be; we read that work, and several of the ideas above reached us because another project showed
they were worth trying.
Trang TopGit này là một snapshot — tab "Readme" hiển thị nguyên văn README của repo (đã bỏ link, giữ ảnh). Repo GitHub ở github.com/huggingface/tokenizers là nguồn chính thức.
huggingface/tokenizers có những chủ đề gì?
GitHub topics của huggingface/tokenizers: "bert", "gpt", "language-model", "natural-language-processing", "natural-language-understanding", "nlp", "transformers". TopGit xếp repo vào nhóm AI Tools.
huggingface/tokenizers có phải mã nguồn mở không?
Có — huggingface/tokenizers phát hành theo license Apache-2.0, nghĩa là mã nguồn mở để đọc, fork và (tùy license) tái sử dụng. Mã: github.com/huggingface/tokenizers.
huggingface/tokenizers có trang demo không?
Dự án có trang chủ ở https://huggingface.co/docs/tokenizers. Tab "Readme" ở trang này thường có ảnh chụp và hướng dẫn bắt đầu nhanh.
huggingface/tokenizers còn đang phát triển không?
Commit gần nhất trên huggingface/tokenizers là 3 tháng trước (theo timestamp GitHub). Repo có 1.3k fork — một chỉ báo về mức độ quan tâm của cộng đồng.
huggingface/tokenizers dùng license gì?
huggingface/tokenizers phát hành theo license Apache-2.0. Nên mở file LICENSE trên GitHub để xác nhận — license metadata đôi khi lệch với thực tế dự án.
Đọc đầy đủ README ở tab phía trên.
tokenizers có đáng để bạn bỏ thời gian?
ChatGPT, Claude và Perplexity đều đọc được trang này. Hỏi thử xem họ nghĩ gì về tokenizers.