meta-pytorch/tokenizers

C++ implementations for various tokenizers (sentencepiece, tiktoken etc).

by Meta PyTorch C++ BSD-3-Clause licence 42 forks Created Nov 2024 View on GitHub

Stars
50
<1% of meta-pytorch's stars
Rank
#838,938
of 852,180 repos, top 99%
In C++
#44,686
of 45,386 repos, top 99%

Among 852,180 repos

Compare with

40–63 stars: 143,595 repos, including tokenizers

Stars along the bottom, and repos in each band on a log scale

Quick stats

Next rank
+1 star to reach #826,147
Next milestone
+7 stars to reach the top 90%
Yearly average
25 stars a year since Nov 2024

Stars counted Recounted daily Next count around