OpenNMT/Tokenizer

Fast and customizable text tokenization library with BPE and SentencePiece support

by OpenNMT C++ MIT licence 83 forks Created Feb 2017 View on GitHub

Stars
344
2% of OpenNMT's stars
Rank
#174,464
of 847,086 repos, top 21%
In C++
#9,277
of 45,203 repos, top 21%

Among 847,086 repos

Compare with

250–400 stars: 77,108 repos, including Tokenizer

Stars along the bottom, and repos in each band on a log scale

Quick stats

Next rank
+1 star to reach #173,996
Next milestone
+12 stars to reach the top 20%
Popular
One of 488,402 repos with 100+ stars
Yearly average
35 stars a year since Feb 2017

Stars counted Recounted daily Next count around