GigaToken: ~1000x faster Language model tokenization

 (github.com)

199 points | by syrusakbary 3 hours ago

35 comments