SentencePiece with ARM64 SIMD
Implementing Google’s SentencePiece in C with aggressive optimizations
Tokenization
Tiktoken with ARM64 SIMD
Implementing Tiktoken in C, with SIMD
Implementing Google’s SentencePiece in C with aggressive optimizations
Implementing Tiktoken in C, with SIMD