Agent Skills: tokenization-sentencepiece

Language-independent tokenizer treating text as raw Unicode. Supports BPE and Unigram algorithms. Fast (50k sentences/sec), lightweight (6MB memory), deterministic vocabulary. Used by T5, ALBERT, XLNet, mBART. Train on raw text without pre-tokenization. Use when you need multilingual support, CJK languages, or reproducible tokenization.

ID: davila7/claude-code-templates/tokenization-sentencepiece

Install this agent skill to your local

npx skills add davila7/claude-code-templates --skill tokenization-sentencepiece

Skill Files

The files that make up tokenization-sentencepiece.

Download Skill

More agent skills