Agent Skills: tokenization-huggingface-tokenizers

Fast tokenizers optimized for research and production. Rust-based implementation tokenizes 1GB in <20 seconds. Supports BPE, WordPiece, and Unigram algorithms. Train custom vocabularies, track alignments, handle padding/truncation. Integrates seamlessly with transformers. Use when you need high-performance tokenization or custom tokenizer training.

ID: davila7/claude-code-templates/tokenization-huggingface-tokenizers

Install this agent skill to your local

npx skills add davila7/claude-code-templates --skill tokenization-huggingface-tokenizers

Skill Files

The files that make up tokenization-huggingface-tokenizers.

Download Skill

More agent skills