Agent Skills: huggingface-tokenizers
Fast tokenizers optimized for research and production. Rust-based implementation tokenizes 1GB in <20 seconds. Supports BPE, WordPiece, and Unigram algorithms. Train custom vocabularies, track alignments, handle padding/truncation. Integrates seamlessly with transformers. Use when you need high-performance tokenization or custom tokenizer training.
[TokenizationHuggingFaceBPEWordPieceUnigramFastTokenizationRustCustomTokenizerAlignmentTrackingProduction]
ID: PoorRican/dotfiles/huggingface-tokenizers
Install this agent skill to your local
Skill Files
The files that make up huggingface-tokenizers.
More agent skills
- android-clean-architectureAndroid と Kotlin Multiplatform プロジェクトのクリーンアーキテクチャパターン — モジュール構造、依存関係ルール、UseCase、Repository、データ層パターン。
- epic-permissionsGuide on RBAC system and permissions for Epic Stack
- extension-creatorCreate AiderDesk extensions by setting up extension files, defining metadata, implementing Extension interface methods, and updating documentation. Use when building a new extension, creating extension commands, tools, or event handlers.
- codebase-designShared vocabulary for designing deep modules. Use when the user wants to design or improve a module's interface, find deepening opportunities, decide where a seam goes, make code more testable or AI-navigable, or when another skill needs the deep-module vocabulary.
- ai-agents-architectExpert in designing and building autonomous AI agents. Masters tool use, memory systems, planning strategies, and multi-agent orchestration. Use when: build agent, AI agent, autonomous agent, tool use, function calling.
- n8n:reproduce-bugReproduce a bug from a Linear ticket with a failing test. Expects the full ticket context (title, description, comments) to be provided as input.