Agent Skills: tokenization-huggingface-tokenizers
Fast tokenizers optimized for research and production. Rust-based implementation tokenizes 1GB in <20 seconds. Supports BPE, WordPiece, and Unigram algorithms. Train custom vocabularies, track alignments, handle padding/truncation. Integrates seamlessly with transformers. Use when you need high-performance tokenization or custom tokenizer training.
[TokenizationHuggingFaceBPEWordPieceUnigramFastTokenizationRustCustomTokenizerAlignmentTrackingProduction]
ID: davila7/claude-code-templates/tokenization-huggingface-tokenizers
33,1943,891
Install this agent skill to your local
Skill Files
The files that make up tokenization-huggingface-tokenizers.
More agent skills
- bash-linuxBash/Linux terminal patterns. Critical commands, piping, error handling, scripting. Use when working on macOS or Linux systems.
- 3d-games3D game development principles. Rendering, shaders, physics, cameras.
- datadog-cliDatadog CLI for searching logs, querying metrics, tracing requests, and managing dashboards. Use this when debugging production issues or working with Datadog observability.
- aws-serverlessSpecialized skill for building production-ready serverless applications on AWS. Covers Lambda functions, API Gateway, DynamoDB, SQS/SNS event-driven patterns, SAM/CDK deployment, and cold start optimization.
- copy-editingWhen the user wants to edit, review, or improve existing marketing copy. Also use when the user mentions 'edit this copy,' 'review my copy,' 'copy feedback,' 'proofread,' 'polish this,' 'make this better,' or 'copy sweep.' This skill provides a systematic approach to editing marketing copy through multiple focused passes.
- database-designDatabase design principles and decision-making. Schema design, indexing strategy, ORM selection, serverless databases.