huggingface-tokenizers
HuggingFace Tokenizers
Fast, production-ready tokenization - Rust-powered, Python API.
When to Use
- High-performance tokenization (<20s per GB)
- Train custom tokenizers from scratch
- Track token-to-text alignment
- Production NLP pipelines
- Need BPE, WordPiece, or Unigram tokenization
Quick Start
from tokenizers import Tokenizer
More from eyadsibai/ltk
document-processing
Use when working with "PDF", "Excel", "Word", "PowerPoint", "XLSX", "DOCX", "PPTX", "spreadsheets", "presentations", "extract text", "merge documents", "convert documents", or asking about "office document manipulation
892file-organization
Use when "organizing files", "cleaning up folders", "finding duplicates", "structuring directories", or asking about "Downloads cleanup", "folder structure", "file management
336literature-review
Use when "literature review", "research synthesis", "systematic review", "academic search", or asking about "find papers", "cite sources", "research gaps", "meta-analysis", "bibliography
226resume-generator
Use when "tailoring resume", "job application", "CV customization", "ATS optimization", or asking about "resume writing", "career transition", "job description matching
138content-writing
Use when "writing articles", "blog posts", "content creation", "research writing", "technical writing", or asking about "outlining", "citations", "improving hooks", "writing feedback
120agent-browser
Use when automating browser interactions via CLI, filling forms, taking screenshots, scraping pages, or asking about "agent-browser", "browser automation", "headless browser", "web scraping", "form filling", "Vercel browser
103