🖼️00GitHub - marcelroed/gigatoken: Language model tokenization at GB/sHacker News·3 months ago#LrKukuZn#github#gigatoken#llama#tokenizers#tokenizer#qwen+4 more🧰Tag tools✨Add tagLanguage model tokenization at GB/s. Contribute to marcelroed/gigatoken development by creating an account on GitHub.15s0Read later0Read More
📰00Using hf tokenizers in RustDEV Community·Wayne·6 months ago#1c5EUsx1#rust#machinelearning#nlp#huggingface#tokenizer#fullscreen+4 more🧰Tag tools✨Add tagMaster Rust tokenizers with Hugging Face's powerful library. Learn to implement text tokenization, encoding/decoding, and work with pretrained models like GPT-2, BERT, and Llama for NLP applications. Uses the from_pretrained method15s0Read later0Read More
