🖼️00Read later0Read MoreGitHub - marcelroed/gigatoken: Language model tokenization at GB/sHacker News·3 months ago#LrKukuZn#github#gigatoken#llama#tokenizers#tokenizer+5 more🧰Tag tools✨Add tagLanguage model tokenization at GB/s. Contribute to marcelroed/gigatoken development by creating an account on GitHub.15s
📰00Read later0Read MoreUsing hf tokenizers in RustDEV Community·Wayne·6 months ago#1c5EUsx1#rust#machinelearning#nlp#huggingface#tokenizer+4 more🧰Tag tools✨Add tagMaster Rust tokenizers with Hugging Face's powerful library. Learn to implement text tokenization, encoding/decoding, and work with pretrained models like GPT-2, BERT, and Llama for NLP applications. Uses the from_pretrained method15s
