A guide to running open-source LLMs locally on your machine.
Why Local AI?
- Privacy
- No API costs
- Works offline
Quick Setup
pip install llama-cpp-python
wget https://huggingface.co/TheBloke/Mistral-7B-GGUF/resolve/main/mistral-7b-instruct.Q4_K_M.gguf
from llama_cpp import Llama
llm = Llama(model_path="./mistral-7b-instruct.Q4_K_M.gguf")
output = llm("Q: Hello! A:", max_tokens=64)
print(output["choices"][0]["text"])