#llama

📰

Nemotron 3.5 Bolt on one GPU: what the 24 GB covers, the drafter flag and the llama.cpp trap

DEV Community: opensource·InferenceKey·about 18 hours ago

#devtoopensource #devto #dev #nvidia #drafter

A 74B vision agent model in 22 GB on one A40: 136 tokens/s with its drafter, 70 without, about 27 GB used. What the 24 GB leaves out.

15s

I tested Nvidia’s Nemotron 3.5 Bolt, and it changed what I thought my 16GB GPU could do

🖼️

0Read More

I tested Nvidia’s Nemotron 3.5 Bolt, and it changed what I thought my 16GB GPU could do

XDA ·Abhinav Raj·about 18 hours ago

#pnqmfe1G

#xdadevelopers #xda-developers #bolt #llama #model

View the full article

Create a free account to read full articles inline — no redirect to the original site.

Create account Log in

Can This Open-Source Model Beat AI Detectors? Let's Test It Locally - KDnuggets

🖼️

0Read More

Can This Open-Source Model Beat AI Detectors? Let's Test It Locally - KDnuggets

KDnuggets·Abid Ali Awan·about 18 hours ago

#zKGdpVGr

#kdnuggets #humanizer #model #llama #generated

Can a free, open-source AI humanizer beat paid tools? We tested a 12B model locally and compared the results against AI detectors to see if the hype is real.

15s

Choosing an Open-Source LLM for Our Waste-Sorting Assistant

🖼️

0Read More

Choosing an Open-Source LLM for Our Waste-Sorting Assistant

DEV Community: machinelearning·Axelle Toni·3 days ago

#q6yVwtjy

#devtomachinelearning #education #devto #dev #instruct

About our project Our project is an assistant that tells people which bin an item belongs in, and...

15s

2026 Llama 3 8B vs GPT-4o-mini Property Report Moderation Using JSON Schema

📰

0Read More

2026 Llama 3 8B vs GPT-4o-mini Property Report Moderation Using JSON Schema

DEV Community: typescript·VelvetDusk629047·4 days ago

#TuSmAjP0

#devtotypescript #devto #dev #report #classifier

2026 Llama 3 8B vs GPT-4o-mini Property Report Moderation Using JSON Schema TL;DR: Use...

15s

Why Docker Can’t Use Your Mac’s GPU — and How to Pool Apple Silicon Macs for LLMs Instead

📰

0Read More

Why Docker Can’t Use Your Mac’s GPU — and How to Pool Apple Silicon Macs for LLMs Instead

DEV Community: ansible·The Operator·4 days ago

#rR8ayNd6

#devtoansible #devto #dev #llama #model

You have three Macs gathering dust — a Mac Studio in the office, a MacBook Pro in your bag, and an M1...

15s

I decided my 10-year-old mini PC hadn’t suffered enough, so I made it run a local LLM

🖼️

0Read More

I decided my 10-year-old mini PC hadn’t suffered enough, so I made it run a local LLM

MakeUseOf ·Gregory Gibson·5 days ago

#Gs64vjjC

#makeuseof #model #server #llama #models

View the full article

Create a free account to read full articles inline — no redirect to the original site.

Create account Log in

Llama 3 8B vs GPT-4o-mini: Node.js Text Moderation JSON Schema, Latency 2026

📰

0Read More

Llama 3 8B vs GPT-4o-mini: Node.js Text Moderation JSON Schema, Latency 2026

DEV Community: node·JaggerBlack5781·5 days ago

#1eyRnFRp

#devtonodejs #devto #dev #model #decision

TL;DR: For a junior developer moderating messy logistics catalog descriptions, start with GPT-4o-mini...

15s

Gemma 4 on a 4 GB GPU in Plain PyTorch: From Out of Memory to Faster Than llama.cpp

📰

0Read More

Gemma 4 on a 4 GB GPU in Plain PyTorch: From Out of Memory to Faster Than llama.cpp

DEV Community: pytorch·xbill·5 days ago

#mal3JuAW

#devtopytorch #devto #dev #pytorch #llama

On a 4 GB GTX 1650 Ti, Gemma 4 E2B does not fit in 16-bit, and Transformers' own 4-bit path runs out of memory on its first forward pass. Kept in Google's QAT 4-bit end to end, with a few Triton kernels and a CUDA graph, plain PyTorch decodes at 88 tok/s,…

15s

📰

0Read More

Why Docker Can’t Use Your Mac’s GPU — and How to Pool Apple Silicon Macs for LLMs Instead

DEV Community: ansible·lucanu·6 days ago

#ZpjJwt3o

#devtoansible #devto #dev #llama #model

You have three Macs gathering dust — a Mac Studio in the office, a MacBook Pro in your bag, and an M1...

15s

I changed one llama.cpp flag, and my 8GB laptop GPU now runs a 35B model faster than I can read

🖼️

0Read More

I changed one llama.cpp flag, and my 8GB laptop GPU now runs a 35B model faster than I can read

XDA ·Abhinav Raj·7 days ago

#7VxGxzaz

#xdadevelopers #xda-developers #llama #model #models

View the full article

Create a free account to read full articles inline — no redirect to the original site.

Create account Log in

📰

0Read More

Building a Llama.cpp Container Image

DEV Community: docker·Sanskriti Harmukh·7 days ago

#EdmXQzAU

#devtodocker #devto #dev #llama #enter

Llama.cpp is a high-performance inference platform designed for Large Language Models (LLMs) like...

15s

39 screenshots and not one useful name, so a local Gemma names them for me

🖼️

0Read More

39 screenshots and not one useful name, so a local Gemma names them for me

DEV Community: go·Om Lahore·8 days ago

#G2FvpwU8

#devtogo #devto #dev #screenshots #name

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady ...

15s

🖼️

362Read More

llama.app - Official home for llama.cpp

Hacker News·3 months ago

#Ax4GHaGn

#llama #lives #computer #open #photo

View the full article

Create a free account to read full articles inline — no redirect to the original site.

Create account Log in

Petals – Run LLMs at home, BitTorrent-style

🖼️

0Read More

Petals – Run LLMs at home, BitTorrent-style

Hacker News·3 months ago

#pfO01z80

#petals #llama #falcon #fine #part

Run large language models at home, BitTorrent‑style Generate text with Llama 3.1 (up to 405B), Mixtral (8x22B), Falcon (40B+) or BLOOM (176B) and fine‑tune them for your tasks — us

15s

GitHub - marcelroed/gigatoken: Language model tokenization at GB/s

🖼️

0Read More

GitHub - marcelroed/gigatoken: Language model tokenization at GB/s

Hacker News·3 months ago

#LrKukuZn

#github #gigatoken #llama #tokenizers #tokenizer

Language model tokenization at GB/s. Contribute to marcelroed/gigatoken development by creating an account on GitHub.

15s

How to Setup a Local Coding Agent on macOS

🖼️

475Read More

How to Setup a Local Coding Agent on macOS

Hacker News·5 months ago

#dEOSDGqk

#ikyle #gguf #gemma #llama #models

Running Gemma 4 26B-A4B and Qwen3.6 35B-A3B locally with llama.cpp, MTP speculative decoding, multimodal support, and PI as a coding agent.

15s

How to Deploy Llama 3.2 with TensorRT-LLM + Quantization on a $14/Month DigitalOcean GPU Droplet: 3x Faster Inference at 1/95th Claude Cost

🖼️

0Read More

How to Deploy Llama 3.2 with TensorRT-LLM + Quantization on a $14/Month DigitalOcean GPU Droplet: 3x Faster Inference at 1/95th Claude Cost

DEV Community: webdev·RamosAI·5 months ago

#5zhYpj9b

#dev #llama #tensorrt #import #article

⚡ Deploy this in under 10 minutes Get $200 free: https://m.do.co/c/9fa609b86a0e ($5/month...

15s

I replaced cloud LLMs with local models running off a Proxmox LXC, and the performance trade-off was worth it

🖼️

0Read More

I replaced cloud LLMs with local models running off a Proxmox LXC, and the performance trade-off was worth it

XDA ·Ayush Pande·5 months ago

#yknCPE4P

#xda-developers #models #llama #server #llms

View the full article

Create a free account to read full articles inline — no redirect to the original site.

Create account Log in

I'm 15, Built My First Real Project in 4 Days, and Put It on Gumroad

🖼️

0Read More

I'm 15, Built My First Real Project in 4 Days, and Put It on Gumroad

DEV Community·Hiroto Yoshida·5 months ago

#luTb9cJn

#dev #energy #groq #real #next

I'm 15 and Built an AI Energy Dashboard with Next.js 15 + Groq Hey Dev.to! 👋 I'm a...

15s

Menu

Nemotron 3.5 Bolt on one GPU: what the 24 GB covers, the drafter flag and the llama.cpp trap

I tested Nvidia’s Nemotron 3.5 Bolt, and it changed what I thought my 16GB GPU could do

Can This Open-Source Model Beat AI Detectors? Let's Test It Locally - KDnuggets

Choosing an Open-Source LLM for Our Waste-Sorting Assistant

2026 Llama 3 8B vs GPT-4o-mini Property Report Moderation Using JSON Schema

Why Docker Can’t Use Your Mac’s GPU — and How to Pool Apple Silicon Macs for LLMs Instead

I decided my 10-year-old mini PC hadn’t suffered enough, so I made it run a local LLM

Llama 3 8B vs GPT-4o-mini: Node.js Text Moderation JSON Schema, Latency 2026

Gemma 4 on a 4 GB GPU in Plain PyTorch: From Out of Memory to Faster Than llama.cpp

Why Docker Can’t Use Your Mac’s GPU — and How to Pool Apple Silicon Macs for LLMs Instead

I changed one llama.cpp flag, and my 8GB laptop GPU now runs a 35B model faster than I can read

Building a Llama.cpp Container Image

39 screenshots and not one useful name, so a local Gemma names them for me

llama.app - Official home for llama.cpp

Petals – Run LLMs at home, BitTorrent-style

GitHub - marcelroed/gigatoken: Language model tokenization at GB/s

How to Setup a Local Coding Agent on macOS

How to Deploy Llama 3.2 with TensorRT-LLM + Quantization on a $14/Month DigitalOcean GPU Droplet: 3x Faster Inference at 1/95th Claude Cost

I replaced cloud LLMs with local models running off a Proxmox LXC, and the performance trade-off was worth it

I'm 15, Built My First Real Project in 4 Days, and Put It on Gumroad