Menu

Show HN: 53ON AI Server – OpenAI-compatible server for AMD Ryzen AI Max
📰

Show HN: 53ON AI Server – OpenAI-compatible server for AMD Ryzen AI Max

DEV Community: node·Enrique Velasquez·about 1 hour ago
#oE3RXYAQ
#devtonodejs#devto#dev#memory#llama
Reading 0:00
15s threshold

Enrique Velasquez

Hi HN. I run my own AI at home on a Ryzen AI Max+ 395 mini PC with 128 GB of unified memory, and this is the server it talks to. It's a single Node.js process with no dependencies, on top of llama.cpp. It speaks the OpenAI and Anthropic APIs, and it has a panel to load models, give each client an API key with a token quota, and relay cloud providers when a local model isn't enough.

We also publish our own llama.cpp build. The main change is that the KV cache takes memory as tokens arrive instead of reserving it all at load (sparse buffers on Vulkan). A 1B model with a 1M-token window starts with 0.86 GiB instead of 8.24 GiB.

It's MIT. I've only tested it on Strix Halo with Vulkan (RADV) on Ubuntu 24.04. I built it with a lot of help from Claude. Feedback and bug reports are very welcome.