jamesob's guide to running SOTA LLMs locally

Note: nothing in this README aside from the tables was written by AI.

Have $2k burning a hole in your pocket and want some local, state-of-the-art machine intelligence?

How about $40k?

If Dario and Altman are giving you heartburn (they should be), read on to figure out how to run this new kind of computing locally.

In this repo you'll find

the hardware I use to run SOTA locally, why I bought what and little-known secrets for configuring it,

how I run speech-to-text (STT) locally,

ready-to-run configuration for running models I think are good within Docker containers.

Contents

Section TL;DR

How much are you willing to spend?…