Menu

Serving a heavy model without blowing up the server bill
📰

Serving a heavy model without blowing up the server bill

DEV Community: fastapi·Amine Affif·5 days ago
#lxhIEdW2
#devtofastapi#devto#dev#fastapi#model
Reading 0:00
15s threshold

An AI model is heavy and hungry. Served carelessly, it blows up response times and the server bill. On an audio-analysis SaaS that I designed and shipped solo, it had to stay fast, secure and economically viable.

The hosting decision. I served the model on a server I run, with NGINX and automatic file cleanup, rather than on usage-billed infrastructure. It means more deployment work, but the bill no longer follows the number of analyses. It can be forecast.

On the code side, two rules matter for a service like this.

1. Load the model once, at startup. Not on every request. With FastAPI, that's what the lifespan parameter is for. The application doesn't receive requests until startup is finished.

from contextlib import asynccontextmanager from fastapi import FastAPI

models = {}

@asynccontextmanager async def lifespan ( app : FastAPI ): models [ " main " ] = load_model () # slow: done once yield models .…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.