Menu

1 / 2

Pickle-only uploads to Hugging Face doubled since 2024

DEV Community: pytorch·Iain·3 days ago
#YEJNTc8E
#devtopytorch#devto#dev#pickle#load
Reading 0:00
15s threshold

Hugging Face is getting more pickle files, not fewer. In the second half of 2024, about one in seventeen people uploading model weights each month put up nothing but pickles. Since March it's been about one in eight.

Pickle-only share of new Hugging Face models ( the data )

Each uploader counted once a month. Pickle only means .pt , .pth , .ckpt or .pkl with no

safetensors, GGUF or ONNX alongside, ignoring trainer leftovers. Course and swarm uploads left

out.

That matters because a pickle runs code when you load it, and safetensors doesn't. Safetensors is what transformers writes, and the old pickles on the Hub are meant to be leftovers from before it caught on.

The numbers come from hub-stats , a daily dump of every public repo on the Hub with its file list, as of 21 October. I counted uploaders rather than repos, because one person can push a thousand repos in an afternoon.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.