Menu

What models/datasets to archive from Hugging Face?

Reading 0:00
15s threshold

Looking to archive ~40TB of new models.

I already have the arXiv dump and Sci-Hub on a separate server, Sci-Hub is ~90TB and arXiv is ~16TB. Off the top of my head I think these are probably the two most important datasets to archive..

On my list so far still to archive is:

  • K3.1 Pro
  • GLM-5.3 (+Flash)
  • DeepSeek V4.1 Flash 1017
  • Qwen3.8 (will replace with Qwen4 when it is out)
  • Gemma 4
  • LFM3 series

Am I missing anything? Is there anything else I should grab? Thinking about making the archive public at some point.

(I've released abliterations on many of the recent large models so I have copies/diffs of most on a separate server.)

submitted by /u/archiv1st_
[link] [comments]