Menu

Tip: you can grab single files out of big dataset zips without downloading the whole zip

#datasets#redditdatasets#files#coco#lots
Reading 0:00
15s threshold

A lot of datasets only come as one giant zip (COCO, lots of Zenodo and Hugging Face uploads). If the server supports partial downloads, and most of these do, you can read just the zip's file list and download only the files or folders you need.

On Linux or in a notebook, the Python packages remotezip and unzip-http do this.

In the browser, I made a free Chrome extension for it (LinkUnzip). Right-click the zip link, tick the files, and only those get downloaded. You can also preview images, CSVs and PDFs inside the zip first. One photo out of COCO's 18 GB train2017 zip cost about 12 MB, mostly the file list.

One catch: Zenodo slows you down if you pick lots of scattered files, so pick whole folders there.

https://linkunzip.app/

Does anyone know other big hosts where this doesn't work? Google Drive and Kaggle links don't, since they're share pages, not files.

submitted by /u/External-Drive6208
[link] [comments]