Menu

Part Size, Not Protocol: Direct Browser Uploads of Large Training Files to Object Storage
πŸ“°

Part Size, Not Protocol: Direct Browser Uploads of Large Training Files to Object Storage

DEV Community: goΒ·BarnabyVance6852Β·6 days ago
#xRaVoIRg
#devtogo#devto#dev#parts#size
Reading 0:00
15s threshold

The constraint that settles this argument is not the protocol, it is the size distribution of what you accept. If the median training artifact your customers push is a 40 MB checkpoint and the p99 is a 60 GB dataset shard, the upload path has to survive the p99 without taxing the median β€” which pushes you toward presigned multipart parts sent direct from the browser to object storage, backed by a signing service that does nothing except sign. Pick the part size from the largest object you will ever accept, publish that number to the client, and treat it as a capacity decision rather than a tuning knob.

Everything below follows from that one number.

The scenario I'm writing against is a SaaS platform team with five engineers, a retention policy that auditors will eventually read back to us, and a Node.js API that has no business moving 60 GB through its own heap. Artifacts arrive from a browser because that is where the data scientists are.…

Continue reading β€” create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.