Menu

1 / 2

The History of Expanso: It's About the Network (It Always Has Been)

DEV Community·David Aronchick·7 months ago
#dVXeFzrq
#expanso#datalocality#network#across#jobs
Reading 0:00
15s threshold

Welcome to our series on the history of Expanso.

The Network is the Problem: How Expanso Began

The first time the scope of the problem facing tech was super clear, I was at Protocol Labs . We had IPFS storing and serving data at exabyte scale. The storage worked. The catalog worked. The network did not. Every user asked for the same thing: "Just move the data here so I can process it." We tried. The math killed us.

Do the simple version. If you need to send 1 GB to 100 nodes in one second, you need 100 GB/s of aggregate throughput. Now scale the dataset and the fan-out. You are into tens to hundreds of terabytes per second. Inside a rack, that is hard. Across a building or a continent, it is fantasy. And the data did not sit still. New events arrived every second. Every plan that started with "first, copy it all" ended in a queue.

I had seen the same pattern before. At Google with GKE and early Kubernetes users.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.