There is a very common way to solve performance problems in software. Buy a bigger machine. If that does not work, add more RAM. Then more CPU. Then another machine. Eventually somebody says βcluster,β and now you have a new architecture, a monitoring stack, distributed execution, and several new job openings.
Sometimes that is exactly the right answer. But sometimes we are using infrastructure to compensate for decisions that were made much earlier β in the way data is represented, moved, allocated, calculated, and eventually shown to a user.
We have been building analytical platforms since 1998. Megaladata is the third generation of that technology. Over that time, one lesson has repeated itself often enough that we no longer think of performance as a feature .
You cannot take a fundamentally slow system, replace one Join implementation, install a better allocator, add multithreading, and suddenly call it a high-performance platform.
Performance is an architectural property .β¦