Blue-Green vs. Canary vs. Rolling Deployments: $460 Million in 45 Minutes and 8.5 Million Machines at Once
On August 1, 2012, the trading firm Knight Capital rolled out new software to eight production servers. A technician copied the new code to seven of them and missed the eighth, and no second person reviewed the deployment. The new code also reused an old feature flag, so on the server still running the old build, that flag woke up a long-dead routine called Power Peg. In the first 45 minutes after the market opened, the firm's router sent more than 4 million orders into the market while trying to fill just 212 customer orders, trading over 397 million shares across 154 stocks. The SEC put the resulting loss at more than $460 million (Knight's own accounting said about $440 million), and the firm was fined $12 million.…