The Titanic effect in practice

Software developers have a term for it — the “Titanic effect”: the bigger the system, the harder it is to find and shore up its weak points; and the safer that system looks, the more terrifying the consequences when it fails.

Cloudflare is the most widely used reverse-proxy service by number of websites. It speeds up and protects websites and applications and provides traffic analytics.

Yesterday, a database-permissions change caused a Bot Management feature file to double in size, triggering widespread failures across Cloudflare’s network. Customer impact began around 11:28 UTC; core traffic was largely flowing normally by 14:30, and all systems were back to normal by 17:06.

Don’t fall into the trap of thinking your product is reliable simply because it looks big and solid. Analyze incidents and apply what they teach you to your own product. There is always something useful to learn.