Postmortem: An Elasticsearch 8.11 Query Bug That Returned Wrong Results for 1 Hour

This postmortem details a production incident where Elasticsearch 8.11.0 returned incorrect query results for 60 minutes, impacting customer-facing search functionality. We cover the timeline, root cause, remediation, and prevention measures.

Incident Summary

Date/Time: 2024-03-15, 14:00 – 15:00 UTC

Duration: 1 hour

Impact: ~12% of all search queries returned incomplete results; 3 customer-facing apps affected; ~2400 failed user searches

Root Cause: Regression in index sort optimization logic for bool filter clauses with range queries in Elasticsearch 8.11.0

Resolution: Rollback to 8.10.4, then upgrade to patched 8.11.1

Timeline

13:45 UTC: Production upgrade to Elasticsearch 8.11.0 completed across 3-node cluster, all health checks pass

14:00 UTC: First customer report of missing search results for recent log entries

14:05 UTC: On-call engineer confirms issue: queries for logs from the last 1 hour return 0 results,…