Menu

1 / 2

15-Minute RPO to 7-Year Retention: Elasticsearch Backups on S3 with SLM (No Scripts, No Keys)

DEV Community: aws·Pranay Ravi·18 days ago
#wXqK6Vmh
#devtoaws#devto#dev#policy#profile
Reading 0:00
15s threshold

Four Elasticsearch SLM policies (15-min, hourly, monthly, yearly) write to four repositories in one S3 bucket.

Nodes authenticate with an EC2 instance profile. Zero access keys in the keystore.

SLM handles retention, from 7 days up to ~7 years. S3 lifecycle rules stay off, on purpose.

The whole thing is about 30 lines of JSON and a handful of AWS CLI commands.

Three questions every backup should answer

Elasticsearch cluster runs on EC2 in a regulated environment. I wanted backups that answer three questions in one sentence each: how much data can we lose, how far back can we go, and what credentials does the job use?

Elasticsearch already ships a scheduler for this: Snapshot Lifecycle Management (SLM). So I moved everything into SLM and let S3 be dumb, durable storage.

The architecture

Read it left to right. Nodes get temporary credentials from an IAM role. SLM fires snapshots on cron schedules. The S3 repository writes objects into an SSE-S3 encrypted bucket, one folder per tier.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.