Menu

Helm reported a successful release while its data job was still failing in the first region
📰

Helm reported a successful release while its data job was still failing in the first region

DEV Community: kubernetes·Sergey Shinder·14 days ago
#YuzXRw6A
#devtokubernetes#devto#dev#release#helm
Reading 0:00
15s threshold

We release our pricing service one region at a time. The pipeline runs helm upgrade with --wait in the first region, and when Helm reports success it moves on to the next, on the reasoning that a release healthy in one place is safe for the others. In October a release reached all three regions in twenty minutes, and in all three our quotes went out with the previous year's fuel surcharge.

The chart contains a Job that loads the current tariff tables into the service's database after each upgrade. It is an ordinary resource, not a Helm hook, and its name carries the release revision, because a Job's pod template cannot be changed once the Job exists and Helm would otherwise fail trying to update it. In this release the job could not read the bucket holding the tariff files, because a bucket policy change had gone out the day before. It failed, retried and kept failing.

Helm said success anyway.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.