Menu

One failing job in our nightly matrix cancelled the eleven that would have told us more
📰

One failing job in our nightly matrix cancelled the eleven that would have told us more

DEV Community: cicd·Sergey Shinder·17 days ago
#zNQ8sP6n
#devtocicd#devto#dev#failure#every
Reading 0:00
15s threshold

Our database client library supports four Postgres versions and three Python versions, and every night a GitHub Actions workflow runs the integration suite across all twelve combinations. In the week before the 3.0 release it failed five nights in a row, and every morning it reported exactly one broken job.

On Monday it was Python 3.10 against Postgres 13, a test matching the exact text of an error message that 3.10 had reworded. We fixed it. On Tuesday the same workflow failed on Python 3.12 against Postgres 16, a test helper still importing distutils, which 3.12 removed. Wednesday was a third failure, Thursday a fourth.

None of those failures was new. All of them had been there on Monday. A matrix in GitHub Actions has fail-fast set to true unless you say otherwise, which means that when any one job fails, every job in the matrix that is still queued or running is cancelled. Our first failure arrived about four minutes in, while most of the other eleven were still installing dependencies.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.