There is a dangerous assumption in modern software engineering: "The pipeline is green, therefore the code is safe."
But what if the pipeline is lying to you?
I recently ran a static analysis on the GitLab CI configuration of GnuTLS , a foundational cryptographic library that secures a massive portion of the internet's TLS traffic.
I found six test jobs that can fail without turning the pipeline red .
Out of 15 jobs in the file that execute a test runner, 6 of them (40%) are configured with allow_failure: true . This means that if a test fails, GitLab still reports the pipeline as passed.
Here is the breakdown of what I found, why it matters, and how it can be fixed in one line.
The 6 Silent Gates
Job allow_failure at What it runs
mingw64/test .gitlab-ci.yml:127 mingw64-make -C tests check under wine
mingw32/test .gitlab-ci.yml:127 same, 32-bit
fedora-threadsan/test .gitlab-ci.yml:480 ThreadSanitizer suite
fedora-cross/aarch64-linux-gnu/test .gitlab-ci.yml:614 full suite under…