A vulnerability scan is an automated tool that checks a system against known weaknesses, and a penetration test is a person who tries to break in the way an attacker would and shows what they reached. To see the gap, we built 27 web flaws on purpose and ran OWASP ZAP and Nuclei against them in six setups. The best setup found 3 of the 27, found 1 in part and missed 23. All six setups together found 4, found 2 more in part and missed 21 outright. The 21 include SQL injection, five login-flow flaws and six authorization flaws.
We use "vulnerability scan" for the automated tool run and "vulnerability assessment" for the wider job of listing and rating weaknesses, which usually starts with a scan. We tested the scan part: two scanners, no person steering them. The figures below describe scanners, and say nothing about a full assessment that adds a human review of the results.
The test apps are the 27 fixtures in our web-security-fixtures repository (commit 2c4a833).…