A test that checks for HTTP 200 and stops there is not testing your API. It is testing that the server is awake.
I have been building a small tool that turns OpenAPI specs into runnable test suites. The hard part was never generating the code. It was deciding what a generated test is allowed to assert.
Here is the trap I keep running into in real public specs: every response is described as type: object , with no properties and no required fields. A generator can write a test against that, watch it return 200, and call it a pass. The suite is green and the contract is untested.
Two fixes, both in the spec
Give response schemas real properties and required fields. If the body can be anything, nothing can fail.
Document the error responses. When 400, 401 and 404 have no shapes, the negative tests have nothing to check.
A generated suite is a mirror of the spec. A vague spec produces vague tests. So before writing a single test, I read the spec for the holes.…