More often than I’d care to say, I work on projects where a client has a vulnerability spreadsheet to rule them all. They’re using the spreadsheet to track all of the open items that were found across all of their projects with different tools and pentests.
One initial interesting point is that these companies don’t particularly seem to consider this data to be extremely sensitive. The spreadsheets get mailed around with different tabs for different apps or organizations to teams across the company. Good thing we just told all of our IT and development resources where the known problems are …
Going a little deeper, I can often tell a lot about a company based on what I see in the spreadsheet. Maybe a simple Apache web server patch hasn’t been applied in 9 months. Maybe some teams respond and others don’t. Maybe it’s hard to find owners or they keep shifting.
Experienced security folks can often map vulnerabilities in a report back to the tools that find them. You know the X-Frame-Options item came from ZAP or Burp. The listable directory too. Content types, password fields that don’t have autocompletion turned off, JS from foreign sites, etc. You know the drill.
Something that I also find very interesting is what is NOT in the report. If there aren’t ANY authorization related findings or other items that I wouldn’t expect a tool to find, I can often be quite confident that either the application testing was very time-limited or the testing methodology did not include human-driven testing. This should be a red flag.
Unless you are trying to pay for only an automated test, look for a vulnerability you know a tool can’t find but ANY tester should. Maybe even consider adding something you can use as a canary to test the tester. There are lots of examples but an easy one is a user from one access role or organization being able to access data or a function from another access role or organization – basically an authorization failure. Tools can’t find these. If you don’t have any, it might be because your testers are only using tools.