Grossman on Web App Vuln Scanners
Jeremiah provides us some interesting comments on the effectiveness of Web Application security scanners for specific types of vulnerabilities. I remember when I used to perform pen tests on web applications that some things were identified in a way that it woul be very hard to achieve the same results with an automated tool. I found very interesting results with blind SQL injection and just by looking at session tokens and realizing some kind of logic behind them. Automating these things will be very hard.
We need to stop thinking that "penetration tests" can be made by some guys running automated tools. I participated on some tests with very skilled people that were able to find subtle configuration vulnerabilities that would have been missed by scanners. I also managed to find ways to jump from one server to another just by browsing through some .ini files, ASP code and so on. The results of a test like those would bring into the light some vulnerabilities that a simple scanner would never show to you, vulnerabilities on processes and procedures. Old files with important information forgetten on the server, comments on script code, just to name a few.
When looking for someone to perform a pen test, try to find those that are able to perform tests like these. You can ask them about the kind of vulnerabilities that they found during their last tests. If they just mention the lack of patches and regular SQL injection and XSS, try another one.