Lots of interesting chat on the reasons to Pen Test, or not from various blogs starting over here at techtarget then rapidly spreading out over the blogosphere (hate that word but it's the only one going) to Matasano mcwresearch, Security Incite and in a bit of a divergence from topic layer 8
And what does all this tell us.... Well there's a lot of confusion around what a penetration test is and at lot of divergence over the value they provide.
So anyway here's my 0.02 on the subject. Penetration tests are not a good way to prove the security of a system, if possible and the application is under your control go for developing good secure coding standards and develop a good Security Development lifecycle including elements like threat modeling.
However in the real world, life doesn't work like that all the time, you'll get applications which are legacy and may not have been securely developed, you'll get 3rd party applications that you've no idea how they've been developed, and at times like that a penetration test is a reasonable way of getting some level of assurance that they're reasonably secure, and there probably aren't any glaring holes there.
A couple of other points that emerged from all this reading is
- I definitely agree with Marcus Ranum that the PCI in particular look like a penetration testers (and code reviewers) full employment act. Every website that processes cardholder data (and that's a lot of sites) needs an annual penetration test and code review (unless they're using a web application firewall)
- I definitely don't agree with Bruce schneier's point about using penetration testing to identify items on the SANS top 20 list. For that if I've got any access to the systems I'd much prefer a patch management and security policy compliance tool, 'cause they're way less hit and miss than a penetration test
- Dave G at Matasano's probably right about a lot of bad penetration tests, although I think a lot of the problem is people not understanding what they're being sold (ie running nessus on someone's site is not a penetration test. Perhaps what's needed is a well accepted certification for penetration testers in different technologies like Nick Baskett talks about.
- Michael at mcwresearch's got an excellent point about using successful penetration tests as a marketing point. I'm constantly surprised at how many companies who are selling web based products, haven't had any penetration testing done
