Tuesday, January 09, 2007

Drawing a line in the "Scan"

Normally I don’t post shameless company promotions on my blog, but this one is different. I thought people might find it interesting to follow the results. Commercial web application scanner vendors (Cenzic, SPI Dynamics, Watchfire, etc.) and service providers like myself from WhiteHat Security go back and forth with claims about what scanning technology can and can’t find. I say scanning is only capable of testing for about half of the issues (technical vulns) – they claim they find logical flaws. Who’s right? It's time to find out.

Enterprises are incentivized to select the best solution to find exactly where the vulnerabilities are. That’s where the focus should be. We all loathe reading lame paid-for 4-star reviews and bogus magazine awards. It’s 2007, and I say it's time to let the results speak for themselves. The hard part about measuring results is you never really know the total number of vulnerabilities present in custom web applications, and demo sites are a poor baseline for measurement. The best results are gathered using real websites when solutions go head-to-head, but obviously you just can't go out and pen-test any website you feel like.

As it happens a large portion of our Sentinel customers, with some of the largest and most popular websites in the world, previously purchased commercial scanners. They said they were complex, reported too many false positives, or the assessments were faster to do by hand. *Survey results back this up* Its not that the tools don’t work. They’re sophisticated and ended up not being the right solution for the job. Unfortunately many others in a similar situation are hesitant to try something new for fear of throwing away good money after bad. Worse still, their websites remain unprotected and head-to-head comparisons between competing solutions become few and far between.

Our results are better, but I'm not here asking people to take my word for it. I have something else in mind. Here's the deal: If someone previously purchased a commercial scanner and ended up not using it, not liking it, or curious about alternatives they can receive up to a $30,000 credit towards an annual Sentinel subscription. Completely risk-free. They'll see our results first hand on their website for comparison against their current scanner reports. (Full details) The enterprise gets to decide what can and can’t be scanned for. Win or lose or draw, good or bad or otherwise - we're all going to learn something.

2 comments:

alikl said...

Jeremiah, love your postings.
I can recall your recent survey where respondents stated they almost do not use commercial or even open source scanners - every body use her own techniques. I do not think, actually i am sure, it is not one or another. MSDN has very nice walkthrough on how to conduct code inspection for security for .net code http://msdn2.microsoft.com/en-us/library/ms998364.aspx. Basicly it has few steps to go, one is preliminary scan which can be automated using mentioned tools (i use windows built in findstr which looks strings in compiled assemblies and source files as well)and then code reading drilling deeply into the logic flaws. So far it works for me.

Jeremiah Grossman said...

Thank alikl.

Basically the surveys said that about 50% of pen-testers use commercial web application scanners. The ones that did viewed those products as performing about half the work and finised the rest by hand. The other half chose not to use them because they felt it was shorter to do the whole assessment by hand.