Monday, January 28, 2008

Technology helps, but people matter most

The other day a friend called me up asking what the best scanner (web application vulnerability) is these days because he hadn’t been following the field closely. He recently left a consulting role and signed on as an InfoSec manager at a large organization. His first action was to roll out a website security initiative. He knew of course that I would be highly biased towards Software-as-a-Service. Apparently he would have gone that route, but the company had a policy against outsourcing. No one could quite remembered why. Anyway, before answering his question, I wanted to know more about his environment.

I asked if he had a website asset inventory prepared. He didn’t since he just landed although was diligently working on it, but estimated roughly 200 websites. Then I asked how many people were allocated towards managing whatever scanner he bought. 1 Full Time Employee (FTE). I stopped short and said, “That’s never going to work!” A little stunned he asked, “Why not?”

I explained by asking from his experience as a consultant how much time they typically allocated for a vulnerability assessment per website, with or without the use of a commercial scanner. He said usually 2 weeks for something comprehensive (consistent with most firms). We figured then to perform just a scan using a commercial scanner (no business logic testing) would require about 1 week to set-up, fine tune, wait, and analyze the results (removing false positives, duplicates, and properly assigning priority) per website. That meant they’d need 4-5 FTEs, not 1!

Remember that’s just to scan the websites, to say nothing of identifying business logic flaws, addressing ongoing application changes, or spending time working with developers on remediation. Is VA worth anything if you got no time to work on fixing the issues found? He didn’t want to think about how many FTE’s that would require.

I then asked if his new employer had the open head count for the project ready. He coughed and then quickly laughed, but the answer was no, not even close. He started to see the writing on the wall, so I finished the conversation by candidly saying then it didn’t really matter what commercial scanner he bought. No one was going to be around to manage it and was destined for life as shelfware. He appreciated the frankness saying said he’d have a chat with upper management and get back to me. :)

21 comments:

bw said...

Great post! It's good to see people thinking about the whole problem, not just pieces individually. I think there is always that "a-ha" moment for people when they become enlightened on the whole vs part thing. Hopefully your friend had that moment.

Andy, ITGuy said...

How true this rings! Management often doesn't think about the man hours required to do something such as this. They say "get it" b/c they think it's needed or required. They expect their current staff to install and configure it (w/o training) and then it will maintain itself. If not they wouldn't have bought it b/c they know that their current staff is already too busy and have no plans to add more.

Anonymous said...

People still need to realize that tools do not equal security. When I talk to development managers about giving their engineers our new static analysis tool, they are usually disappointed when I tell them they still need to allocate time/resources to fix the problems that the tool finds. "Wait - you want us to do MORE work?" is usually the response. Great post.

Jeremiah Grossman said...

Thanks much guys. Its just one of those things that happens and people often don't compensate for. Then later they wonder why nothing got better. and/or they still got hacked.

Anonymous said...

This post is close to our scenario as well. We ended with a hybrid model of service and scanner as it was determined during the POC we conducted that we would not be able to manage this with 1 or 2 staff members so by engaging in managed scans and utilizing the corresponding tool we felt it would allow positive initial results immediately while allowing the learning curve to be off-set by learning from the managed scans utilizing the same scan tool. We arrived at a 3 year conversion to complete in-house control of the process. I also found that the licensing model pitched by each POC vendo carried a great deal of weight in the decision from the managers perspective. If they could not put their head around the licensing model, you were all but done.

Anton Chuvakin said...

This is one of those "... and he became enlightened." posts! Holy cow, if that happens for scanners, imagine how bad the reality is for IDS, IPS, log management, etc where the human had to be present and actively involved all the time, not just during the 2 weeks of assessment....

Jeremiah Grossman said...

I think for each technology piece people must go through a separate learning curve. No one argues the people component of IDS/Log management anymore where they used to just gloss over it. In webappsec, the realization is not quite there yet as most are familiar with network scanners first with the fire and forget mindset. So they have to overcome that hurdle and it takes some time.

Unknown said...

Is there a chance this is just the start of their initiative, or the scope is far smaller than 200 websites, at least for now? Maybe hit the legacy sites later on... I can maybe see starting with small bites and see where the needs are to complement the 1 FTE and commercial product.

It could be a good, long-term gig for that single FTE, although it could still be the red flag that mgmt really doesn't care or understand the full scope of what needs done.

Jeremiah Grossman said...

@LonerVamp, well like I said it was the company's first attempt at a website security program so they lacked appreciation for what the job entails. My friend was in charge of the project so he was researching tings. Of course they had some expectations up front that proved to be impossible to meet, but as far as scope is concerned... he figured there were 200 important sites out of a possible 600 or so. So... small bits at a time with 1 FTE would take years to complete.

Shoaib Yousuf said...

Hi,

How about employing some contractors for short time and keeping 1 fulltime who will keep on eye on monthly vulnerability report?

Cheers

Shoaib

Jeremiah Grossman said...

@Shoaib, you certainly could do that, but a lot of time and cost will still be there that the organization did not properly compensate for. I mean, 5 people and 5 product licenses is much different that 1 person and 1 license.

dre said...

Security professionals can be replaced by tools. However, a web application security scanner is the worst kind of tool to put into the hands of a newbie. They need the right tools.

Yes - security professionals can also be replaced by other people who use tools that they are used to. Particularly, system administrators, network engineers, developers, software quality assurance professionals, database administrators, and any other IT professional.

For example, there is no reason for system administrators to allow command injection on their machines when defenses such as GRSecurity, SELinux, AppArmor, Samhain, and OpenBSD are available. System administrators should scan their machines for both regular rootkits as well as web rootkits (using SpyBye).

System administrators, network engineers, and database administrators can all perform threat-modeling in the way described by Ivan Ristic in this document: Threat Modeling for Web Applications Deployment. This should be a mandatory part of their workflow (another tool).

Database administrators can run Oracle 10g after release 2, which introduced a new tool called DBMS_ASSERT. Packages like this prevent SQL injection vulnerabilities of most types. Keeping systems/applications patched and properly configured is the job of many IT professionals, but database software is particularly notorious for being not only 1-2 years out of date, but often 6-7 years.

Anyone can run Nessus and Nikto on the systems/networks they are responsible for. Make it a part of the workflow! There are also 3-4 completely free web application security scanners available from Acunetix, Syhunt, N-Stalker, and Watchfire. Paros can be both a point-and-click tool, as well as an advanced tool. W3AF is also getting to the point where any person can point-and-click.

A "security team" doesn't have to be made up of only security professionals. It's probably best that it's not in most organizations. See Adam Muntner's How to create a security team for $4.95 plus tax as a great example of what organizations need to being doing if they can't hire a competent CISO who reads the Pragmatic CSO.

Foundstone (now McAfee) says that security must come from the top-down. Microsoft was proof of this.

In the case of your friend's situation -- this is a problem with decision-making inside that organization. It is not a problem of people or tools. They have plenty of access to both.

People can sit around and identify and build asset maps/tables all they want. There is no action behind this, however. It's a dead report at the end of the project.

I suggest a different kind of vulnerability assessment project. One that looks at software acquisition. How does software get into the organization? How does infrastructure rely on this software? Who builds this software and how is vulnerability management handled?

If the software is not custom, and it's off-the-shelf or a prominent open-source project that anyone can download from the Internet - then regular good old vulnerability management probably works rather well to find and prevent vulnerabilities. I suggest a MITRE OVAL-Compatible solution as action.

Custom software must be handled differently. Who wrote the software? Can it be replaced by COTS or prominent open-source? If the people that wrote the software work for your organization, then it's their fault if there are vulnerabilities in the software. No one is going to provide a third-party patch to your first-party software. In this case, you only have one option: a secure SDLC program. This involves something similar to my CPSL security process. I suggest MITRE CWE-Compatible solutions as an action plan, which would probably include a web application security scanner (although probably AppScan DE or DevInspect along with Armorize, Fortify SCA, and/or Ounce). For non-web applications, different solutions must be used, such as IDA Pro (web application security professionals hate it when I mention this tool) or .NET Reflector (happens to be a good, free web application security tool).

If you really only want one simple answer: any auditor can run Core Impact and provide you with an audit report. Core includes command/SQL injection and RFI. The auditor should be a third-party external auditor who doesn't do only PCI or SOX. If you can only afford to hire one team for two weeks every quarter (or year), I suggest they stick with one tool and one report - all from Core Impact. However, this method is as expensive, if not more expensive than most of the above added up all together.

Core Impact is also missing a few critical pieces, such as scanning for XSS, CSRF, and generic header injections - but what current commercial web application security scanner (besides AppScan DE or DevInpsect) can get those three scanned correctly/completely? I would say for XSS, W3AF + HackVertor + scanajax come close (especially when used along with SWFIntruder, Firebug, and JSView), and are probably better than any commercial products. For CSRF, the OWASP CSRFTester project is an obvious choice. HTTP header injections would best be done by Suru (which includes fault-injections), or possibly Burp Suite Professional.

Tools are the answer because automated testing is the answer.

Shoaib Yousuf said...

Excellent comment dre, but here the company has 200 websites.

Monitoring them, performing vulnerability assessments and security check is beyond the control of 1 fulltime person.

I do agree tools are more important but end of the day you do need someone to run those tools efficiently.

Here Jeremiah's friend is not looking for a monthly audit report. He is more likely looking for security team to perform everyday checklist and cover the company back, lol.

Cheers

Shoaib

dre said...

@ shoaib:

Excellent comment dre, but here the company has 200 websites

Hey, thanks. I know Jeremiah doesn't probably like it when I encourage conversation like this. However, probably not too many people read what I have to say. Let's review a few of your points as I think you ask the right type of questions.

Monitoring them, performing vulnerability assessments and security check is beyond the control of 1 fulltime person

This company appears to have lots of people. It's all of their responsibility - not one person. One fulltime person should have enough power in any organization to control a project that has tasks that involve at least 50 people. Those people need to be involved - yet they are not. Why?

I'm talking about system administrators, network engineers, database administrators, developers, software architects, software engineers, quality testing professionals, etc. I think I already mentioned this, but maybe it wasn't clear enough.

I do agree tools are more important but end of the day you do need someone to run those tools efficiently

Well, according to Jeremiah, you can outsource his Software-as-a-Service and use his people and tools. So you really don't need people, you need *his* people.

Here Jeremiah's friend is not looking for a monthly audit report

To be honest, it appears that both Jeremiah and this guy haven't done a proper SWOT analysis or anything to denote the efficiency/productivity/quality of the organization. We have no idea where they stand on a development maturity model, nor an IT maturity model. Therefore, an audit is the PERFECT solution for this type of situation -- although I would prefer strategy consulting first, followed by internal auditing second, followed lastly by third-party external auditing.

He is more likely looking for security team to perform everyday checklist and cover the company back, lol

Dood, an everyday checklist with runtime analysis? What are you, crazy? That's a total and complete waste of time!

The only proper/cheap/efficient way in 2007/2008 to perform automated {quality|performance|security} software testing is by using declarative-procedural testing. Whitehat does the special kind of capture-replay or data-driven testing that is so inefficient it makes baby Jesus cry (see also: 1972).

Quality testers (SQA's) refer to this commonly as "regression testing". Regression testing was replaced by continuous-prevention development at least 4 or 5 years ago.

Continuous-prevention development takes functional and structural testing and code generates a unit test. This unit test can be used during both the programming and integration phases of a lifecycle. In this way, the build server will at least find a large amount of recurring bugs (and much more often than any functional or even structural test because unit tests have much lower rates of errors -- which we refer to as false positives or false negatives). However, the build server doesn't have to run these tests everyday, especially during maintenance cycles. It only has to run when the software is built.

I take all this into account in my CPSL security process. It's much better to look at these problems at the source (hahahaha double-entendre there).

Shoaib Yousuf said...

@dre
Wonderful comment. Well, i think these type of conversation is really good in a sense you get to know other people views . I think thats all blog is all about. Share your ideas with people and get other people views on it.
Anyways, as you said:
“This company appears to have lots of people........................................I think I already mentioned this, but maybe it wasn't clear enough.”
I totally agree and you really pointed out a very good point. But, we need to keep in mind what Jeremiah has said in his post:
“He recently left a consulting role and signed on as an InfoSec manager at a large organization. His first action was to roll out a website security initiative.”
His friend is looking for a website security initiative not overall enterprise security. At initial level, he would need to go for a good security team to perform vulnerability assessment, risk analysis and report. Maybe, after that he can goto management with the report to start a project to fix it and involve all the people you mention?
At this point of time they don’t have any website security initiative. Involving everyone won’t be a good idea.
“Well, according to Jeremiah, you can outsource his Software-as-a-Service and use his people and tools. So you really don't need people, you need *his* people.”
Again, i agree with you but we need to keep in mind the requirements what Jeremiah has mentioned in his post.
“Apparently he would have gone that route, but the company had a policy against outsourcing. No one could quite remembered why.”
As you said:
“To be honest,........................................ -- although I would prefer strategy consulting first, followed by internal auditing second, followed lastly by third-party external auditing.”
You are absolutely right on this one. But, as i said they don’t have websites security initiative at this point. I would suggest that they should start websites security initiative first and then perform internal audit followed lastly by third-party external auditing. Strategy consulting is a good idea too but we don’t know whats their strategy is at the moment or maybe they have already done one.
In the last, answer to what you have said:
“The only proper/cheap/efficient................................... especially during maintenance cycles. It only has to run when the software is built.”
I agree but here the question is that this company has only one FT employee. In order to perform the analysis and review the report in timely mannar for 200 websites is still beyond the control of 1 employee. I understand your point but we have to look at the scenario what Jeremiah has mention in his post.

I really liked what you said:

"I take all this into account in my CPSL security process. It's much better to look at these problems at the source (hahahaha double-entendre there)."

But thats not the case here :)


Cheers
Shoaib

Jeremiah Grossman said...

@dre, thanks very much for the comment, and for the record, I do read em, just don’t have to respond to all the things I’d like to.

> Security professionals can be replaced by tools

I’ve never seen this happen in information security, personally I think tools add work for people. They’re able to do something they couldn’t have otherwise without it.

To your comment about stack and DB protection software, I’m in agreement that value can be derived, especially for newly or soon-to-be built systems. The problem is integrating them into existing systems, hundreds of them, is basically out of the question when considering the time/resources required.

> Make it a part of the workflow!

This isn’t the problem though, we can always document a process, its just who is going to do the work. That’s the point.

> In the case of your friend's situation -- this is a problem with decision-making inside that organization.

Not exactly. The problem is lack of understanding of the problem and lack of resources. Even if they see the light, are they going to invest what required?

> People can sit around and identify and build asset maps/tables all they want.

Still this has to be done, just so you have to the choice to do or not do the work.

> Tools are the answer because automated testing is the answer.

Then of course, who gets to gix the vulns?

dre said...

@ Shoaib: I would suggest that they should start websites security initiative first and then perform internal audit followed lastly by third-party external auditing. Strategy consulting is a good idea too

1) They already started the web application security initiative when they hired that guy
2) Strategy consulting is the only option for handling change this large inside any sized organization that does not have the decision-making support to handle web application security.

but we don’t know whats their strategy is at the moment or maybe they have already done one

Strategy consulting provides those answers. It can take one day if you use the right business tools and can interview enough people.

@ Jeremiah:

Security professionals can be replaced by tools

I’ve never seen this happen in information security, personally I think tools add work for people. They’re able to do something they couldn’t have otherwise without it


I've never seen it happen either. I'm not saying that the overall head count of the entire organization will decrease. I'm saying that the number of security professionals will decrease if security is built into the SDLC. An organization would have to hire more developer-testers in order to accomplish this; although the payout is huge.

The problem is integrating them into existing systems, hundreds of them, is basically out of the question when considering the time/resources required

I'm not sure I agree with this. Changing one line of an Apache config on thousands of hosts can take an hour or two (using the right tools). Tools automate this; tools such as STAF, isconf, cfengine (all open-source) or commercial tools such as HP Opsware, Microsoft SMS, IBM Tivoli Configuration Manager, and Solidcore. A classic example of where people can be replaced by tools.

Tools are the answer because automated testing is the answer.

Then of course, who gets to fix the vulns?


A unit test, which is automated from a continuous-testing IDE (during the programming phase) or a component test on the build server using a test harness (during the integration phase). Some types of vulnerabilities would have to be modeled in order to work as unit tests. Component tests using a virtual mock or in-container testing wouldn't likely require any modeling. Unit/component tests could just assert the defect (catching other similar defects) and provide the fix.

Shoaib Yousuf said...

@dre

“Tools automate this; tools such as STAF, isconf, cfengine (all open-source) or commercial tools such as HP Opsware, Microsoft SMS, IBM Tivoli Configuration Manager, and Solidcore. A classic example of where people can be replaced by tools."

End of the day still you need enough resources to run these tools efficiently according to company infrastructure....You can only do that by employing enough professionals.

Still one fulltime employee will not be able to run these tools efficiently on 200 websites??

I agree with your comments and everything you have said but I personally think organisation will still need enough professionals regardless of whatever tools you use….

Cheers

Shoaib

Anonymous said...

In a twist of irony, Kaspersky is currently flagging this blog (and other blogspot blogs) as a dangerous Phishing site!

From Kaspersky popup alert: "This web page is used for stealing credit card numbers, PIN-codes, passwords and other personal data. Web page address: http://jeremiahgrossman.blogspot.com/"

A reminder on how automatic tools often fail.

Jeremiah Grossman said...

OMG, they found!? :)

Anonymous said...

How true this rings! Management often doesn't think about the man hours required to do something such as this. They say "get it" b/c they think it's needed or required. They expect their current staff to install and configure it (w/o training) and then it will maintain itself