Sunday, April 15, 2007

How I got my start

I’ve told this story from time to time when asked, but never written about it. The inspiration to do so came from posts by Security Catalyst and Andrew Hay. Computers have been a part of my life for nearly 20 years, starting with a Commodore 64 when I was ~10, a few x286s and x386s, then a Power Macintosh 6100 my mother bought for me when I was 16. Coded a lot in between, but that’s ancient history and a story you’ve probably heard a hundred times. So let’s skip to where it gets interesting. And to borrow on line from Great Expectations: “I'm not going to tell the story the way it happened. I'm going to tell it the way I remember it.”

As a late teenager I left Maui for California to attend college, find a decent job, and seek out greater opportunity. I enrolled in a trade school for electronics engineering. At the same time my couple years of web development and Unix experience was enough to land me a job as an entry-level Unix Administrator for Amgen (a large bio-tech company). I got to work on mega-big Sun (Solaris) servers as part of a team responsible for backups, disaster recovery, and web-enabling day-to-day operation. Amgen was very wealthy, treated their employees exceedingly well, and threw lots of posh parties. Most of which I couldn’t attend because they served beer & wine and I wasn’t yet of age. Funny eh. Being only 19 or 20, this was a kickass job, learned a lot, and loved it.

Then in the autumn of 99’ something strange happened. A slew of well-publicized articles hit the Web saying someone found vulnerabilities in many of the major websites like Yahoo!, Amazon, and eBay. As a result they were “insecure”. I knew making a secure website was exceeding hard, and as far as I was concerned, impossible. So being a naive youngster, I couldn’t figure out why this was newsworthy and thought everyone already knew! What happened next was even more interesting. Weeks later the same companies issued statements that they had fixed the issues and were now completely “secure”. Amazing, naïveté kicking in, how did they do this!?!? I had to know!

After work I registered a few shiny new Yahoo! Mail accounts, and thought to myself – “how hard could this possibly be (to break into my own accounts)?” I don’t remember the actual vulnerability, but it took only a few minutes to find. Probably had something to do with an XSS / JavaScript filter-evasion or meta-character injection. Wrote up a simple minded advisory and sent it anonymously to the only internal Yahoo! corp email address I could find (specific address withheld). The message explained what the issue was, said I didn’t want any credit or press, and to let me know if they had any questions. Had my fun for the evening figuring that was the end of that.

The next morning I checked the account and lo’ and behold someone responded! Wow, Yahoo! is talking to ME! During those days there was no company bigger or more exciting than Internet darling Yahoo! The email read thank you for letting us know about the issue, we have a couple of questions, appreciate you wanting to remain anonymous, but we’d like to send you a t-shirt. WHOA HO! Not only was I able to find a vulnerability in Yahoo!, but it was important enough that they’re asking about it and want to supply me with clothing. Cool. They also said to let them know if you find anything else. Sweet, an open invitation.

Over the next week or two I’m finding bugs, reporting them, and having a casual dialog with somebody at Yahoo!. I mentioned to a co-worker what was going on and they were excited about it as well. They asked whom I was communicating with, but hadn’t thought to check. I forwarded one of the emails so they could see. They facetiously told me to have a look for myself. I took the name from the email address (withheld), and I kid you not, it was from David Filo, one of the two Yahoo! founders and so-called Chief Yahoo!. OMG! Here I am some dumb kid with a few web app vulns and I’m having an informal nonchalant conversation with a billionaire. BTW, Filo is a very cool guy and if you ever met him, he’s an amazingly smart engineer without an ounce of arrogance or superiority complex that one might expect.

A few days later a Yahoo recruiter emailed and asked if I’d like to fly up to Silicon Valley for an interview. Stunned is the word that best describes my reaction. I made sure they knew that I had no formal education or security experience to speak of. It didn’t bother them, so it didn’t bother me. At their purple and yellow laced offices, they grilled me on all sorts of technology questions for 6 hours, most of which I didn’t know. C’mon, this was Yahoo, how could I? Figuring I had no shot at all I promptly flew home. I heard back a couple days later via FedEx. YES! An offer letter from Yahoo! OMG, look at the salary! COOL, job description was nothing but security! NO WAY, stock options! YES! Wait, what the heck are those?

I took the prized yellow piece of paper to my school counselor and asked her a very simple question, “What’s the top student out of here expect to make at their next job?” Begrudgingly she told me and it was nowhere near what my offer letter said. Then asked if she had any GOOD reason why I should say. The only answer I recall was that you’ll never go back and finish your degree (she was right of course). And just like that I was gone and on my way. Took a month off back on Maui, hiding out from Y2K, then started my new life in January of 2000.

I became part of Yahoo! Engineering and felt completely out of my league around the real super brains of Web technology. My job was simple. Identify as many security vulnerabilities as possible, formulate solutions, and chase down developers to resolve them. Talk about awesome with my new "hacker yahoo" job title. It took about 6 months to get my arms around web application security and how big and important this job actually was. The scope was roughly 180 million users and 17,000 publicly facing web servers. 12+ hour days were the norm. On a system that massive, IDSs say only 1 thing that matters, everyone is attacking you with everything they got all the time. Unscientifically we thought about 1% of our user-base was malicious, a similar number we heard from our industry peers. Yes, 1.8 million bad guys.

I performed what seemed like a never-ending supply of web application security assessments. It was rare for a website to be completely free of security issues. To streamline the workload I developed an assessment methodology consisting of a few thousand security tests averaging 40 hours to complete per website. The eminent domain was over 600+ websites, enterprise-wide, in a dozen or so languages from english-to-french-to-korean. The way the math worked out assessing the security of every website would have taken over 11 years (24,000 total hours/ 2080 working hours per year) to complete if nothing ever changed. Yah, right.

Obviously we needed to reduce workload because few experts existed and we certainly weren’t going to hire a dedicated team of 10. Seeking help I talked to every expert I could find and experimented with early commercial scanners and web application firewall solutions. None of what I saw was going to solve our problem. Almost two years in what I did see was opportunity. Yahoo! was certainly not the only one in the e-commerce business or grappling with too much work, too few experts, and a lack of decent tools. I felt I could do better, which ultimately led to my founding of WhiteHat Security.

Well, that’s how I got my start. And the last 6 years have only gotten more interesting.

15 comments:

Andrew said...

Wow....

Kind of makes my initial posts on how I got my start sound rather insignificant ;)

Maybe the next section of my series will sound a little more interesting :P

Jeremiah Grossman said...

Hey Andrew, Thank you. :), my intent wasn't to one-up you or anything. I've just try to follow the fun in whatever I do. As a result things tend to turn out interesting no matter what.

"Check back shortly for my next posting which details my move from my first call center job, to my next call center job, to my ‘dream job’ at Nortel Networks."

And you know what...some of the funniest and amazing stories I've EVER heard came from people working call center jobs. I bet you have some great tales to tell.

Anonymous said...

Now now Andrew, we all can't be the infamous Jeremiah Grossman, Mr CNN, mr techtarget :) Mr XST himself

Anonymous said...

ah jeremiah, how interesting your life is. I guess being working in the states gives you a lot more opportunites than being in Asia in terms in IT. States has all the advanced technology and the latest stuff happening there. Big companies are there too. In Asia, its hell hard to get into international companies like yahoo. Still, i love your story and i find it motivating for me.

hackathology.

Jeremiah Grossman said...

@Anonymous, yah, I'm so famous I'm in-famous. The Amigos. :)

@hackathology: Thanks. :) The only place I've really worked in is the U.S, except for a few short excursions, so I have no basis for context. But there certainly is a lot of opportunity here to take advantage of.

LonerVamp said...

Definitely an interesting read! Being still in the burgeoning part of my own career, I love these stories of how people got started. It is, well, maybe not inspiring so much as it helps see the potentials of reality. Definitely love that topic on the Catalysts site for the same reasons!

Anonymous said...

The Commodore 64 is where its at. When someone comes out with an automated scanning tool for the C64, I am buying it. I wonder what headers are returned from a web server being run on a C64?

Anonymous said...

jeremiah, keep your story coming man. I am camping here.

hackathology

Jeremiah Grossman said...

Thanks, I'll do my best to keep the posts coming. All this travel is killing me though.

Jer in NY.

Mephisto said...

Excellent story Jeremiah, I always wonder how people get started in this industry and it appears, as was the case with me, it's usually just by accident.

Unfortunately, the majority of the application security jobs are on the east and west coasts, not much here in the midwest.

Anonymous said...

easy jeremiah. Thank your time

hackathology

Alex said...

Wow, you got a t-shirt. :)
I only got a watch from a bank, but it's not a Rolex. ;)

Jeremiah Grossman said...

Actually, I never got the shirt. Everyone just forgot in the including me for several years. Until just recently Yahoo! invited me back to speak at an internal conference and while telling the same story it occurred to me. I made mention of the lack of payment. They made good with a t-shirt that is very hard to get. You can actually see me wearing the Paranoid Yahoo Security Team shirt at the recent WASC meet-up:

http://myappsecurity.blogspot.com/2007/04/wasc-meetup-april-18-pictures.html

Anonymous said...

I got a Google t-shirt as thanks for reporting a 5-6 XSS vulnerabilities on Google's main search page. Starting a conversation directly with a companies security team via security@ is a good way to start an interview. It wasn't my intention in this case.

Phil call centers said...

great post jeremiah... even up to now i can use the information that you shared...