When his servers went up in smoke and nearly half the company's data went with them, Hal Baumgardner set out to rebuild the IT shop so this could never, ever happen again
When Hal Baumgardner took over as IT manager for C2 Technologies, Inc. in October 2005, he inherited a shop that hadn't been updated in a long while.
Its infrastructure was old, its additions were patchworked in, and, as Baumgardner says, "there was no thinking outside of fixing the problem. That's fine for a small shop but when I asked, Where's our continuity of operations?, one of the techs looked at me blankly."
At the time, the Vienna, Virginia-based consulting company for the defense industry had about 250 people and was growing rapidly.
"I walked into a shop that was basically small for a long time," he said. "It ran smoothly but the company was now literally picking up people daily."
Within the first few weeks of his tenure at C2, Baumgardner walked into the office of the company's COO to discuss the state of C2's IT shop. "I said, if you want this company to survive we're going to need money," Baumgardner recounted. "He said, how much? I said a half a million to get caught up. He said, 'What?!'"
It turns out Baumgardner didn't have to further his case: Circumstances did it for him.
Baumgardner was home sick with an infection when he got the call from one of the techs in his department. "He said, one of the servers crashed really badly," said Baumgardner.
"Really badly" turned out to be a euphemism. As Baumgardner put it, "one of the daughter boards fried. There was smoke."
Two of the five drives on the RAID configuration failed, and it appeared that about 50 percent of the company's data was lost.
For any company, that's a disaster. For an IT consulting company, that could mean the end. Baumgardner's thoughts at the time? "Holy crud, we're in trouble."
Experience Counts
Baumgardner is no tech newbie. As he is the first to point out, he's "been around the block." He worked for Nortel, Booz Allen Hamilton and the U.S. Navy, where he managed 5,000 users and 250 servers. "Coming from that background, I've seen large [IT shops] and small ones. I've seen the implementation and I've seen what can go wrong."
Things went very wrong at C2 but Baumgardner managed to make some of it right. He ultimately recovered 75 percent of the data that had been lost, kept downtime to two weeks, and then went about rebuilding and reconfiguring C2's IT department to ensure that nothing like this could ever happen again.
"It's all about planning," said Baumgardner. "It's about understanding where you are now and where you'll be in five years."
Bonus Link: "See the World According to Hal"
In C2's case it's also about understanding the unique nature of a consulting firm, which is dependent on contracts to keep breathing and whose staff number is constantly fluctuating depending on the work. "You need a talent to manage a network and infrastructure from minimal usage to picking up 500 people," said Baumgardner. "We've gone from 250 people to 480 people in a year. The real problem is I walk in and I'm still new in a growing company."
Next Page: What Disaster Plan?
