These are tough times to start up a security company, but entrepreneurs should not give up hope altogether, a panel of venture capitalists said Thursday at the Black Hat conference in Las Vegas.

Though security was one of the hottest investment areas during the post-9/11 economic downturn, investors today feel that those investments generally have not paid the returns they expected with few initial public offerings or acquisitions of security startups. "The reality is, not as many exits came out of that era as people were hoping for," said Mark McGovern, tech lead with the In-Q-Tel investment firm.

In the past few years, more startups have been chasing the same ideas, said Maria Cirino, general partner with .406 Ventures. "It became very crowded, and it became sort of a lemming space," she said. "When a space becomes crowded and overheated, I think it puts pressure on the space in general in terms of exit valuations."

Still, though there may be too many security startups right now, that doesn't mean there's no room for innovation.

Regulatory compliance is the most active area of investment today, but the panelists saw a few other areas that were underrepresented as well.

Good service companies that are "technically driven" are hot, said Cirino. "Those are more valuable than enterprise software companies nowadays," she said. "We just think that's the way that big companies like to do technology."

Google's recent purchase of e-mail security service provider Postini and GreenBorder Technologies has been "a very energizing trend," for investors, she added in an interview after the panel discussion.

Code-review service provider Veracode initially approached investors pitching itself as a developer tools company with little success, Cirino said. But when the company changed its model and planned to offer its code-reviewing software as a service, it got funded, she said.

Another area that seems underinvested right now is physical security, where there is a lot of opportunity to integrate security into network technologies, McGovern said. "There's remarkably little money being put into developing new capabilities for physical security," he said. Though physical security is an important concern, right now there is little alternative to "paying a guard at the front door," he said.

Looking to cash in on any new security ideas that may be percolating at this week's conference, show organizers have announced a new entrepreneurial contest.

The Black Hat Defcon Open business plan competition, announced Thursday, will be asking conference attendees to submit business plans beginning in January 2008. The winner, selected by a panel of security and venture industry judges, will get cash and services to help get their winning idea off the ground. "It's a startup in a box," a Black Hat spokeswoman said.

Thursday's panelists had a little advice for those looking for venture funding. Be humble, and don't think of your venture investors as a bank. "We're looking to become partners in a business," said McGovern. "If you're coming to a VC for just money, you're in the wrong place."

Another word of advice: Stay away from mixing personal hygiene with stimulants. When asked to name the worst business pitch he'd ever heard, New Enterprise Associates Partner Patrick Chung gave a two-word answer: "caffeinated soap."