About four years ago, Medidata Solutions decided to switch from its traditional "waterfall" method of software development to an agile methodology. Medidata provides clinical testing solutions in a software-as-a-service model. "We made the change for all the usual reasons," says Andrew Newbigging, senior vice president of research and development. "We wanted to be more responsive to customer needs." At the same time, Medidata's IT leaders explored the possibility of outsourcing some of the company's software development. Though that might have made sense in the traditional waterfall world, they concluded that it was the wrong way to do agile.
"We do an iteration, get feedback, and iterate again," Newbigging says. "That works well for us, and we were concerned that in any sort of outsourcing approach we would lose that ability to be responsive -- and end up with a product our customers wouldn't want to buy."
It's an opinion that many industry experts share. "We have seen cases where folks were successful outsourcing agile development. But they were the exception, not the rule," says Sean Kenefick, an analyst at Gartner.
He and many other experts believe that the best way to create a pure agile methodology is to keep all development work in-house. Companies that do outsource agile development will inevitably face tough tradeoffs: They'll likely have to give up some agile principles, as well as some of the cost savings typically associated with outsourcing. And managing an outsourced agile project is likely to be more difficult than managing that same project in-house.
Nonetheless, a growing number of companies will have to face the challenge of outsourcing agile development. In a survey of more than 6,000 software developers conducted by VersionOne, which makes a tool for managing agile projects, more than 80 percent of the respondents reported that their companies had adopted agile development.
Meanwhile, the trend of outsourcing more and more IT functions continues. According to Gartner's IT Outsourcing Forecast, outsourcing expenditures worldwide grew 8 percent from 2008 to 2011, and will grow an additional 6.6 percent in the next two years. Judging from those statistics, it's clear that at least some agile development projects will be outsourced.
The case against outsourcing agile
Why is it so rare for an outsourcer to successfully deliver agile development? Consider that agile development is always challenging, even in-house, with everyone at the same location. "It requires a huge amount of discipline and cultural change, and a lot of buy-in from the group, from the lowest-level person to the highest executive," Kenefick says.
Even a company that is successfully using agile may find it hard to maintain, he adds. "It's a very delicate balance. I've seen a situation where everyone was working in a bullpen and the developers wanted their own offices. That change was enough to disrupt the agile process. Just by moving a function from one side of a room to another, you make something difficult even more difficult. If you outsource it and take it to another company, you've made it that much harder."
To make matters worse, using an agile methodology runs counter to most outsourcers' business models. "Every outsourcing company has its own methodology for developing software, which it wants to use for all its clients," says Alex Adamopoulos, CEO of Emergn, an IT consulting firm focused on agile and lean development. "That's not necessarily a bad thing -- they want to find ways to run their organization most efficiently. But it makes them less likely to adopt whatever methodology a client company is bringing."
Indeed, embracing an agile methodology is likely to make the outsourcing company's work considerably less efficient. "One of the benefits of agile is that you can take shortcuts while you develop in order to release more quickly. Then you go back and refactor the code later," Kenefick says. "But how do you do that when a third party is doing the coding? They want to get it done once, the right way. As an outside vendor, why would I want to rewrite the same code more than once if I didn't have to?"
Then there's geography. Many outsourcing arrangements involve working with programmers in parts of the world where labor is cheaper than it is in the U.S. But if moving a function across a room can disturb the agile process, moving it across an ocean is worse.
"A fundamental aspect of agile is really quick feedback," Kenefick says. "If you've got people in different time zones, you won't have those conversations as often as you need to. I'll write some code today, but then I won't get feedback on it for 24 hours."
That's one reason why the agile methodology works so well when teams are all in the same place, he adds. "It is very hard to colocate teams, but it's the first thing I'd put on my list of effective agile practices."
If outsourcing agile development makes a challenging job even more so, is it a better strategy to keep all agile development in-house? For a growing number of companies, the answer seems to be yes. "We're seeing a trend toward companies that have outsourced in the past choosing to invest in their own people's skills and capabilities instead," Adamopoulos says. "Smart companies are figuring out that the cost savings from outsourcing aren't that significant. For instance, one insurance company we work with is bringing more of its development for key projects back in-house so as to have better employee retention and less risk. Their view -- which has been that of a number of our clients here and in Europe -- is that traditional outsourcers aren't helping them get products to market fast enough or well enough. The change rate is slower on the outsourcer side, so to offset that, they're building up the capabilities of their own people."
That's the decision Medidata made last year when the company decided to expand one of its software offerings and once again took a serious look at outsourcing agile development. "We decided to hire and increase the size of our internal development team instead," Newbigging says. "We feel quite strongly that we're better off having our own developers building the software, and also being responsible for maintaining it, responding quickly to any issues. They are able to understand the software because they built it in the first place, and that's harder to achieve if the original development was done elsewhere. You don't have that same depth of knowledge."
Outsource for the right reasons
While keeping all agile development in-house may top the wish list of many IT leaders, it's not a realistic plan for every company or every situation. With looming technology skills shortages and pressure on IT budgets, some agile software projects will have to be outsourced.
So how do you decide if it's right to outsource an agile project? Begin by asking yourself what you hope to gain from working with an outsourcer. "The ideal reason is so as to give immediate response to a business need," says Max Rayner, executive-in-residence at consulting firm Hudson Crossing. "Can you find ways to also have it lower cost? Yes -- but something important enough to be done in an agile way is likely to be worthwhile, whether you're paying $1,500 a day or $1,000 a day or $750 a day. By going to some of the lowest-cost countries, you can get a very competent programmer for $25,000 a year, including benefits -- but that's not necessarily going to help your project succeed."
Before joining Hudson Crossing, Rayner was CTO at TravelZoo and oversaw the creation of the travel search site Fly.com, for which he used an outsourcing company based in Lisbon, Portugal. "That was a 100 percent agile project using scrum," he says. (In scrum, small teams work on a specified portion of the requirements for a limited amount of time and hold daily meetings to assess progress and address any questions or problems.) "I was in California, so we had hardly any overlap in our work days, but it worked brilliantly. My morning was their evening so I would get to say, 'OK, yesterday we agreed you were going to do these five things. Did you do them? How's it going?' "
There were members of Rayner's team working on the project in California as well, he adds. "At the end of the workday, each team would hand off to the other shift. That's more complex than a traditional agile arrangement where everyone is in one location, but I was willing to take responsibility for the time, cost, and feature tradeoffs," Rayner says.
That factor -- having an IT leader take responsibility for the project -- is a key differentiator, he adds. "A lot of IT managers make the following mistake: They're under pressure to reduce costs, so they decide to go to an outsourced company that promises to use agile methodologies. They think that not only will they get lower labor costs, they'll get the higher productivity of agile."
That kind of thinking can lead to trouble because using the agile methodology means giving up some of the cost savings traditionally associated with offshore outsourcing. Indeed, Rayner says, Fly.com's success was due in part to the fact that his company paid a higher price to work with the outsourcer's most experienced developers. "They were every bit as important in solving business problems as they were in solving coding problems," he says.
Outsourcing agile development may not save that much time and effort either. "You have to be willing to work as hard with an outsourced partner as you would with your own people," says Rayner. "And your super users need to be involved to help determine features."
The worst outsourced agile disasters occur when the client company thinks it can hand off responsibility to the outsourcer. "At one company I worked with, speed of delivery was worse than before the agile outsourcing," Rayner recalls. "At the root of the problem was this attitude that, 'now that we have an outsourcing contract, they have to be the ones to do it. They have to be agile and fast, and we don't have to be in touch every day.' "
He notes that there's another reason -- usually unspoken -- why IT leaders sometimes choose to outsource: So they can deflect responsibility and gain a scapegoat in case things go wrong. "You can't go to the business sponsor and say, 'I told them to do what you asked, and it doesn't work. We're going to fire this outsourcer and get a new one,' " Rayner says. "That's often what happens when a project fails."
This kind of thinking can kill an agile project before it even gets started, he says. "The very spirit of agile is to have mutual trust and respect, and a flexible relationship where you know at each decision point exactly what cost, time and feature tradeoffs you're making. That's hair-raising to some engineering leaders because they can no longer hide behind the contract."
Is 'partly agile' enough?
If you do decide to outsource an agile project, one question to consider early on is just how much of the traditional agile methodology you want to adhere to. Because working with an outsourcer will almost certainly prevent you from using a completely agile framework.
"In most cases, the outsourcing company would be using scrum as an agile practice," Adamopoulos says. "While this is fine, more and more companies are figuring out that they can use agile across a whole set of areas in the software life cycle. They might use agile in the early idea-management phase and the vetting of an idea well before requirements are even needed. It might also mean they use agile practices to develop comprehensive business cases and metrics. All of that is usually not a discipline an outsourcing company brings."
"In principle, the agile methodology says that you have a cross-functional team that is colocated. You can make decisions on the spot and you can look at things together," adds Rene Rosendahl, senior manager in the project management office at Kelley Blue Book in Irvine, Calif. The company uses an outsourcer in Beijing to provide agile development for its website KBB.com and other products. "With offshoring, you are forced to separate the product owner from the rest of the team, and you need to write things down and expect delays in decision-making. Does that mean you have to compromise some of these agile principles? I think the answer is yes. You cannot apply the principles in the same way you can with in-house teams, but you hold on to them as much as possible."
Picking the right agile outsourcer
Choosing the right outsourcing company may be the biggest part of the challenge. "Pick a partner that's really going to be your partner, not someone who'll just deliver that [statement of work] to you," advises Daryl Broddle, vice president of technology for SciQuest, an online procurement technology provider. SciQuest has used an outsourcer to do agile development for years. "I have a personal relationship with the CEO of that company," Broddle says. "He visits me once or twice a year when he's passing through. Neither company would be where it is without the other."