When a big new project lands in a company’s lap, the instincts of upper management, while well intentioned, are usually to go with people and paradigms that have worked in the past.
The culture of my former employer was forced to change suddenly to support new business, and it ultimately resulted in failure. I’ve had several months now to think about what happened. In retrospect, a lot of it could have been avoided if the company’s leaders had ignored their instincts, and been daring enough to rely more on different people with the right expertise.

This is more long-winded than most of my posts, so those with short attention spans should probably turn back now!

Background
I started working for a small company (fewer than 20 employees) in 2000 that provided billing services to small, rural wireless carriers in the United States. That company eventually grew to about sixty employees and was acquired by another medium-sized company about three years later. The new company provided other services to wireless companies, but lacked a post-paid billing platform. It seemed like a perfect fit. The smaller billing company would become a satellite office for the corporate mothership.

A Corporate Culture Shift
In the past, the team working on the billing system had been allowed a lot of latitude to work independently from the corporate mothership. However, this was a high-profile project with a large national carrier, so the company decided to bring in a lot of its leadership’s “big guns,” who had experience with large carriers. Unfortunately, they had little or no experience in the business domain, namely post-paid wireless billing.

Unlike the small team that had managed the product in the past, the new folks had little sense of what the carrier’s priorities should be. This new team included client relations representatives, many of the project managers, and even a majority of the technical team. The people who had dutifully supported the small billing program for so many years would continue to work in “maintenance mode” on the existing platform for the existing rural clients. The implication was clear – they’d been relegated to the “less important” clients.

The User Interface
The billing system had a user interface which allowed representatives at the customer call centers to change things for the subscriber in real time – things like rate plans, billing addresses, automatic payment, allowed managers to configure their own rate plans and user privileges and roles, and it even had an integrated Point-of-Sale system.

The system was “good enough,” but barely. Unfortunately, it was written using a poorly supported, obscure 4/GL whose scalability was in doubt. Deploying it at carrier sites also created some sketchy security issues. We had been meaning to re-write it for a long time, but didn’t have the resources to do so. It wasn’t going to be good enough for our new client.

Offshoring to the Rescue
At about that time, the parent company acquired another company in India that processed payments for wireless carriers. Suddenly we had nearly limitless programming resources at our disposal. All we’d need to do is mock up some new screens, write a few specs, and send the work to our new development staff. We’d “keep a close eye” on their progress, strip the functionality down to the bare necessities, and we’d have a re-vamped UI in a matter of months. As an added bonus, the new programmers already knew Java, so we wouldn’t need to re-train any of the existing workforce to get started.

The first thing the new leadership did was to consult with a UI expert to do the mockups – one that (of course) had no experience with the product or with wireless carriers. She was mentored by a product manager who had very little experience with post-paid billing.

Fortunately for me, I got to work on a SOAP-based API for the product to integrate with the other elements of the back-end. I saw the train wreck coming with the UI project, and I managed to avoid it. I worked quite independently with another contractor based near the corporate mothership.

Reviewing the Mockup
The first thing we did was review the new UI mockups. The new leadership had the right idea, and decided that it might be a good to include some people familiar with the “legacy” system in the review. The legacy team was horrified. The new application looked completely dysfunctional. Naturally, the “new team” got defensive and insisted that the crufty old-timers were thinking too much in the past, that this was the future. Their comments fell on deaf ears. They weren’t invited to participate any more.

Developing the UI
So here’s how development worked: The company hired some brand-new Java superstar contractors to work in their corporate office to “oversee” the development effort. The contractors were great Java programmers, but they had literally zero business domain knowledge. Every morning, they would get onto a conference call with India to review the previous night’s work and discuss the next day’s priorities.

Nobody on the project knew if they were doing it right. There was no connection between the back-end and the wireframes. They started consulting me with business-related questions, and I helped out where I could, but that ended up being like the game of telephone that you play in grade school – things are lost when they are translated three times over, and communicated over a garbled conference call and IM. The rest of the legacy team was too busy supporting the existing customers to help answer questions.

So domain knowledge continued to drip very slowly from wherever it could, over to the mothership, and finally to India. Too slowly. They began to sit idly on a UI that, because it followed the mockups religiously, was fairly unusable.

Eventually, the developers on the other side of the ocean started to get fed up, and were leaving the company. I couldn’t blame them, and the fact that there are so many jobs available in that market made it easy for them to leave. What little domain knowledge had been built up was constantly fading away. They began hiring furiously, which compounded the problem further.

I heard complaints from the superstar contractors that the quality of the work was a bit sketchy. I imagine that the quality of the work must have worsened as the rate of turnover increased. We started bringing on extra contractors stateside to help out. Have you ever read Brooks’ The Mythical Man Month? Yeah, he knew what he was talking about. Adding developers to a late project makes it later.

One night I was visiting the mothership, as I had to do almost weekly to keep up with the latest news on the project (it was about an hour-long drive from our satellite office). We were weeks away from the launch date. The lead UI developer called me into his cubicle and asked if I could spare a few minutes. We started reviewing the UI, and he began asking me how to wire up the pages with the back-end. We looked at page after page and I told him “you can’t do that – it doesn’t make sense,” or “we have no feature like that, I don’t know what it is supposed to mean.”

We were supposed to use my SOAP API to do the business logic, but they had laid none of the groundwork for it. The programmers were pretty good at calling stored procedures. So, on page after page, I told him I’d write a stored procedure to do the work. Add a rate plan? I’ll throw together a procedure. View the customer detail? Yup, that too. We started eliminating unnecessary pages, buttons, and fields left and right.

The next day I started hacking frantically on a raft of stored procedures that did most of the back-end logic for the UI, and I did a pretty mediocre job. I was in a hurried marathon session, and I like to believe that I did the best that I could given the circumstances, but at that point it was definitely all about volume, not quality. Work on the API was pretty much halted, but it was nearly finished anyways. Soon it was time to start training the call center.

Was the Customer Happy?
Fortunately, the client had problems of their own with ramping up. Their operations people were initially satisfied with the product because they had very little experience in the business domain, either. But it didn’t last.

As time went on, and more experienced management was hired by the carrier, they became incredulous that the UI couldn’t perform the simplest of tasks that competitor’s systems could. They were frustrated that it was so buggy. However, the crappy Customer Care UI ended up being the least of the carrier’s problems – that story is for another day. Eventually, both companies more-or-less crashed and burned, which is why I hope it’s safe to share this story with you now.

Takeaways and Lessons that I Learned
So, here are the things that I can impart based on what happened to this project:

• Domain Knowledge is king. Programmers are a tenacious bunch, and we can learn technology fairly well. Years of insight from business relationships can’t be learned overnight. The mothership had the roles reversed – the big guns with experience with big carriers should have been mentoring the employees with the domain knowledge, instead of the other way around.
• Working with people half a world away is hard. I don’t think that it had anything to do with the fact that the offshore programmers were in India. I think it would have been just as difficult if they had been Canadian or Australian or any other English-speaking country. India has plenty of talented programmers. But sharing domain knowledge becomes even harder over conference calls, Skype, and IM. When developing an application in a short timespan, you need to be able to holler questions over the cubicle wall, or grab a whiteboard on a moment’s notice, which isn’t an option when offshoring. I’m sure the Indian team was just as frustrated as we were. I know that there are companies out there who have been successful at this, but it’s difficult to pull off, and our company didn’t manage to do it. There are probably tools out there to make this easier – do you know of any that you can recommend?
• The User Interface is the Face of the Application. Your UI probably needs the most attention, and probably gets the least. Your client’s perception of the quality of your product and engineering is largely based on what they see. Our billing back-end was quite sophisticated and managed to scale well enough, but the client didn’t see that. They saw the crummy exception messages and the things that they couldn’t do from the interface. You can often fake the back-end for a while by throwing more people and process at fixing problems – you can’t easily fake quality in your UI. To the client, our whole product was crap because the quality of the UI was poor, and this should be obvious to everyone. Have you ever worked with a pathetic UI, and concluded that the company did a shoddy job of engineering?

Phew! What a long post. Thanks for sticking it out and reading the whole thing. I’d be interested in hearing if anyone else has gone through a similar situation, and their company found ways to overcome the knowledge gap. Have you been in an acquisition where your formerly tightknit team had to integrate and ramp up within a new corporate culture? How’d you get through it?

This entry was posted on Tuesday, March 25th, 2008 at 6:12 am and is filed under outsourcing, programming, wireless. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.