It’s been almost a year since Amp’d Mobile imploded. Amp’d Mobile was a Mobile Virtual Network Operator (MVNO). This meant that they were a wireless carrier without a network – instead, they relied on a “real” national carrier’s network (in Amp’d’s case, Verizon’s) to deliver voice and data services. As one of the software architects who designed their central integration point and contributed to their billing platform, I had a unique view into it’s rise and abrupt demise. I’ve wanted to write a post about my experience for quite some time now, and I think that enough time has passed that I can safely speak my mind – both companies are more-or-less defunct at this point.

I should note that all of these opinions are my own – I don’t speak on behalf of BCGI, Amp’d Mobile, or anybody else.

My Role
Amp’d Mobile contracted with my former employer, Boston Communications Group (usually known by its acronym, BCGI). Among other things, BCGI sold a Postpaid Billing platform called Voyager. Up until Amp’d Mobile, Voyager’s primary market was small, regional wireless carriers with fewer than 100,000 subscribers. Before Amp’d, the Voyager product didn’t contribute much to BCGI’s bottom line and served more as a product portfolio completer than as a real potential for revenue. BCGI’s main focus had instead been on Prepaid Wireless and Payment Services.

My role was primarily to design and create a web service API to Voyager, so that other components in Amp’d’s back office could interact with the billing system. This not only included billing for voice and content, but also provisioning prepaid and postpaid services on Verizon’s network, accepting payments, and providing a customer care UI for Amp’d’s call center. Additionally, I did some testing to gauge how well the system would scale once Amp’d added the half-million-or-so subscribers that they were projecting they’d have in their first year.

What Went Well
I really don’t intend for this post to be an airing of dirty laundry. Despite the implosion of both companies, there were a few things that went very well.

• BCGI was a fairly stodgy company for its size, with a heritage in larger telecom and wireless companies like Verizon and EDS. Having said that, they gave a lot of latitude to the developers involved with this project. The pace of the development necessitated a more agile approach than BCGI was accustomed to, and the upper management at BCGI did a good job of trying to accommodate the furious speed at which things needed to be done. A lot of the normal paperwork and bureaucracy was pushed aside to get things done more quickly, to everyone’s benefit.
• Some of the development work for the project was offshored, with mixed success. The projects that I’d consider successes related to components that did real-time provisioning of services with the carrier’s network elements. These projects were small, self-contained, with clear objectives that did require a lot of business domain knowledge.
• With a few exceptions, the API scaled quite well. We quickly ramped up to support hundreds of thousands of transactions per day. These came handsets, web sites, payment platforms, etc.
• Integration between our API and Amp’d’s web components went very well. Developers from geographically dispersed areas used Instant Messaging quite extensively to communicate, and it was a far more productive means of communication than the old “our developer talks to our project manager who talks to their project manager who talks to their developer” scheme that seems so pervasive in larger companies.

What Didn’t Go Well
There wouldn’t be much to write about if there weren’t a few negatives in the postmortem, right? So here they are; first, the technical Challenges:

• Scaling the batch billing processes were more difficult than I anticipated. When I did my initial testing, I used an existing carrier’s data, and multiplied their user data gradually to project how quickly we’d be able to rate call records with larger carriers. I didn’t count on the fact that my source carrier’s subscribers made far fewer calls per subscriber than Amp’d’s subscribers eventually made. Amp’d’s market demographic was much younger than the demographic for the source data that I used, and younger subscribers make more calls. In retrospect, this should have been obvious.
• Another scaling difficulty was caused by the manner by which we received call detail records. Mediation is the process by which you obtain raw call records from a network device, and turn those records into a billable records in your billing system. For a traditional carrier, this process takes place daily (at an absolute minimum) or more often as necessary. In Amp’d’s case, we received call records once per month. This meant that we needed to process and rate over 50 million call records in a matter of hours, which is no small feat. It’s especially difficult when you’re using a billing system that was designed to mediate only hundreds of thousands of records daily, and bill an order of magnitude fewer records once per month. We managed to scrape by, but there were some pretty severe growing pains.
• We didn’t adequately anticipate the level of fraud that we encountered. We were required to make changes to head off fraud problems rather hurriedly, and that led to slapdash implementations that were prone to errors.
• The customer care UI was buggy. I previously did an entire post just about that topic.

Of course, there were business-related challenges, too.

• Most of the executive staff in both companies (Amp’d and BCGI) had a lot of experience with prepaid environments and mobile content, but very little experience in postpaid environments. This was particularly true when Amp’d Mobile was first getting started. By the time Amp’d brought on more seasoned management with post-paid experience, it was already too late. It’s important to note that the postpaid business was about three-quarters of Amp’d’s subscriber base.
• Because many of the key players came from a prepaid background, the enhancements that were requested for the platform often involved marketing content or prepaid services, instead off addressing fundamental business processes like billing and collections. BCGI, itself of prepaid pedigree, was happy to oblige. Thus, the limited development resources were allocated to content, marketing, and prepaid tasks instead of revenue assurance, payment, and other postpaid tasks. Sadly, Voyager had built-in collections modules that were never used by Amp’d because it was never a priority for them. If this had been made a priority in the very beginning, Amp’d might still be in business today.
• People still buy handsets with the intention to talk on them, not to purchase media content. When Amp’d launched, their voice plans were not competitive, and the rate of adoption was low. I think the belief was that the content was a compelling enough reason to sell the phone. However, having a good plan for good old send-to-end voice calls is still critical to attract new subscribers. Once they adjusted their rate plans for this, net add rates improved significantly.

What I Learned
It’s easy to armchair-quarterback the Amp’d experience. I obviously have the benefit of hindsight. Here are some of the things that I took away from the experience:

• As I wrote in an earlier post, business domain knowledge is key. If you’re entering a new business space, make sure you have an expert on your side who knows what the pitfalls will be, from the very beginning.
• Building a national MVNO, particularly one that offers post-paid service, is a lot harder than it sounds. I think that a lot of people mistakenly believe that the only difficult part of building a nationwide wireless carrier is having the network. But building the operational staff to support billing, credit, collections, and customer care is a huge undertaking that takes years to get right at traditional carriers. These critical business processes aren’t cheap, either, even if you manage to outsource them. I’m not sure if an MVNO can be competitive given the cost of using the host network, plus the cost of the back-office processes.
• Always plan for fraud. The bigger you get, the bigger target you become for fraud. Look at every process as a thief would – payment, distribution, service delivery, etc. Everywhere in your system where there is something of value is a target. Plan for it in advance.
• Scalability tests might be less useful than you think they are. It’s hard to predict how things will scale, and which APIs or processes will take the hardest beating. It’s probably safe to just assume that your processes will need to adapt quickly to cope with increased volume, and to be prepared for it by designing your processes to scale horizontally wherever possible.
• I had a blast. This project was one of the most rewarding experiences I’ve ever had as a developer, and it’ll be a long time before I can top it. I worked with some amazing people at Amp’d, BCGI, and a lot of Amp’d’s other vendors. The brainstorming sessions in Amp’d’s amphitheater with the whiteboard is the stuff that developer/architect types love. I still pine for the good old days when I felt like I was a part of something revolutionary in the wireless space.

This entry was posted on Wednesday, May 14th, 2008 at 7:26 am and is filed under java, programming, wireless. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.