I had an amazing revelation at work last week.

At my previous job, the company culture steadfastly clung to the waterfall model as the one true way to develop software. The model was imposed as a series of documents with cryptic, archaic acronyms for names. I actually had to draw a flowchart for my team to help them keep track of when each document was required in the life cycle of an enhancement request.

No code was changed in our code-base without a minimum of literally five separate documents, each usually spanning tens of pages. Project managers at the company enforced the production of these documents in a near-religious deference to “the process.” Each document was reviewed, often via conference call, to receive it’s “stamp of approval” (basically a C.Y.A. cross-check) among the affected teams (usually development, production support, professional services, client management, security, perhaps others).

It often felt as though I worked at a documentation company that happened to create software as a by-product.

It was in this atmosphere that I first started reading about so-called Agile methodologies. Iterative prototyping, unit testing, and tight user/developer communications, their practitioners argued, were a more effective way to deliver software that met customer expectations. Because specifications change, and users don’t know what they want, you’re already behind schedule by the time you get done with all of that front-end junk.

It all sounded so wonderful – too good to be true. If only my company were more like that.

Now I’m at a new job. There’s very little process here – it goes something like this: A nominal business analyst creates a document with a punch-list of specifications, and the developers go off and implement it. That’s it.

You’d think I was in paradise! I would be freed of the shackles of Microsoft Word, and I’d be able to churn out code as efficiently as possible.

What Really Happened
I soon discovered myself going in circles. I’d spend a lot of time in front of the whiteboard drawing ERDs for the database, and then I wouldn’t know where to go next. Start Object-Relational mapping? Hmm… seems to soon for that. The business model? And what is the business model… what does it need to do? And will the business model’s persistent data be adequately served by the data model on this white board?

Maybe I’ll just dive in and start implementing… Agile-style. I guess that means I should start with the UI; how do I do the UI? That’s what the user wants, right? The spec says I need to do such-and-such… I know! I’ll crate a UI, and wire it up to the model’s interface, and mock-out the actual implementation of the model! So there’s really no ugly business model code to worry about.

So what should the model’s API’s be? What parameters do I need to pass around to make this thing re-usable? Will I end up needing to refactor the whole thing to support some relationship I haven’t considered yet? Oh well. Don’t worry about that for now.

And, by the way, the UI is just an admistrative, ancillary part of this project. It’s actually a batch process that runs nightly which is configured via the UI. The UI isn’t even technically necessary. Am I putting the proverbial cart before the horse? Hmmph – probably I shouldn’t start with the UI, then.

Hey – look at this whiteboard… an ERD… let’s work on that for a while!

What Works for Me
What was actually wrong with my old job was the sheer volume of documentation, and the ratio of useful information to the total number of pages. That documentation was almost never for me – it was for other people (although I still wonder how valuable it really was to other people).

Personally, I need something to ground the project. I need a place to organize my thoughts. For me, a “specification” isn’t a giant UML class diagram that I slavishly adhere to, and keep in sync with the code at all times. It’s a place where I enumerate the affected components, discover how they interact, and itemize the work that needs to be done. It’s where I validate that the thing I’m about to build is going to come somewhat close to meeting the user’s expectations.

Sometimes, I share the document with other people. Sometimes I don’t. But I’ve learned that personally, for me, I need a place to focus, and collect the pieces that need to interact, before I start writing.

For some reason, this was an amazing revelation to me. My experience at my last job had led me to believe that I was firmly in the anti-documentation camp. It turns out that, in my case, that wasn’t true at all. I need some documentation. And I don’t think it’s true for everybody, either. I’m sure there are people who are better at diving right in, and still others that want even more documentation than I do.

What works for you?
Photo Credit: Fredrik Lundh

It’s been almost a year since Amp’d Mobile imploded. Amp’d Mobile was a Mobile Virtual Network Operator (MVNO). This meant that they were a wireless carrier without a network – instead, they relied on a “real” national carrier’s network (in Amp’d’s case, Verizon’s) to deliver voice and data services. As one of the software architects who designed their central integration point and contributed to their billing platform, I had a unique view into it’s rise and abrupt demise. I’ve wanted to write a post about my experience for quite some time now, and I think that enough time has passed that I can safely speak my mind – both companies are more-or-less defunct at this point.

I should note that all of these opinions are my own – I don’t speak on behalf of BCGI, Amp’d Mobile, or anybody else.

My Role
Amp’d Mobile contracted with my former employer, Boston Communications Group (usually known by its acronym, BCGI). Among other things, BCGI sold a Postpaid Billing platform called Voyager. Up until Amp’d Mobile, Voyager’s primary market was small, regional wireless carriers with fewer than 100,000 subscribers. Before Amp’d, the Voyager product didn’t contribute much to BCGI’s bottom line and served more as a product portfolio completer than as a real potential for revenue. BCGI’s main focus had instead been on Prepaid Wireless and Payment Services.

My role was primarily to design and create a web service API to Voyager, so that other components in Amp’d’s back office could interact with the billing system. This not only included billing for voice and content, but also provisioning prepaid and postpaid services on Verizon’s network, accepting payments, and providing a customer care UI for Amp’d’s call center. Additionally, I did some testing to gauge how well the system would scale once Amp’d added the half-million-or-so subscribers that they were projecting they’d have in their first year.

What Went Well
I really don’t intend for this post to be an airing of dirty laundry. Despite the implosion of both companies, there were a few things that went very well.

• BCGI was a fairly stodgy company for its size, with a heritage in larger telecom and wireless companies like Verizon and EDS. Having said that, they gave a lot of latitude to the developers involved with this project. The pace of the development necessitated a more agile approach than BCGI was accustomed to, and the upper management at BCGI did a good job of trying to accommodate the furious speed at which things needed to be done. A lot of the normal paperwork and bureaucracy was pushed aside to get things done more quickly, to everyone’s benefit.
• Some of the development work for the project was offshored, with mixed success. The projects that I’d consider successes related to components that did real-time provisioning of services with the carrier’s network elements. These projects were small, self-contained, with clear objectives that did require a lot of business domain knowledge.
• With a few exceptions, the API scaled quite well. We quickly ramped up to support hundreds of thousands of transactions per day. These came handsets, web sites, payment platforms, etc.
• Integration between our API and Amp’d’s web components went very well. Developers from geographically dispersed areas used Instant Messaging quite extensively to communicate, and it was a far more productive means of communication than the old “our developer talks to our project manager who talks to their project manager who talks to their developer” scheme that seems so pervasive in larger companies.

What Didn’t Go Well
There wouldn’t be much to write about if there weren’t a few negatives in the postmortem, right? So here they are; first, the technical Challenges:

• Scaling the batch billing processes were more difficult than I anticipated. When I did my initial testing, I used an existing carrier’s data, and multiplied their user data gradually to project how quickly we’d be able to rate call records with larger carriers. I didn’t count on the fact that my source carrier’s subscribers made far fewer calls per subscriber than Amp’d’s subscribers eventually made. Amp’d’s market demographic was much younger than the demographic for the source data that I used, and younger subscribers make more calls. In retrospect, this should have been obvious.
• Another scaling difficulty was caused by the manner by which we received call detail records. Mediation is the process by which you obtain raw call records from a network device, and turn those records into a billable records in your billing system. For a traditional carrier, this process takes place daily (at an absolute minimum) or more often as necessary. In Amp’d’s case, we received call records once per month. This meant that we needed to process and rate over 50 million call records in a matter of hours, which is no small feat. It’s especially difficult when you’re using a billing system that was designed to mediate only hundreds of thousands of records daily, and bill an order of magnitude fewer records once per month. We managed to scrape by, but there were some pretty severe growing pains.
• We didn’t adequately anticipate the level of fraud that we encountered. We were required to make changes to head off fraud problems rather hurriedly, and that led to slapdash implementations that were prone to errors.
• The customer care UI was buggy. I previously did an entire post just about that topic.

Of course, there were business-related challenges, too.

• Most of the executive staff in both companies (Amp’d and BCGI) had a lot of experience with prepaid environments and mobile content, but very little experience in postpaid environments. This was particularly true when Amp’d Mobile was first getting started. By the time Amp’d brought on more seasoned management with post-paid experience, it was already too late. It’s important to note that the postpaid business was about three-quarters of Amp’d’s subscriber base.
• Because many of the key players came from a prepaid background, the enhancements that were requested for the platform often involved marketing content or prepaid services, instead off addressing fundamental business processes like billing and collections. BCGI, itself of prepaid pedigree, was happy to oblige. Thus, the limited development resources were allocated to content, marketing, and prepaid tasks instead of revenue assurance, payment, and other postpaid tasks. Sadly, Voyager had built-in collections modules that were never used by Amp’d because it was never a priority for them. If this had been made a priority in the very beginning, Amp’d might still be in business today.
• People still buy handsets with the intention to talk on them, not to purchase media content. When Amp’d launched, their voice plans were not competitive, and the rate of adoption was low. I think the belief was that the content was a compelling enough reason to sell the phone. However, having a good plan for good old send-to-end voice calls is still critical to attract new subscribers. Once they adjusted their rate plans for this, net add rates improved significantly.

What I Learned
It’s easy to armchair-quarterback the Amp’d experience. I obviously have the benefit of hindsight. Here are some of the things that I took away from the experience:

• As I wrote in an earlier post, business domain knowledge is key. If you’re entering a new business space, make sure you have an expert on your side who knows what the pitfalls will be, from the very beginning.
• Building a national MVNO, particularly one that offers post-paid service, is a lot harder than it sounds. I think that a lot of people mistakenly believe that the only difficult part of building a nationwide wireless carrier is having the network. But building the operational staff to support billing, credit, collections, and customer care is a huge undertaking that takes years to get right at traditional carriers. These critical business processes aren’t cheap, either, even if you manage to outsource them. I’m not sure if an MVNO can be competitive given the cost of using the host network, plus the cost of the back-office processes.
• Always plan for fraud. The bigger you get, the bigger target you become for fraud. Look at every process as a thief would – payment, distribution, service delivery, etc. Everywhere in your system where there is something of value is a target. Plan for it in advance.
• Scalability tests might be less useful than you think they are. It’s hard to predict how things will scale, and which APIs or processes will take the hardest beating. It’s probably safe to just assume that your processes will need to adapt quickly to cope with increased volume, and to be prepared for it by designing your processes to scale horizontally wherever possible.
• I had a blast. This project was one of the most rewarding experiences I’ve ever had as a developer, and it’ll be a long time before I can top it. I worked with some amazing people at Amp’d, BCGI, and a lot of Amp’d’s other vendors. The brainstorming sessions in Amp’d’s amphitheater with the whiteboard is the stuff that developer/architect types love. I still pine for the good old days when I felt like I was a part of something revolutionary in the wireless space.

I’m a big fan of Nate Weiner’s Idea Shower. He recently wrote a really good blog post entitled “What Will the Web See When You Die?” In it, he wrote about the death of a snowboarding colleague, and how the traditional media publications cobbled a rather terse biography of the man by copying some of his profile information from a company website. His post was a good read. Go read it. Check out his other stuff, too, he’s brilliant.

Anyway, the story reminded me of one of my former colleagues who committed suicide a few years ago (in fact, I left a comment about it on Nate’s blog… most of this post is just an expansion of that comment!). He died several years after we had drifted apart (he had moved out to the west coast, and I moved back up to Maine), so I didn’t find out about it until months after it happened.

After I found out, the first thing that I did was go to his web site. There he was, smiling at the camera for a blog entry about how his and his girlfriend’s guacamole dip recipe. I found it rather eerie that his web site stayed up for so long after his death. I found myself re-visiting it days later, perhaps I was half-expecting new content to magically appear. Eventually the site just disappeared into the ether.

At the time I wondered if I should keep a copy of it and host it as sort of a tribute to him, but in the end I decided it was better to let the site go with him; keeping it was like the parents you read about who lose a child and can’t bring themselves to redecorate their room. I doubt he would’ve wanted that.

In this age of caching pages and leaving parts of yourself scattered over the social web, it’s interesting to think that digital bits of your life will live on long after you die. Our lives are far shorter than the magnetic tapes and spinning disks that prop up the web. In fact, my friend’s pages are still in the internet wayback machine, so I didn’t even need to save them.

Photo Credit: ReefRaff