My first inclination was to try to do this with a JNDI Environment Entry, so I began researching that approach. However, while the EJB spec states in 20.2.4 that the container must “provide a deployment tool that allows the Deployer to set and modify the values of the enterprise bean’s environment entries” (thanks for finding that one, ipage), JBoss does not seem to have such a facility.

Soon I started to wonder if JNDI wasn’t a bit overkill for what I needed to do, anyway.  I didn’t want to specify my parameters on the command-line; I wanted to simplify deployment and wanted to be able to change these values at runtime without restarting the server. But perhaps a System Property was all I needed.

As it turns out, JBoss has the System Properties Management Service for such things.  Here’s what you need to do:

1. Make sure properties-plugin.jar is in your ${JBOSS_HOME}/server/<server>/lib directory.
2. Make sure the properties-service.xml is in your deploy directory (you can find a copy in the “default” server directory)
3. You now have two options, either edit the URLList to have a comma-separated list of locations of properties files, or you can specify your properties directly in properties-service.xml in the <attribute name=”Properties”> element.

Now, to access your property, all you need to do is call the venerable System.getProperty() method.

Photo Credit: Shelley Gibb

In a former life, I designed and did programming for billing systems for “tier three” (i.e., small, regional) wireless carriers. In many ways, it’s saddening to see what the mobile carrier landscape has become in the United States. One of the things that made it fun to work in the industry was the funky, inspired little companies that were created by scrappy hometown entrepreneurs. It took a certain amount of chutzpah and ingenuity to take on the behemoth teleco’s, even if it was a small rural corner of East Podunk, U.S.A.

Lots of people worked for the “Cellular Ones” of the world. I met some folks who got pretty wealthy by building medium-sized businesses by taking on the incumbent wireline carrier in their neck of the woods. These people hailed from small towns like Beckley, West Virginia, or Fort Morgan, Colorado, or Traverse City, Michigan. A lot of the employees were self-trained and didn’t have years of experience as Billing Directors or Network Engineers; sometimes it was so-and-so’s brother-in-law who was appointed to be the “Switch Tech” because he was good with electronic stuff.

I believe that these homegrown businesses are good for America. The people who operate them care about their communities because they see their neighbors every day. They hire local people to staff their call centers instead of outsourcing to distant continents. Their leaders do business with their local friends. These businesses help give their part of the world its own personality.

At one time, I thought that MVNO’s might fill this void – I had hoped that they could supplement the market with their own quirky personalities. But so far, MVNO’s have failed to gain much traction in the U.S.

I realize that some good things will come from consolidation. Bigger companies can often roll out better technology more quickly. It’s harder for, e.g., a small time operation in Decatur, Illinois to roll out 3G data services, than it is for a Goliath like AT&T or Verizon Wireless. But I still mourn the loss of the tier-3′s; for me, they exemplified the American entrepreneurial spirit.

Photo Credit: KB35

I should note that all of these opinions are my own – I don’t speak on behalf of BCGI, Amp’d Mobile, or anybody else.

My Role
Amp’d Mobile contracted with my former employer, Boston Communications Group (usually known by its acronym, BCGI). Among other things, BCGI sold a Postpaid Billing platform called Voyager. Up until Amp’d Mobile, Voyager’s primary market was small, regional wireless carriers with fewer than 100,000 subscribers. Before Amp’d, the Voyager product didn’t contribute much to BCGI’s bottom line and served more as a product portfolio completer than as a real potential for revenue. BCGI’s main focus had instead been on Prepaid Wireless and Payment Services.

My role was primarily to design and create a web service API to Voyager, so that other components in Amp’d's back office could interact with the billing system. This not only included billing for voice and content, but also provisioning prepaid and postpaid services on Verizon’s network, accepting payments, and providing a customer care UI for Amp’d's call center. Additionally, I did some testing to gauge how well the system would scale once Amp’d added the half-million-or-so subscribers that they were projecting they’d have in their first year.

What Went Well
I really don’t intend for this post to be an airing of dirty laundry. Despite the implosion of both companies, there were a few things that went very well.

• BCGI was a fairly stodgy company for its size, with a heritage in larger telecom and wireless companies like Verizon and EDS. Having said that, they gave a lot of latitude to the developers involved with this project. The pace of the development necessitated a more agile approach than BCGI was accustomed to, and the upper management at BCGI did a good job of trying to accommodate the furious speed at which things needed to be done. A lot of the normal paperwork and bureaucracy was pushed aside to get things done more quickly, to everyone’s benefit.
• Some of the development work for the project was offshored, with mixed success. The projects that I’d consider successes related to components that did real-time provisioning of services with the carrier’s network elements. These projects were small, self-contained, with clear objectives that did require a lot of business domain knowledge.
• With a few exceptions, the API scaled quite well. We quickly ramped up to support hundreds of thousands of transactions per day. These came handsets, web sites, payment platforms, etc.
• Integration between our API and Amp’d's web components went very well. Developers from geographically dispersed areas used Instant Messaging quite extensively to communicate, and it was a far more productive means of communication than the old “our developer talks to our project manager who talks to their project manager who talks to their developer” scheme that seems so pervasive in larger companies.

What Didn’t Go Well
There wouldn’t be much to write about if there weren’t a few negatives in the postmortem, right? So here they are; first, the technical Challenges:

• Scaling the batch billing processes were more difficult than I anticipated. When I did my initial testing, I used an existing carrier’s data, and multiplied their user data gradually to project how quickly we’d be able to rate call records with larger carriers. I didn’t count on the fact that my source carrier’s subscribers made far fewer calls per subscriber than Amp’d's subscribers eventually made. Amp’d's market demographic was much younger than the demographic for the source data that I used, and younger subscribers make more calls. In retrospect, this should have been obvious.
• Another scaling difficulty was caused by the manner by which we received call detail records. Mediation is the process by which you obtain raw call records from a network device, and turn those records into a billable records in your billing system. For a traditional carrier, this process takes place daily (at an absolute minimum) or more often as necessary. In Amp’d's case, we received call records once per month. This meant that we needed to process and rate over 50 million call records in a matter of hours, which is no small feat. It’s especially difficult when you’re using a billing system that was designed to mediate only hundreds of thousands of records daily, and bill an order of magnitude fewer records once per month. We managed to scrape by, but there were some pretty severe growing pains.
• We didn’t adequately anticipate the level of fraud that we encountered. We were required to make changes to head off fraud problems rather hurriedly, and that led to slapdash implementations that were prone to errors.
• The customer care UI was buggy. I previously did an entire post just about that topic.

Of course, there were business-related challenges, too.

• Most of the executive staff in both companies (Amp’d and BCGI) had a lot of experience with prepaid environments and mobile content, but very little experience in postpaid environments. This was particularly true when Amp’d Mobile was first getting started. By the time Amp’d brought on more seasoned management with post-paid experience, it was already too late. It’s important to note that the postpaid business was about three-quarters of Amp’d's subscriber base.
• Because many of the key players came from a prepaid background, the enhancements that were requested for the platform often involved marketing content or prepaid services, instead off addressing fundamental business processes like billing and collections. BCGI, itself of prepaid pedigree, was happy to oblige. Thus, the limited development resources were allocated to content, marketing, and prepaid tasks instead of revenue assurance, payment, and other postpaid tasks. Sadly, Voyager had built-in collections modules that were never used by Amp’d because it was never a priority for them. If this had been made a priority in the very beginning, Amp’d might still be in business today.
• People still buy handsets with the intention to talk on them, not to purchase media content. When Amp’d launched, their voice plans were not competitive, and the rate of adoption was low. I think the belief was that the content was a compelling enough reason to sell the phone. However, having a good plan for good old send-to-end voice calls is still critical to attract new subscribers. Once they adjusted their rate plans for this, net add rates improved significantly.

What I Learned
It’s easy to armchair-quarterback the Amp’d experience. I obviously have the benefit of hindsight. Here are some of the things that I took away from the experience:

• As I wrote in an earlier post, business domain knowledge is key. If you’re entering a new business space, make sure you have an expert on your side who knows what the pitfalls will be, from the very beginning.
• Building a national MVNO, particularly one that offers post-paid service, is a lot harder than it sounds. I think that a lot of people mistakenly believe that the only difficult part of building a nationwide wireless carrier is having the network. But building the operational staff to support billing, credit, collections, and customer care is a huge undertaking that takes years to get right at traditional carriers. These critical business processes aren’t cheap, either, even if you manage to outsource them. I’m not sure if an MVNO can be competitive given the cost of using the host network, plus the cost of the back-office processes.
• Always plan for fraud. The bigger you get, the bigger target you become for fraud. Look at every process as a thief would – payment, distribution, service delivery, etc. Everywhere in your system where there is something of value is a target. Plan for it in advance.
• Scalability tests might be less useful than you think they are. It’s hard to predict how things will scale, and which APIs or processes will take the hardest beating. It’s probably safe to just assume that your processes will need to adapt quickly to cope with increased volume, and to be prepared for it by designing your processes to scale horizontally wherever possible.
• I had a blast. This project was one of the most rewarding experiences I’ve ever had as a developer, and it’ll be a long time before I can top it. I worked with some amazing people at Amp’d, BCGI, and a lot of Amp’d's other vendors. The brainstorming sessions in Amp’d's amphitheater with the whiteboard is the stuff that developer/architect types love. I still pine for the good old days when I felt like I was a part of something revolutionary in the wireless space.

But Read What He’s Actually Saying
If you read further, and ignore the provocative headlines, and understand what Beattie is actually saying, he’s not claiming the Mobile Web is dead at all:

“I think anyone currently developing sites using XHTML-MP markup, no Javascript, geared towards cellular connections and two inch screens are simply wasting their time [...]“

This is hardly the death of the mobile web (shame on you, ReadWriteWeb!). All that this means is that the rules of the game have changed.

What “The Mobile Web” Used to Be
What do people mean when they talk about the Mobile Web? I think the usual context of the question is “how can we get people to look at our content from their mobile device?” In the past, simply presenting one’s content on the mobile web was a challenge. Browser technology was abysmal. CSS and JavaScript were out of the question. “Adapted” sites were developed in the crippled WML markup, or other proprietary formats, which made the browsing experience more reminiscent of using Gopher than a real web browser. Services like Mowser and Skweezer were created to adapt the existing, vibrant content into low-fi versions for phones.

What “The Mobile Web” Is Today
What I took away from Russell Beattie’s post was that the need for tools like Mowser and Skweezer have been obviated by advances in mobile browser technology. Does this mean that the mobile web is dead? Of course not! And, as far as I can tell, Beattie wasn’t even claiming that to be the case. Instead, I think the definition of the mobile web is changing.

The internet is comprised of three types of pages (behold my amazing OmniGraffle skills):

There are sites that are intended for viewing on a stationary computer, and sites that are intended for viewing on a mobile device. There is also a huge overlap of pages that work well for both contexts. This is because the barrier of how to present data on a mobile device, which was initially a barrier, has been mostly overcome.

Examples of using sites in the “purple zone” on a mobile device might be

• Wikipedia – looking up a bit of trivia when among friends at a coffee shop (got this one from Tichy on News.YC… thanks, Tichy!)
• IMDB – looking up who the did the voice in the Pixar film that you just saw with your kids.
• Google Maps – Wandering around in Chicago, and you want to know, what the heck is that building?

It’s perhaps the most difficult to define sites in the “red” PC-Only area. I can think of a few:

• Items requiring intensive keyboard work, like word processing or blogging.
• Anything involving heavy graphics, such as online games (and pornographic sites? Perhaps I’m just old-fashioned)
• Number-crunching or mathematical modelling and simulations.

Where Does This Leave the Mobile Web?
So, what is left in the light blue section? It has to do with context. What does someone with a mobile device look at, that a person browsing from a PC or laptop does not? What makes a mobile device unique? Well, duh, it’s mobility!

For years people have been prophecizing the coming of Location Based Services. I think that this area has lots of potential. Companies like Loopt have married social services with location awareness to create a compelling product that started out on Boost and Sprint, and it is supposedly coming to other carriers soon. Similarly, Meetro combines location and Instant Messaging. There are companies who produce audio tours which are location-aware. Fleet management services are exploring the mobile web for business opportunities. It’s not hard to imagine that the next killer app is waiting to be discovered in this space, making our iPhones indispensible. Mobile devices have now advanced to the point where it’s Location-Based Services turn to shine.

The New Mobile Web
The key is context. What do people need from the web when they’re using a mobile device? What do you think?

Graveyard Photo Credit: Qole Pejorian

This is more long-winded than most of my posts, so those with short attention spans should probably turn back now!

Background
I started working for a small company (fewer than 20 employees) in 2000 that provided billing services to small, rural wireless carriers in the United States. That company eventually grew to about sixty employees and was acquired by another medium-sized company about three years later. The new company provided other services to wireless companies, but lacked a post-paid billing platform. It seemed like a perfect fit. The smaller billing company would become a satellite office for the corporate mothership.

A Corporate Culture Shift
In the past, the team working on the billing system had been allowed a lot of latitude to work independently from the corporate mothership. However, this was a high-profile project with a large national carrier, so the company decided to bring in a lot of its leadership’s “big guns,” who had experience with large carriers. Unfortunately, they had little or no experience in the business domain, namely post-paid wireless billing.

Unlike the small team that had managed the product in the past, the new folks had little sense of what the carrier’s priorities should be. This new team included client relations representatives, many of the project managers, and even a majority of the technical team. The people who had dutifully supported the small billing program for so many years would continue to work in “maintenance mode” on the existing platform for the existing rural clients. The implication was clear – they’d been relegated to the “less important” clients.

The User Interface
The billing system had a user interface which allowed representatives at the customer call centers to change things for the subscriber in real time – things like rate plans, billing addresses, automatic payment, allowed managers to configure their own rate plans and user privileges and roles, and it even had an integrated Point-of-Sale system.

The system was “good enough,” but barely. Unfortunately, it was written using a poorly supported, obscure 4/GL whose scalability was in doubt. Deploying it at carrier sites also created some sketchy security issues. We had been meaning to re-write it for a long time, but didn’t have the resources to do so. It wasn’t going to be good enough for our new client.

Offshoring to the Rescue
At about that time, the parent company acquired another company in India that processed payments for wireless carriers. Suddenly we had nearly limitless programming resources at our disposal. All we’d need to do is mock up some new screens, write a few specs, and send the work to our new development staff. We’d “keep a close eye” on their progress, strip the functionality down to the bare necessities, and we’d have a re-vamped UI in a matter of months. As an added bonus, the new programmers already knew Java, so we wouldn’t need to re-train any of the existing workforce to get started.

The first thing the new leadership did was to consult with a UI expert to do the mockups – one that (of course) had no experience with the product or with wireless carriers. She was mentored by a product manager who had very little experience with post-paid billing.

Fortunately for me, I got to work on a SOAP-based API for the product to integrate with the other elements of the back-end. I saw the train wreck coming with the UI project, and I managed to avoid it. I worked quite independently with another contractor based near the corporate mothership.

Reviewing the Mockup
The first thing we did was review the new UI mockups. The new leadership had the right idea, and decided that it might be a good to include some people familiar with the “legacy” system in the review. The legacy team was horrified. The new application looked completely dysfunctional. Naturally, the “new team” got defensive and insisted that the crufty old-timers were thinking too much in the past, that this was the future. Their comments fell on deaf ears. They weren’t invited to participate any more.

Developing the UI
So here’s how development worked: The company hired some brand-new Java superstar contractors to work in their corporate office to “oversee” the development effort. The contractors were great Java programmers, but they had literally zero business domain knowledge. Every morning, they would get onto a conference call with India to review the previous night’s work and discuss the next day’s priorities.

Nobody on the project knew if they were doing it right. There was no connection between the back-end and the wireframes. They started consulting me with business-related questions, and I helped out where I could, but that ended up being like the game of telephone that you play in grade school – things are lost when they are translated three times over, and communicated over a garbled conference call and IM. The rest of the legacy team was too busy supporting the existing customers to help answer questions.

So domain knowledge continued to drip very slowly from wherever it could, over to the mothership, and finally to India. Too slowly. They began to sit idly on a UI that, because it followed the mockups religiously, was fairly unusable.

Eventually, the developers on the other side of the ocean started to get fed up, and were leaving the company. I couldn’t blame them, and the fact that there are so many jobs available in that market made it easy for them to leave. What little domain knowledge had been built up was constantly fading away. They began hiring furiously, which compounded the problem further.

I heard complaints from the superstar contractors that the quality of the work was a bit sketchy. I imagine that the quality of the work must have worsened as the rate of turnover increased. We started bringing on extra contractors stateside to help out. Have you ever read Brooks’ The Mythical Man Month? Yeah, he knew what he was talking about. Adding developers to a late project makes it later.

One night I was visiting the mothership, as I had to do almost weekly to keep up with the latest news on the project (it was about an hour-long drive from our satellite office). We were weeks away from the launch date. The lead UI developer called me into his cubicle and asked if I could spare a few minutes. We started reviewing the UI, and he began asking me how to wire up the pages with the back-end. We looked at page after page and I told him “you can’t do that – it doesn’t make sense,” or “we have no feature like that, I don’t know what it is supposed to mean.”

We were supposed to use my SOAP API to do the business logic, but they had laid none of the groundwork for it. The programmers were pretty good at calling stored procedures. So, on page after page, I told him I’d write a stored procedure to do the work. Add a rate plan? I’ll throw together a procedure. View the customer detail? Yup, that too. We started eliminating unnecessary pages, buttons, and fields left and right.

The next day I started hacking frantically on a raft of stored procedures that did most of the back-end logic for the UI, and I did a pretty mediocre job. I was in a hurried marathon session, and I like to believe that I did the best that I could given the circumstances, but at that point it was definitely all about volume, not quality. Work on the API was pretty much halted, but it was nearly finished anyways. Soon it was time to start training the call center.

Was the Customer Happy?
Fortunately, the client had problems of their own with ramping up. Their operations people were initially satisfied with the product because they had very little experience in the business domain, either. But it didn’t last.

As time went on, and more experienced management was hired by the carrier, they became incredulous that the UI couldn’t perform the simplest of tasks that competitor’s systems could. They were frustrated that it was so buggy. However, the crappy Customer Care UI ended up being the least of the carrier’s problems – that story is for another day. Eventually, both companies more-or-less crashed and burned, which is why I hope it’s safe to share this story with you now.

Takeaways and Lessons that I Learned
So, here are the things that I can impart based on what happened to this project:

• Domain Knowledge is king. Programmers are a tenacious bunch, and we can learn technology fairly well. Years of insight from business relationships can’t be learned overnight. The mothership had the roles reversed – the big guns with experience with big carriers should have been mentoring the employees with the domain knowledge, instead of the other way around.
• Working with people half a world away is hard. I don’t think that it had anything to do with the fact that the offshore programmers were in India. I think it would have been just as difficult if they had been Canadian or Australian or any other English-speaking country. India has plenty of talented programmers. But sharing domain knowledge becomes even harder over conference calls, Skype, and IM. When developing an application in a short timespan, you need to be able to holler questions over the cubicle wall, or grab a whiteboard on a moment’s notice, which isn’t an option when offshoring. I’m sure the Indian team was just as frustrated as we were. I know that there are companies out there who have been successful at this, but it’s difficult to pull off, and our company didn’t manage to do it. There are probably tools out there to make this easier – do you know of any that you can recommend?
• The User Interface is the Face of the Application. Your UI probably needs the most attention, and probably gets the least. Your client’s perception of the quality of your product and engineering is largely based on what they see. Our billing back-end was quite sophisticated and managed to scale well enough, but the client didn’t see that. They saw the crummy exception messages and the things that they couldn’t do from the interface. You can often fake the back-end for a while by throwing more people and process at fixing problems – you can’t easily fake quality in your UI. To the client, our whole product was crap because the quality of the UI was poor, and this should be obvious to everyone. Have you ever worked with a pathetic UI, and concluded that the company did a shoddy job of engineering?

Phew! What a long post. Thanks for sticking it out and reading the whole thing. I’d be interested in hearing if anyone else has gone through a similar situation, and their company found ways to overcome the knowledge gap. Have you been in an acquisition where your formerly tightknit team had to integrate and ramp up within a new corporate culture? How’d you get through it?