SMSC Address Caching in iPhones

I'm going to make one of my increasingly-rare technical posts today.

When we got back from Burning Man this year we received a few instances of a really interesting bug report. It goes like this: "My friend X and I texted each other on the playa with your system, but now that we are back home, we can't text each other anymore. Texts to and from other people still work. We both have iPhones." After some head-scratching, I recommended a brute-force fix: "Delete all of the messages you exchanged through our system at Burning Man." It worked.

So what happened? When you receive or send a text message via SMS, it has two E.164 addresses. One address, in layer 5, is the mobile phone number of the subscriber sending or receiving the message. The other address, in layer 4, is the SMSC that processes the text message. (SMSCs are to text messages what SMTP servers are to e-mail, and they have E.164 addresses just like telephones.) Normally, the SMSC address for your outgoing text messages is set by your carrier. In some phones, you can also override the carrier's SMSC address and provide one of your own. Here's my working theory: It appears that certain models of iPhone, under some set of conditions, remember the SMSC addresses from which you received a text message and then use that SMSC address for future outbound messages. When you send a text to a given person, these iPhones appear to send the outbound message through the same SMSC used for the most recent message received from that person, or, possibly, the SMSC address used for an error message associated with an attempt to send to that person. Since the OpenBTS system doesn't have a real SMSC address, we were just filling in the SMSC address field in the L4 header with a fake number. When people got home, these iPhones continued to use this fake SMSC address to send texts to anyone from whom they received text messages through our network. Those send attempts failed. When they deleted the messages with the fake SMSC address, everything worked normally again.

So what do we do to prevent this? One solution would be to stick a known SMSC address into the L4 header, instead of the fake one, but that might have unexpected effects of its own. A better solution is to preserve the L4 SMSC address end-to-end, even though we don't use it, which is what we will do in the future.

Live and learn. It is interesting to know, though, that you can use a BTS tool like OpenBTS or OpenBSC to control SMSC settings in a closed device like the iPhone. That probably deserves more investigation.

The Man Burns in 341 Days

We returned from Burning Man over two weeks ago and are still unpacking and recovering. The skin on my hands still feels like old leather, the rebar cuts and blisters on my legs and feet are still healing and Jessica is still sorting through the boxes of dusty camping gear and surplus food in the garage. It was a weird burn; some of our friends and campmates had powerful experiences, good, bad, transformative and moving. Even a weird time in Black Rock City leaves you looking forward to next year, and reminds you that the event is more than just a big party. But that's not what brought you to this blog, so here are the technical highlights:

• We ran a 2-sector, 5-TRX system (3/2 configuration) from a 25 m tower. We ignored RACH bursts with TA>10, limiting our range to 5 km, deliberately excluding nearby towns from the test. Our coverage footprint was roughly 80 sq-km, solid over most of Black Rock City, with the exception of some of the outer streets past 3:00 & I. We could make very good-sounding calls from the airport, Center Camp, the gates and from 9:00 & the trash fence.
• Our second-generation radio worked like a champ.
• Speech calls were limited to three minutes.
• We powered the BTS units from a PV solar array. We had a generator, too, but that was mostly for power tools and the blender.
• We encountered roughly 40,000 unique IMSIs. Really. We were shocked, too.
• We had challenges, even for Burning Man. Our neighbors advised us that Mercury was in a retrograde phase, putting a kind of curse on all communications systems, but we and Papa Legba prevailed.
• It was Tuesday before we had a stable backhaul.
• Commnet Wireless unexpectedly came online on Thursday afternoon. Even more unexpectedly, they did so in the license block for which we had previously been cleared by Verizon. So we lost half a day double checking our license and re-coordinating spectrum with the Commnet NOC.
• We had congestion on Friday and Saturday. We were running nearly twice the capacity as in 2009, but there seemed to be at least twice as many phones in the environment, maybe more.
• We had about 4,000 autoprovisioned users, connected about 7,000 phone calls and processed about 50,000 text messages.

Heros of the Burn for 2010:

• Arturo Mayorga Cerda, for setting up the tower with nothing but hand tools.
• Jessica Burgess, for climbing the tower on the last day to connect the crane.
• Mark Petersen, for staying late to help pack up the camp, even though he was ill and feeling badly and probably should have been in bed resting.

And general thanks for 2010:

• DPW for driving and removing our anchors and for sending the boom truck to take down the tower.
• John Gilmore for supporting smqueue and loaning us a switch, Christmas lights and lots of other goodies. And for bringing some interesting people to the camp.
• Donald Kirker for on-site e-mail and SMS hacking.
• Mark Petersen again for Asterisk support and camp photography.
• Glenn Edens for staying home, manning the office and dealing with the thousands of e-mails we have been receiving as a result of recent press coverage.
• Lisa Hyde for the great 3-minute warning message and our BMIR PSA, which I would like to find a copy of. (If anyone has a copy, that would be great.)
• Jessica Burgess again for rounding up the groceries, organizing the bar and generally keeping the camp organized while we played with computers and radios.
• Tim Bowden for loaning us the solar panels.
• Ralf Muehlen for IP support and general moral support.
• All of the well-wishers who stopped by the camp.

I'll add some photos later, but Donald posted a good set on the Flickr that will do nicely for now. We are already looking forward to next year.

Niue #11: Don't Get Me Wrong...

Don't get me wrong. Despite all the whinging about IUSN, Niue was a good experience over all. Niue is one of the few unspoiled places left on Earth and it was a privilege to go there. For the most part the installation is a success. Most of the people we dealt with were competent, friendly and supportive of the project. The system is up and running. I logged in remotely this morning (with Telecom's permission) and saw some control channel activity:

OpenBTS> chans

TN chan transaction UPFER RSSI TXPWR TXTA DNLEV DNBER

TN type id pct dB dBm sym dBm pct

0 SDCCH/4-0 1804303614 0.00 -61 33 15 -102 0.00

0 SDCCH/4-1 1804303619 0.00 -58 33 8 ----- ------

So at that moment there were two handsets on control channels at distances of roughly 7.5 km and 4.0 km, they were both transmitting at 2 W and the channels were error-free. Not bad for a prototype. We will try to do a software update later this evening to make the task of provisioning a little easier for the Telecom staff and continue to monitor performance as conditions allow.

And Tim's Xorcom box finally arrived, even though Tim himself is back in the UK. Installation has been a little more hairy than expected, but it is happening. When that is complete, Telecom Niue will be able to connect calls between OpenBTS and their wireline switch, which is one more step toward a public mobile network.

Yes, there are still problems and loose ends. This is a test network and nobody expects everything to be perfect at this point. We understand the problems. We have a plan. It might take a few weeks for everything to come together, but it will happen.

There are plenty of technical details that I'm leaving out for now. All of that will be released when it is ready. For now, I want to thank the people who have supported this project, especially Taiichi, Frank and the people at Telecom Niue and in the government. We look forward to working with you all as this project moves forward. And I thank those people of Niue who have show patience and offered kind words, because I know you outnumber those few who were cursing and blaming. Faka'aue lahi. We will do our best for you.

Niue #10: Settling this Nonsense Once and for All

Thursday was our last full day in Niue. Our equipment was turned off the afternoon before because, as best I could tell, a public disinformation campaign from IUSN's operators had lead their subscribers to blame us for widespread WISP outages. I had had my fill of the whole mess and took comfort in the fact that I would be on the next flight out.

That morning, Frank came to guesthouse and ask what my plans were. I said Jessica and I would probably go snorkeling again at Limu and maybe have a picnic, but didn't have any technical work planned. Since everyone was blaming us for IUSN's outages, I would not turn on the equipment unless someone specifically asked me to do so. Frank's response was clear: We were acting with cabinet authorization, at the request of the acting Premier, to test a mobile phone system. There was no higher authority in the country. I should do whatever I thought was reasonable to advance that testing.

I went to the Telecom office and spoke with the Director. We were turning on the GSM system again, but would not announce it yet. We wanted to determine if we were really the cause of the outages. The process would take about an hour.

Sitting at the guesthouse in North Alofi, Harvind started a ping to gatech.edu, a server in Atlanta. 800 ms, no packet loss. I turned on the NS5 at telecom. 800 ms, no packet loss. Harvind turned on the NS5 at the guesthouse. 800 ms, no packet loss.

I drove up to Sekena and called Harvind from the AMPS phone. Still 800 ms, no packet loss. I turned on the NS5 at the tower site. Harvind could get the web interface on the access point. Our whole backhaul network was running. Altanta was still 800 ms, no packet loss. I booted the the BTS and turned on the power amp. Still 800 ms, no packet loss. We waiting another ten minutes. No change.

I drove down to the internet cafe in Alofi, IUSN's retail outlet. I asked, "Now that the mobile stuff is shut down, is everything working again?"

"Yes, just fine."

I asked to be sure, "Is it working right now?" They looked over at a screen and said it was.

We left the system on all day. The Telecom Director called around to people who had been complaining of service outages. There were no problems. We had discovered how to prevent our equipment from interfering with IUSN: Just don't tell them it's on. By the end of the day, we were sitting on the porch in Alofi, making cheap GSM calls overseas and using the internet at the same time. Later that day, the Director sent out an e-mail explaining that we had determined that the GSM system was not causing internet outages. Thanks to IUSN's misinformation campaign, we probably lost a full day of testing and some die-hards out there are still blaming us for everything bad that happens to their internet service.

(Next door to the IUSN/RockET internet cafe, there is a combination bakery and pool hall run by a Kenyan man who lost his passport (in red). The bread he makes is very good for breakfast toast, but molds over fast in the tropical climate. He was enthusiastic about the GSM project and I hope he eventually gets good use of it. This photo has little to do with the blog post, other than proximity, but I'm tossing it in here anyway just to help give a sense of the place.)

Niue #9: Up and Running, for a Little While

On our second Tuesday in Niue, we were finally going to fix our antenna, using the parts fabricated the day before at the government's marine repair shop. We met the Telecom techs at the tower, turned off the BTS PA and broadcast equipment for safety, and got to work.

About half an hour later, we got a call on the AMPS phone at the tower site. It was BCN asking if we had turned off their FM transmitter. It turns out that there had been a miscommunication about which morning we would be working, so this was an unscheduled outage. We explained that the techs were already up the tower and everyone agreed that the safest move was just to let them finish their work. A couple of hours later, the BTS antenna was fixed and everyone was back on the air.

(James Mataele (upper) and Kone Magatogia (lower) installing the antenna mount on the "Chinese TV tower" at Sekena, about 53 meters up. Photos courtesy of Toki Talagi.)

We spent the next afternoon and morning doing some coverage tests. We were still loosing range because of interference from IUSN's US-stye 900 MHz network, but we could use downlink RSSI to estimate what the uplink coverage would look like were IUSN to stop jamming us. It looked like we would have good outdoor coverage in Alofi and all along Alofi Bay down to Halagigie, about 6.5 km from the tower site. Indoor coverage would probably be good in North Alfoi and marginal in most of the rest of town. We had marginal coverage in the hospital parking lot, but none at the airport. There was room for improvement and solid coverage of the populated areas of Niue will definitely require additional sites. That was all in line with the Hata suburban propagation model. The rural model did not apply; the bush vegetation was too dense. Still, in a lot of places in and around Alofi, signals were strong enough for the system to work, even with the interference.

(Harvind at Opaahi Reef, 4.5 km from site, "talking to Allison" at a very strong -65 dBm.)

Back at the Telecom building, Tim was trying to connect OpenBTS to the rest of the world, despite the missing Xorcom analog gateway. Using the new Asterisk-Skype interface, he provisioned a few specific handsets to support calls to a few specific international numbers, just to prove it could be done. For about 2 days, a lucky few of us where making international calls from mobile handsets in Niue at about US$0.03 per minute. Ironically, it was easier (and much cheaper) to call the UK and Japan than to call a wired phone in the same room.

During all of this testing, IUSN's WISP was still just as broken as it had been the week before, except now they had someone to blame. On Wednesday morning, IUSN forwarded me a sample complaint:

"Morning all,

To my surprise my internet is working this morning at 5:30am. I've had no internet connection since Thursday last week. Thanks to the GSM mobile people whom are here on the island doing testing and in the process ...... blocking internet to all users north of the NDB bank and Telecom NIUE. Apparently they put up a machine at Telecom NIUE with the signal beamed at the Makapu tower using the same frequency as IUSN is using. No notification whatsoever - how rude!! They even deny that their machine is blocking internet for some people.... and yesterday even turn off the radio to parts of NIUE without letting the general public and BCN know. Anyway, I hope this will be sorted out today!!"

The reported days and times of the service outage did not correlate with our activities, but IUSN didn't let ignorance and bad facts get in the way of good finger-pointing. By Wednesday afternoon, I was literally getting stopped in the street by angry old men shaking their fists at me and yelling, "Mr. St. Clair says you cut the internet!! Internet very important for this island!!" Disgruntled IUSN subscribers were showing up at the guesthouse to harass us in person. I suspected that there was an active campaign of blame and defamation going on somewhere. (If you wonder why I have no kind words for IUSN's operators...) My suspicions were confirmed when saw this:

To: All Users

Re: Wifi Interference

IUS-N has learned only recently that technical consultants have been on the island for the past two weeks and have been testing a wireless GSM phone system which may have been interrupting your ability to connect with IUS-N's WiFi services over the past few days, in particular, in the Alofi North area. We have learned they will continue to do those tests, sporadically, with no warning, today and possibly in the future.

IUS-N is not able to control the timing of the consultants' tests, nor are the consultants informing IUS-N of the dates or times of these tests.

This email is our warning to users that you likely can expect more unannounced WiFi interference in the Alofi North area today, and possibly in the future, without warning.

This interference may cause connection problems from your location.

If you do experience any problems connecting, or you have in the past few days, please email support@niue.nu with detailed information, to help us keep track of these events.

Regards,

Richard StClair

Technical Manager, IUSN

Around 14:00 Wednesday, to satisfy public complaints, the Director of Telecommunications asked that we shut down all of our equipment. By 14:20 everything was powered off. Around 18:00, internet service was restored in Alofi. So what happened in those three and a half hours? We didn't know. I was still confident that we were not the cause of this week-long internet outage, but open minded enough to want a serious investigation. The problem is that the afternoon's sequence of events didn't provide any solid information about anything.

(While all of this was going on, a cruise ship anchored in Alofi Bay and tended 100 or so German tourists into town. In my best broken German, I greeted them "Guten Tag! Willkommen bei schönes Niue. And you should really be wearing a hat in this sun." Surreal for all involved.)

Niue #8: A Kick in the Pants

Monday started with a meeting with the Minister (acting head of state while the Premier is out of the country) and the principals in the project. What's the status? I told him the system was installed and running, but there were problems:

1. The antenna was at an odd angle because of hardware problems.
2. IUSN's 900 MHz network was interfering with us and limiting our range.
3. Something (probably IUSN) was interfering with our 5 GHz link to the tower.
4. Tim's analog gateway box was still in New Zealand.

We agreed there wasn't much we could do about IUSN or New Zealand's customs office on short notice, but we might fix the antenna problem and still make some calls to the outside world via a VoIP carrier and Telecom's satellite link.

The Minister called the Director of Agriculture and Fisheries. The Director called their marine repair shop and told them to expect us later in the morning. The marine repair shop had welding equipment and good stocks of stainless steel plate and threaded rod. Surely, they could build us an antenna mount. Next stop, Telecom. We wanted to talk to the technicians about the antenna mount to be sure we were building the right thing. This time, Toki and Kone drew on the whiteboard the picture I wish I had seen back in January: a detailed, dimensioned drawing of the TV tower hand rail and their preferred antenna mounting technique. It was a revelation. We went to the fisheries shop and showed the mechanics what we needed. They got to work. A little later, the Minister stopped by to check the progress.

(Dept. of Fisheries marine repair shop. On a remote island, you learn to work with what you have.)

While the shop worked, we played "telephone". Tim provisioned a few handsets with 2xxx numbers, including "2009", a woman with a Fijian SIM who got accidentally included in the test group. (She was amazed when one of us dialed her on a wrong number.) Speech calls were spotty in Alofi, but SMS was working reasonably well and we were texting just because we could.

That evening, we went to the BCN studios for a radio interview and call-in program. One of the signs on the wall said "Less English, More Niuean", so Frank did most of the talking.

The interviewer was kind enough to make notes for me in English summarizing the calls to the program. One stuck out as particularly important for OpenBTS. It was something like, "We have the old AMPS system and we can't fix it when it breaks. We have the Chinese TV system and we can't fix it when it breaks. How will the new mobile system be any different?" That's a darn good question, and the answer, I hope, is a good argument for open designs: Telecom Niue has a full bill of materials for their BTS unit, a complete description of the electronics and all of the source code to the software that is running in it. Telecom Niue has all of the information they need to build another BTS just like the one we left behind, even the names of the vendors who supplied the components to us. And I would hope that if they post to openbts-disucss, people there will help them even if we are not around.

Niue #7: Day of Rest

Sunday started with an IUSN technician coming to the guest house to return the key to the TV tower site and complain that we "blew out" their Trango power supplies by cycling them the day before. That was surprising and I apologized, although I still doubt that's what actually happened. He then went on to complain that he had to work on a weekend and even on his birthday. That was annoying, so I assured him that however inconvenienced he was by us, we where much more inconvenienced by IUSN.

After that little spat, Frank and Taiichi took us on a driving tour of the coastline. Niue is a big chunk of limestone surrounded by a narrow shelf. There are no real beaches, but there are amazing caves and chasms and tide pools 100 yards long and 20-30 feet deep.

In the middle of all that, we stopped at Taiichi's for lunch. We had green coconuts straight from the tree, Cookie grilled some pork and lamb and Frank brought a collection of local foods cooked in an "umu", an underground oven.

Niue #6: Installation

On our second Saturday morning, we met the Telecom technicians at Sekena. We were finally going to install the GSM equipment.

The first immediate glitch was that the u-bolts and pipe we had scavenged would not work. The pipe was too short and the u-bolts were too small. (It would really have been nice to have a mechanical drawing of that safety rail back in January...) The techs said they could probably bolt the antenna directly to the railing, with no pipe or u-bolts at all. At the very least, they could install the antenna cable and reposition the NS5 to shorten its CAT5 run.

(Toki spooling out the cable.)

(Sam hoisting the antenna.)

(Looking up the tower.)

While the Telecom guys worked the tower, we installed the BTS in the rack we had stripped a week earlier. By dumb luck, the 200' LMR-600 cable we brought was exactly the right length.

(Harvind and the newly-installed BTS unit.)

Once the techs cleared the tower, we engaged the power amp. A few handsets started beeping and buzzing as OpenBTS pushed the welcome message into them, but something was wrong. Even right under the tower, speech calls barely worked. Harvind started poking around in the radio layer and announced that we were getting hit with serious interference in the high-end of the downlink spectrum, probably from IUSN's 900 MHz gear. He made some gain and IF adjustments to get the best performance we could manage under the circumstances, but we were still loosing 6 dB of our noise floor. That's a factor of 2 in range and a factor of 4 in subscriber battery life. And the tower techs told us the antenna is at a funky angle because it didn't fit the hand rail very well without the pipe, so we might not even had coverage in Alofi. Arg!!

At this point we wondered: was the interference from IUSN equipment on the tower, or from remote sites beaming at the tower. There was a cabinet on the wall with two POE injectors in it. We had four good work days left on the island and not a lot of time for dodgy e-mails with IUSN's remote managers. Their service was completely unusable that morning, so it seemed unlikely that anyone would even notice a 5-minute link outage. I unplugged the IUSN equipment. The interference want away. I plugged it back in. The interference returned. It wasn't the most prudent thing I've ever done, but now we knew exactly where our problems were coming from.

Despite the interference, on the waterfront in Alofi, about 5 km away, we could get cellular coverage and LOS wifi to the tower site at the same time, so we set up an evening work session in a seaside park to play around with the first real timing advance we'd seen since Burning Man. It's not the worst place I've had to work.

Niue #5: The Gear Arrives!

Friday is airplane day in Niue. The one weekly flight from Auckland arrives a little after noon and about 1/4 of the country turns out to meet it. This is the off-season for tourists, so the flights are running about 1/2 full, mostly shuttling the 30,000-strong Niuean diaspora to and from their ancestral home. My wife, Jessica, was arriving on this flight. So was Taiichi's girlfriend, Cookie. So was our equipment. It was a big day and we were at the airport waiting with everyone else.

After meeting the ladies, we had lunch at Mr. Lee's house, near the airport. Mr. Lee was a Chinese chef who had been stranded in Niue in part of a work-permit scam (long story). He had been fishing the night before and presented us with a wonderful meal assembled from local ingredients. We couldn't stay as long as we liked, though, because our cargo was ready for pick-up and the customs office closed at 16:00, just like everything else.

That evening, we set up the BTS at the guest house to be sure nothing was damaged in shipping. After a visual inspection, we connected the unit to a battery, booted the CPU and then... not much. Asterisk was hanging on DNS because there was no real network to connect it to, so anything that relied on a SIP transaction was timing out. But unlike at Burning Man, we had Tim there and he knew how to fix it. A few minutes later, we placed the first-ever GSM call in Niue. Later that night, Tim made his own blog post, with some photos, too.

Niue #4: License?! We Don't Need No Stinkin' License!

By Wednesday afternoon, pings to California through IUSN showed 95% packet loss. They also showed 10-20 second latency, a sign of some serious network management problems. But by getting up really early, Tim managed to make the project's first blog post from Niue. A few hours later, via Telecom's private network, we got our first e-mail from IUSN telling us that they run a lot of US-style 900 MHz broadband equipment all over Niue and it was too bad that we didn't coordinate with IUSN and New Zealand's infrastructure consultant before we arrived. Their attitude seemed to be that Niue's spectrum is unregulated and they grabbed it first. They claimed that they needed no license, but still took issue with us referring to their system as "unlicensed". At that point, I gave up trying to make sense of their communications.

Silly us. We had been told our client had a license for GSM900. We had coordinated with the government regulator and with the state-owned telco. How foolish of us, not getting permission from some US-based non-profit we had never heard of, not consulting with some advisor from some other government and not expecting to see a ton of US-market ISM-900 gear in ITU region 3, stomping all over the GSM uplink band. We told IUSN that we were surprised to see ISM-900 equipment in Niue, since it is generally illegal outside of North America, and that we were surprised, license or not, that the official regulator had no specific technical information about what they were doing.

So here's the 900 MHz spectrum plan for Niue, as we now understand it:

880-890: Telecom Niue's AMPS WLL downlink, 7 kW EIRP, 2 sites

890-915: GSM900 uplink , up to 1 W EIRP, potentially hundreds of sites

903-928: IUSN's 900 MHz network, 6 W EIRP, ~20 sites

935-960: GSM900 downlink, up to 50 W EIRP, 1 site

IUSN was worried that GSM900 downlink would interfere with their ISM-900 stuff. That was unlikely from the start. ISM-900 radios are designed for unlicensed operation, so they need to tolerate high-power near-band interferers, like public service radios and cell towers. The fact that IUSN's network could coexist with Telecom's scorching hot AMPS sites meant that we were unlike to cause any new problems. But we also knew that ISM-900 gear jams the GSM900 uplink, since they overlap in the 902-915 MHz range ... which is why that stuff is illegal in most of the world. And we could see that cell phones themselves might disrupt IUSN's network, if there were ever enough of them out there. We recommended, as an immediate measure, that IUSN retune their links to avoid operating below 915 MHz within 3 km of the GSM site, at least to the greatest degree possible, and that GSM avoid the 902-915 MHz range. To our knowledge the Niue GSM system does avoid 902-915 MHz, but we never got a response from IUSN. And we don't need one now, since we turned over operation of the GSM site to Telecom Niue when we left.

On Thursday evening, IUSN was already reporting 900 MHz link failures. Our GSM gear was still in Auckland. Powerful stuff, huh?

Niue Episode 3: Linking the Site

Monday morning started with a meeting at Telecom Niue's switching room. All of the administrators had stepped out of the way at this point, so it was us, senior engineer Carlos Tukutama and a handful of technicians and junior engineers. Among them was Toki, who we had met in the airport in Auckland a few days earlier. The switch manager was out sick and couldn't make the meeting, but for the most part, this was the Telecom technical staff. It was time to actually start doing something.

The plan for the week was

1. Establish an IP link between Sekena and the Telecom switch room.
2. Set up an Asterisk box in the switch room.
3. Strip the equipment rack and set up our power supply.
4. Put the antenna mounting hardware in place on the TV tower.

Now, here are two important parts of the story that become the foundation for a lot of what happened over the next week and a half:

First, there is a group called the Internet Users' Society Niue (IUSN) that runs a public WISP on the island, the "free" one that costs NZ$25 to join and blocks outgoing mail and UDP applications. The history of the relationship between IUSN and the Niue gov't is messy, but that's tangential to our story here. We knew they were running a lot of 2.4 GHz gear in Alofi because we could see it on our laptops, but we didn't know much beyond that. As it turns out, neither did anyone else on the island.

Second, we were told that our client had a license for the GSM-900 band. We were also told several times that we were acting with "cabinet authority". According to IUSN's reading of Niue law, "cabinet authority" means that we don't actually need radio licenses. IUSN also claims that since Niue is not a member of the ITU that there is no spectrum regulation, but we were told that Telecom Niue's senior engineer, Carlos, was the official spectrum regulator for the country. I know that Telecom Niue does issue amateur radio licenses; their telephone book says so. It seemed prudent and respectful to coordinate our radio activities through Carlos, whether we had a legal obligation to do so or not. We presumed that IUSN were doing the same.

Back to the narrative...

I showed Carlos our Nanastations and told him that we would need spectrum in the 5.2, 5.3, 5.7 or 5.8 GHz band and that we would prefer the 5.2 GHz band. He said he was not aware of anyone else in the country using that band and we were free to do so. He asked what our fading margin would be. We said we were expecting 20 dB. He said that sounded OK. Easy enough, right? So the first order of business Monday morning was to put up the Nanostations, one on the TV tower and one on the utility mast at the Telecom office.

(James Mataele and Kone Magatogia on Telecom's utility tower.)

Word of the project was spreading. When we got to Sekena Monday afternoon to install the other Nanostation, the BCN TV crew was not far behind.

The Telecom guys put the NS5 on the tower and routed us a cable. We used an unshielded cable because our shielded cable was in a box in Auckland and would have been too short anyway. Tim started some connectivity tests.

(Tim, Frank and Taiichi Fox, the private investor in the project.)

The link was flakey as hell. The first problem was cable length. We fixed that later in the day by cutting out a lot of excess line. The second problem appeared to be interference. One minute we'd have our expected 20 dB margin, the next minute the link would disappear completely. A band scan didn't show any 5 GHz 802.11a systems, but there are plenty of possibilities beyond 802.11a. We tried lots of different frequencies in the 5 GHz band, but there were drop-outs on every one of them. We turned off the NS5s for a while.

We figured that if IUSN were running 5 GHz, surely they would have told Carlos. So maybe the interferer was a non-comms system, something outside of Niue's control, like a mobile radar. We went to the Dept. of Fisheries and asked about marine radars, but they said there were no ships in the area that day. What the heck? Were our radios just broken? Was there a configuration problem? Some resonance from the broadcasting equipment in our unshielded ethernet cable?

We were also on the hunt for u-bolts. We could not get mechanical data on the TV tower before we got here and from what we now understood, we would need some large galvanized u-bolts and a 1.5 meter section of 5 cm pipe. After a day of driving all over the island and collecting several plumbing samples, Taiichi and I found a spare fence post at the airport that looked perfect for a pipe, but the u-bolts were a problem. And everywhere we went, we got the same two questions, "When will my phone work?" and "How much will it cost to call New Zealand?"

Meanwhile, the IUSN WISP was failing badly in Alofi. We considered the possibility that IUSN had 5 GHz gear after all and just never bothered to tell anyone, but the WISP failures did not correlate with the state of our NS5s. By Tuesday, the IUSN service was just unusable, regardless of what we were doing. But to be safe, we stopped by the IUSN ground station and asked the technician there about 5 GHz equipment. He said he had no idea what kind of equipment was out there. Everyone who knew was out of the country. He also said they were having problems with their satellite equipment.

(IUSN's ground station near Avacele.)

Niue Episode 2: Site Prep

After meeting with the government, we took a look at the Telecom Niue switching room, home of a big Redcomm analog switch that serves all of the wireline phones in the country. Tim's mission, should the Xorcom box arrive, is to connect OpenBTS/Asterisk to this switch. In the meantime, Tim is thinking of what else he might connect us to. Telecom's attitude about the Redcomm seems to be that it is big and it is old and it works OK, so don't screw with it. We respect that point of view. When we suggested direct VoIP connections for international calls out of the GSM system, Telecom was skeptical. They were decidedly against anything that would connect their private IP network to the island's public ISP. We respect that, too, and respected it more and more as we started to understand the country's connectivity situation.

Our next stop was the installation site, a hilltop called Sekena, about 1 km south of Makefu village. The Chinese had put a TV tower there as part of a reconstruction package following cyclone Heta in 2004 and we were co-siting with the Broadcasting Company of Niue (BCN). Sekena is the second highest point on the island and an excellent site for a radio systems, so 240 meters to the north, there is a 700 Watt AMPS-850 system being used for WLL service to the north part of the island.

(lat -19.018, long -169.918)

About 2/3 of the way up the tower, 53 meters up, there is a platform with a sturdy handrail. That's where our antenna will go, alongside some existing VHF police radios.

At the tower base there is concrete shed with grid power and air conditioning. BCN gave us use of an old rack that housed a defunct TV repeater. Strip the rack and it's ours. We were feeling a little better about making progress without our cargo, but it was late on Friday and Niueans take their weekends seriously, so we told Telecom that we will have work for their technicians on Monday and we headed back to the guesthouse to get some showers and clean clothes. Then we went to Alofi and paid NZ$25 each to get our laptops provisioned in the "free" wifi system. Ping time to California was about 800 ms, packet loss about 5% and most ports were blocked, including outbound SMTP and every UDP-based application we could think of. (And by Monday, we would think that was really good.)

(Project funder Taiichi Fox helps strip the old TV repeater rack.)

On Saturday, we went back out to Sekena to strip the equipment rack. On Sunday, in proper Niuean style, we took a day off. Our government contact, Frank Sioneholo, took us to his village of Mutalau to see a "hair-cutting" ceremony and the sea cave where his ancestors welcomed the first missionaries to the island.

(Two pickup trucks of slaughtered pigs to celebrate a little boy's first haircut.)

(Frank Sioneholo at the sea cave where his ancestors greeted the first missionaries to Niue.)

Niue Episode 1: A Rough Start

The first hard step of doing anything in Niue is that of actually getting there. It is not near anything. There is one flight per week, from Auckland. Harvind and I flew into Auckland a day early, just to be safe, booked our GSM gear through Air New Zealand's cargo office at the highest priority and met Tim Panton when he arrived a few hours later. Tim had been in transit for over 24 hours already but managed to be in good spirits anyway.

(Auckland)

Our first snag was that some VoIP hardware that was supposed to be waiting for Tim at the hotel wasn't there. Since that gear wasn't absolutely critical to the project and since we had another week to get it through, we didn't worry too much just yet. We left Auckland on a Saturday morning and arrived at Hanan International Airport in Alofi on a Friday afternoon. (Hanan's terminal building is a lot like Black Rock City's terminal building, just with pavement.)

About an hour later we hit our second snag: our GSM cargo, the BTS, antenna and cables, had not been on the airplane. It would be at least a week before we installed a BTS. At least we had a couple of Nanostation-5 radios and plenty of time for site prep, right?

After verifying that our cargo really was stuck in Auckland for another week, we went to a meeting with the acting Premier, the Director of Telecommunications, the project's private funder, the Director of Economic Development and an infrastructure consultant from the New Zealand Commissioner's office. We proposed our newly-improvised schedule for the week, a "prep" schedule intended to allow fast installation of the BTS as soon as it arrives. We talked about risks: the risk of our cargo missing next week's flight, Tim's ideas about how to connect the BTS to anything else if his Xorcom box never shows up, and some concerns we had about the mechanical details the installation site. We had not changed clothes since arriving, so we met the acting head of state in blue jeans and golf shirts. As we walked out someone said we "didn't need to dress up next time".

It was a bad start, but not a disaster. If we could have the installation site, backhaul and PBX ready by Friday and then work through the weekend, we might still have a working GSM system by the next Monday.

FAKALOFA LAHI ATU

"FAKALOFA LAHI ATU! Please respond with your provisioning code..."

There is now an OpenBTS pilot site in Niue, installed with the cooperation of Telecom Niue under a license from the government. The system is still in a closed evaluation, but when the evaluation phase ends the Niue system will probably be the first OpenBTS installation to provide common-carrier service to the general public. This is a very big step for the project and will bring a much-missed service to the residents, many of whom already own GSM handsets when they travel in New Zealand. It will be a learning process for everyone involved.

Installation took two weeks and is still incomplete, mostly due to customs delays in New Zealand and incomplete documentation on the installation site. We also had serious problems coordinating spectrum with a large public wifi system who's operators seem to think that they can use whatever spectrum they want without consulting the regulators. I would have blogged about all of this on the spot, but the public internet service was unusable most of the time we were there. (Naturally, they blamed us. More on that later. UPDATED BONUS: They are STILL BLAMING US.) If you need a blog fix right away though, Tim Panton managed to squeeze a posting out.

The short status summary is this: Telecom Niue's technicians put a 13 dBi sector antenna about 53 meters up on a platform. From there, we should be able to get reasonably good coverage over Alofi, 3-5 km away, once the wifi people quit jamming our uplink with their unlicensed 900 MHz gear. We managed to make a few international calls from cellphones in Alofi and we sent a lot of text messages among ourselves around the island. We look forward to working with Telecom Niue over the next few weeks to get the system better configured and tied-in to their existing wireline switch. The details will follow over the next few days.

I also want to say that most of the people we encountered in Niue were remarkably nice to us and that the natural beauty of the island's coastline is stunning ... even for someone who lives in California.

(Kone Magatogia setting the antenna, 53 meters AGL. Thanks to Toki Talagi for this photo.)

GSM WLLs and Carrier Acceptance

The biggest challenge to the deployment of OpenBTS is that all of the world's cellular spectrum is already licensed, most of it to very big companies.  These big companies don't have strong motivation to deploy low-cost services in rural areas.  First, their actual cost of operation is fairly high in rural areas.  Second, even if that cost of operation could be lowered dramatically, it would create a marketing problem.  Solving the first problem will only magnify the second.

Suppose you're "Big Cellular" and you run a GSM network in the developing world.  It costs you $4-$8 per subscriber per month to operate, costing less in urban areas and more in rural areas.  But the people who actually live in rural areas can only afford about $2/month, so you mostly avoid those areas, unless a major road happens to pass through them, carrying your richer urban customers between cities.  Government regulators may pressure you to serve the rural areas, but you can always just show them your balance sheets and argue (honestly, even) that you are already giving the broadest service that can reasonably be expected for a profitable network.  Everyone's happy -- expect for the rural poor who, will never get telephone service under this model.

This is all cozy until a disruptive technology makes $2/month rural service a real possibility.  If you're Big Cellular, that's not good news.  You already have a legacy network that you're still paying for and the new technology is not directly compatible with it.  Even if it were compatible, the new technology creates a marketing problem because your urban customers paying $12/month will soon be demanding to know why they can't get $2 service like their country cousins.  You can try starting a second brand, but that's very expensive and you fear that your new, cheap brand will simply erode your existing market along the urban-rural edge.

The solution here is to make sure that the new service is not a viable substitute for normal cellular.  I'm not saying give the rural poor broken service.  I'm saying give them what they really need, which is reliable telephone service at a very low price, which is not the same thing as cellular, even if the "subscriber terminal" was built to be a cellphone.

The purpose of the new network is to provide basic telephone service in rural areas.  You don't need full cellular functionality to do that.  For example, maybe you don't implement handovers of active calls between cells.  Maybe you don't allow your rural subscribers to roam into "real" cellular networks.  If you are really cheap, maybe you even bind each SIM to a specific cell site, eliminating all of the mobility management functions.  This functionality already has a name: wireless local loop (WLL).  You use GSM like you might use DECT or WiFi, but with much larger service areas and much cheaper handsets.

Operating in WLL mode offers several advantages in this scenario.  There is the technical advantage of a much simpler core network, although a carrier can still support roaming for conventional cellular subscribers if it chooses.  There is the business advantage of no longer being a direct competitor to legacy cellular networks.  And depending on what country you are in, there may be regulatory advantages as well.

If you are Big Cellular, this new low-cost WLL is not a particular threat to your existing business.  It serves a market you would rather not deal with.  Maybe you can open a new subsidiary to operate WLL networks, or, depending on your local regulations, you can lease your fallow rural spectrum to a WLL carrier.  The WLL becomes a modest source of profit.  Universal service can be someone else's problem while you, Big Cellular, can do what comes naturally: market ever more complex services to the cities and gouge tourists with crazy roaming fees.  Everyone is happy again, and maybe this time we can spread it around a bit more.

What Stuff Costs, Part 2: CAPEX

There are no "list prices" in the global telecom industry. Every purchase is a negotiated deal with the details covered by NDAs. Prices are arbitrary.  How do the equipment providers get away with that? Let's take a look...

Consider the cost of installing a BTS in rural site.  Something like this:

That costs $200k-$250k, depending on what part of the world you're in.  Most of that money is for "civil installation": site prep, concrete pads, backup power, the mast, that little shack, etc.  There's well over $150k worth of stuff there just to support the BTS.  So what should the actual BTS cost?  As long as it's a lot less than the infrastructure cost, the buyer doesn't care because it won't be a significant part of the total site cost. The baseband processors, transceivers, power supplies and amplifiers for a 3-sector 3-TRX ("1/1/1") kit typically run $20k-$50k, depending on the vendor, the buyer, the specific product and whatever side deals the vendor can offer.  That will give 21 Bm channels at full rate.  There's no point in going below $20k because the savings to the carrier are insignificant below that point.  And the price can't go much above $50k before the BTS becomes significant in the total.  Notice that this price range has nothing to do with the actual cost of producing a BTS, as long as that cost is well below $20k.  The total installed cost is around $75k per fielded TRX, or around $11k per Bm channel.

That's the equipment in the field.  You also need a core network.  The core network gets installed carrier-grade data centers.  As long as the equipment costs less than the data centers, prices just don't matter much.  Together, the BSCs, MSCs and location registers in the core network can easily cost over $5k per fielded TRX, or about $700 per Bm channel.  The civil part probably costs twice that, bringing to total to around $15k/TRX or $2,100 per Bm channel.  The core network also creates a floor for a viable network size, since even a "small" MSC is built to support hundreds of cell sites and priced accordingly.

So the rollout cost is around $15k/TRX for electronics and totals around $90k/TRX for a low-density network when you include all of the civil infrastructure.  One TRX can serve about 1,000 subscribers in the developing world so your rollout capital is least $90 per subscriber, not counting counting other costs ignored here.  Note, though, that the dominant cost is civil infrastructure.  Even if the electronics were free, the total capital would not change by more than about 25%.

The only way to dramatically change the cost of a cellular network is to simplify the infrastructure, something that the existing equipment providers have little motivation to do.  For example, if the whole BTS package can be mounted directly onto the mast and left out in the weather, you can get rid of that air conditioned shack.  If you cut the power requirements, you also cut the cost of the backup power systems.  OpenBTS is radical, though, in its approach to the core network: get rid of it and run BTS units as peers.  Don't just reduce the cost of equipment.  Reduce the amount of equipment.

This is one way that OpenBTS hopes to change the economics of rural cellular service: reducing the capital requirements to build a network. The OpenBTS model can reduce the rollout capital from over $90/sub to around $25/sub, not by offering a "cheap BTS" but by eliminating most of the steel and concrete and generators that a conventional GSM network requires.  OpenBTS can also reduce the minimum size of a viable network to something as small as a single cell site, allowing a carrier to start service with an initial capital investment of less than $30k.  Will carriers go for it, though?  Is there any spectrum available for this new kind of carrier to emerge?  We're working on it...

What Stuff Costs, Part 1: OPEX

Most African cellular carriers are partly owned by corporations like Millicom and Vodaphone that are traded on stock exchanges in Europe and America.  They publish regular financial reports.  From those reports we can tell that the typical 2007 African cellular subscriber paid $10-$12/month to talk on the phone for just over half an hour.  That sounds like a rip-off until you do a little more math and realize that it actually cost the carrier about $6/month to provide the service, not counting the cost of internetworking.  What the heck?

Let's say, for simplicity, that all of the traffic is compressed into 6 hours each day, so that you see a load of about 0.003 Erlang per subscriber during this peak traffic time.  A minimum 3-sector GSM BTS site provides about 10.5 Erlangs at 2% blocking and thus serves 3,500 subscribers at your typical daily peak load.  If your cost of operation is $6/sub/mo, that corresponds to a cost of about $252k/year per BTS site to run your network, with most of that cost in the BTS site itself: about $200k/year. (!)  When we first estimated this, we though we'd misplaced a decimal point somewhere.  Then we did we read this article in Balancing Act that put the cost of operating an off-grid BTS site in Africa at around $210k/year.  Then we talked to some telecom people from Africa who said the cost was well over $150k/yr but they didn't know by how much.  So it probably really is around $200k/yr.  Why?

It's all about power.  Suppose you have a BTS that draws 5 kW.  And since it's in the tropics you have to cool it, which brings your power budget up to 7 kW.  To supply that, you need a generator.  And since a generator is a target for theft, you need security lighting and cameras, which drive up your power budget and add at least 1 Mb/s to your backhaul requirement, which requires yet more power.  Before long, the site is drawing over 1o kW continuously and you are burning at least 25 gallons of diesel fuel every day.  Now you need a crew with a truck to drive around fixing generators and fences and filling fuel tanks, which is complicated by the fact that most of these sites aren't even near roads.  It starts looking like war logistics, where Sun Tzu tells us that every sack of rice at the front cost 10 more just to get there.  By the time you have everything in place you're spending nearly $20k/mo to keep this beast running.

This matters a lot to the long term development of these countries, because most of the people who live out in the countryside cannot afford $6/mo for anything, meaning that they will never get telephone service, not even on a non-profit basis.  To achieve universal service, someone will need to try something completely different.

So here's the good news: if you can keep site power consumption down to just a few hundred Watts, this all changes dramatically.  Instead of a generator, you can run the whole site on solar panels or microturbines in many parts of the world.  No more diesel fuel.  No more crews in trucks.  Every two years, you replace the batteries in the power system.  That's all.  That's why the design target for OpenBTS is 75 Watts per transceiver, a target that we are very near already just using off the shelf equipment.

Other other cost components in the subscriber rate are internetworking and capital amortization. Most connections between African carriers happen in Europe. That means that if you call from your MTN cell phone to a wired phone down the street that call may well get routed through France at French long distance rates. And the capital cost of rolling out a rural GSM network is at least $100/subscriber.  But those are topics for other posts.

Putting SMS in the SIP World

SIP/SIMPLE defines two models for messaging.  In the "pager model" (RFC-3428), the message is simply transferred with the MESSAGE method and the receiver responds with "ok" or "accepted".  In the "session model", the two SIP endpoints establish a session with the INVITE procedure, transfer messages within the session, and then close the session with BYE.

SMS naturally follows the paged model.  Radio channels are scarce resources and cannot be held idle for long periods in any practical system.  A typical SMS transaction occurs outside of an established call and the radio channel is held just long enough to move a single message.  Store-and-forward handling is an absolute requirement, too.  When a short message is submitted to the network by a handset, the payload part (the TPDU) must be held until a delivery channel can be established to the destination . Even if the destination handset has service, this can take several seconds. And if the handset does not have service this can take hours or days.

When we first contemplated SMS for a SIP-oriented core network, we thought we could use a messaging server like Jabber to replace the SMSC.  That doesn't work.  Most messaging servers follow the session model.  And why not?  IM applications are session oriented and maintaining an open session in the IP world is very cheap.  These messaging servers are built for chat sessions and presence, not for storing and forwarding one-off messages to intermittently connected handsets.

So the OpenBTS project is off to build its own store-and-forward page-mode messaging server, probably from MySQL and some simple scripts, using Kannel as a gateway to conventional GSM SMSCs.  We'll post more as it happens and welcome any suggestions.

A Lie Agreed Upon: Getting Hybrid Cellular into the Field

I just got back from 25C3 in Berlin. Thanks to Delta, I had a miserable time getting there, but I'm still glad I went. I met a lot of good people. I tried Club-Mate. I tried my best mangled German with the Reisegepäck staff at Flughafen Tegel. On the last night, Harald Welte treated me to a steak dinner, one of the few proper sit-down meals I had on the whole trip.

I met a lot of telecommunications professionals and we discussed the problem of carrier acceptance of the OpenBTS approach: providing a simple set of services at minimal cost and replacing the GSM "core network" with collection of peer-to-peer SIP applications.

There was a general consensus that the OpenBTS approach was technically feasible, even on large scales, and could be integrated into existing GSM core networks if needed.  There was also a general consensus that most incumbent carriers would reject the technology, even if integration into core networks is easy, largely on economic grounds.  The simple truth is that nobody in the telecommunications industry is really interested in making a modest profit by serving large numbers of very poor people.  The typical cellular executive would rather talk about extending 3G or 4G networks into rural Africa and then not do it.  Talking about bringing scorching fast networks to the poor is much more exciting than actually building a 2G system that they can afford to use.  

This consensus is not new to me.  I have had nearly the same conversation several times over the last two years with a handful of ex-employees from African cellular carriers. The big carriers will continue to concentrate on squeezing more revenue from their more affluent customers by offering more complex services. In the meantime, these big carriers will continue to sit on spectrum that is completely unavailable to the people who live under it, or, in the case of South Africa, cover the whole country with services that only a small minority can actually afford to access.  To borrow a phrase from Mark Twain, universal service is "a lie agreed upon" for the telecommunications industry.  It won't happen, even at modestly profitable levels, unless regulators force it.

This does not mean that OpenBTS will not find a commercial market.  It just means that it will not find a market with incumbent commercial carriers until regulators force them to get serious about universal service. That won't happen until early adopters, most likely small rural carriers, use OpenBTS to demonstrate that self-sustaining universal service really is possible.  So we're looking for early adopters.