The OpenBTS Chronicles

UMTS: Truly, you have a dizzying intellect

2011-12-08T12:24:00.000-08:00

We have spent a lot of time in recent months studying the specifications for "Uu", the UMTS 3G subscriber interface. The spec is an obfuscated mess and the experts who publish books seem to disagree on a lot of critical details. In fact, the literature is so inconsistent that we are starting a public wiki-based documentation project.

In its basic form (non-HSPA), UMTS can deliver 384 kb/s per channel with up to 4 such channels active at once, assuming good link margins, small delay spreads, good power control, proper phase of moon, low traffic levels in surrounding cells, and generally clean living and happy thoughts on the part of everyone involved. That's just over 1.5 Mb/s on a really, really good day. To do this, UMTS musters just about all of the fancy math that the 1990's had to offer: orthogonal spreading codes (at least, orthogonal until they hit the real world), turbo codes, rake receivers, multisensor diversity demodulators. In exchange for all of this complexity you get roughly half (yes, half) the usable bit rate that you could get from an EDGE-style PSK/TDMA interface in the same bandwidth, except that the PSK/TDMA approach would provide more solid QoS guarantees, use a lot fewer transistors and be more power-efficient due to smaller crest factors in the amplifiers.

So, you might ask, why all of the new complexity if we get no performance advantage? Because simple approaches don't produce enough new intellectual property to keep the usual suspects in business. If you just take the GSM/GPRS/EDGE specification and change some parameters (like symbol rate and number of timeslots) you can get a very efficient, flexible system, but you won't have a much fodder for IEEE papers. Worse yet, you don't generate a lot of new patents, and with a lot of the patents on 2.xG systems expiring early in this decade, the NEPs needed a new gravy train. This complexity also deters small players from building their own UMTS implementations. (We have taken that as a challenge, but then we are unreasonable people.)

And what did UMTS do for the carriers? It nearly killed them. They overbid for the spectrum, barely had enough money left to roll out these really expensive new networks and when it was over they discovered that all their customers really wanted was better coverage and lower bills, neither of which have anything to do with UMTS. (The iPhone later proved that lack of demand for data was mostly due to the carriers' walled-garden approach to the itnernet, but that's fodder for a different post.) The killer app for the next five years turned out to be text messaging and economists got to write lots of papers saying things like "We conclude that the rationalization of bidding in the United Kingdom’s UMTS auction remains problematic." And just as the carriers are starting to recover from UMTS, the usual suspects start pushing the next shiny, new thing: LTE/IMS.

Burning Man 2011 - Yes we were there.

2011-09-06T23:09:00.000-07:00

For those who are still wondering, we did go to Burning Man. We ran a DCS1800 cellular network based on OpenBTS. We actually posted our plans a couple of months ago in a pubic wiki, but if you don't follow Jackrabbit Speaks, the Burning Man newsletter, you probably didn't know that. This year, the network was a real collaboration. The participants were:

Range Networks, Inc., the San Francsico-based company that actually produces commercial basestation units based on OpenBTS. (That's us!)
MedWeb, Inc., a San Francisco-based telemedicine company involved in disaster response and humanitarian projects in several countries. MedWeb provided a lot of our technical infrastructure this year, in terms of mast trucks, satellite links and power systems.
The UC Berkeley TIER group, a research group focused on information technology for humanitarian and development projects. TIER people brought some experimental applications to run on top of the OpenBTS VoIP network and also provided a lot of labor for the deployment and tear-down.
Voxeo, a VoIP services company. Voxeo provided a lot of our off-playa call routing, along with some very clever call routing services.

The other big difference from previous years was a that we ran a 3-site network instead of a single large site, making our 2011 plan considerably more ambitious than those of previous years. This year we also supported inbound call routing for everyone using a slick "NAT for telephone numbers" trick from Voxeo. We passed out expired China Mobile SIMs so that we would avoid accidentally catching handsets from the Commnet commercial network and we passed out about 150 Nokia C1 knock-offs for people who didn't have unlocked phones. Between the SIM requirement and the use of the DCS1800 band, we were deliberately limiting the user set to people who intentionally sought out our service, avoiding a lot of the congestion problems that has set in by Thursday or Friday on previous years.

So how did it go? There were the usual aggravations and delays associated with any complex operation at Burning Man, but in the end the system worked and we had about 200 provisioned users. As usual, we learned a lot along the way. A lot of information is out there already, so instead of repeating it all, I'm just going to link to some of the more meaningful existing material.

We will post some performance statistics once we are finished compiling them. We are already planning next year's system.

Making GSM Future-Compatible

2011-02-08T12:41:00.000-08:00

Over the last few weeks, I have been reading through the 3GPP IMS specifications. IMS is the core network for next-generation 4G/LTE mobile data and telephony. Going through the specs is more like bush-whacking than reading; I still can't look at most of the network diagrams without getting dizzy. But I am starting to get a feel for it. In it's essence, IMS is a SIP core network for cellular. Granted, it still looks way more complicated than it needs to be to serve that function, but that's what it is.

Lucky for us, one of the key ideas of OpenBTS is to also use a SIP core network for cellular. So in terms of core networks, we are about five years ahead of the industry, even if the air interface is Um or Uu. We expect the commercial release of OpenBTS to "just plug in" to IMS core networks within a few weeks. IMS compatibility has two big implications for OpenBTS moving forward.

First, it means that there is an application for OpenBTS in incumbent carrier networks that are moving to 4G in the next few years. I've had the opportunity to talk to executives and network engineers from a few carriers who are planning their 4G transitions and have heard the same story over and over. Here it is: "The 4G rollout is expensive, but the performance improvements justify the cost. Except in rural areas, where the subscriber density is too low to justify the expense. But if we keep running GSM/EPGRS or 3G in those areas, then we will have to continue running the old SS7-style core network in addition to the new IMS core network. So we either waste money running two core networks or we waste money installing 4G basestations in the middle of nowhere." The OpenBTS approach offers a solution: Refit your rural sites with an inexpensive OpenBTS-based RAN and then turn off all those BSCs and MSCs.

Second, it eases the minds of carriers looking at greenfield rollouts in the developing world. These carriers need inexpensive networks, but don't want to feel like they are installing obsolete technology. Installing some low-end BTS/BSC/MSC combination just because it's cheap is installing obsolete technology because it will saddle you with an end-of-life core network that you will need to continue to support for years. Sure, you might run your circuit-switched protocols over 802.whatever, a la SIGTRAN, but all that means is that you're not completely stupid; Abis-over-IP is so 1998. On the other hand, installing an IMS-compatible OpenBTS-based network is a first step toward 4G, even if the initial rollout only supports 2G handsets. When the future arrives in your corner of the world, you'll be ready.

SMSC Address Caching in iPhones

2010-10-30T16:11:00.001-07:00

I'm going to make one of my increasingly-rare technical posts today.

When we got back from Burning Man this year we received a few instances of a really interesting bug report. It goes like this: "My friend X and I texted each other on the playa with your system, but now that we are back home, we can't text each other anymore. Texts to and from other people still work. We both have iPhones." After some head-scratching, I recommended a brute-force fix: "Delete all of the messages you exchanged through our system at Burning Man." It worked.

So what happened? When you receive or send a text message via SMS, it has two E.164 addresses. One address, in layer 5, is the mobile phone number of the subscriber sending or receiving the message. The other address, in layer 4, is the SMSC that processes the text message. (SMSCs are to text messages what SMTP servers are to e-mail, and they have E.164 addresses just like telephones.) Normally, the SMSC address for your outgoing text messages is set by your carrier. In some phones, you can also override the carrier's SMSC address and provide one of your own. Here's my working theory: It appears that certain models of iPhone, under some set of conditions, remember the SMSC addresses from which you received a text message and then use that SMSC address for future outbound messages. When you send a text to a given person, these iPhones appear to send the outbound message through the same SMSC used for the most recent message received from that person, or, possibly, the SMSC address used for an error message associated with an attempt to send to that person. Since the OpenBTS system doesn't have a real SMSC address, we were just filling in the SMSC address field in the L4 header with a fake number. When people got home, these iPhones continued to use this fake SMSC address to send texts to anyone from whom they received text messages through our network. Those send attempts failed. When they deleted the messages with the fake SMSC address, everything worked normally again.

So what do we do to prevent this? One solution would be to stick a known SMSC address into the L4 header, instead of the fake one, but that might have unexpected effects of its own. A better solution is to preserve the L4 SMSC address end-to-end, even though we don't use it, which is what we will do in the future.

Live and learn. It is interesting to know, though, that you can use a BTS tool like OpenBTS or OpenBSC to control SMSC settings in a closed device like the iPhone. That probably deserves more investigation.

GSM Security Workshop in Lucerne

2010-10-28T16:54:00.000-07:00

I should have posted something about this a few weeks ago, but next week Harald Welte, Karsten Nohl and I will be presenting a workshop on GSM security at the Lucerne Hashdays conference, the Swiss counterpart to DEFCON. Under a test license from the Swiss regulator, we will be demonstrating a range of GSM attacks and countermeasures against them, using systems derived from OpenBSC, OpenBTS and OsmocomBB. Online registration is closed already, but there is still space available. This workshop should be especially illuminating for journalists, aid workers, diplomatic and corporate security specialists and anyone else concerned about the risks associated with using mobile handsets in unfriendly countries.

And if you are already attending, I look forward to seeing you there.

The Man Burns in 341 Days

2010-09-25T22:33:00.000-07:00

We returned from Burning Man over two weeks ago and are still unpacking and recovering. The skin on my hands still feels like old leather, the rebar cuts and blisters on my legs and feet are still healing and Jessica is still sorting through the boxes of dusty camping gear and surplus food in the garage. It was a weird burn; some of our friends and campmates had powerful experiences, good, bad, transformative and moving. Even a weird time in Black Rock City leaves you looking forward to next year, and reminds you that the event is more than just a big party. But that's not what brought you to this blog, so here are the technical highlights:

We ran a 2-sector, 5-TRX system (3/2 configuration) from a 25 m tower. We ignored RACH bursts with TA>10, limiting our range to 5 km, deliberately excluding nearby towns from the test. Our coverage footprint was roughly 80 sq-km, solid over most of Black Rock City, with the exception of some of the outer streets past 3:00 & I. We could make very good-sounding calls from the airport, Center Camp, the gates and from 9:00 & the trash fence.
Our second-generation radio worked like a champ.
Speech calls were limited to three minutes.
We powered the BTS units from a PV solar array. We had a generator, too, but that was mostly for power tools and the blender.
We encountered roughly 40,000 unique IMSIs. Really. We were shocked, too.
We had challenges, even for Burning Man. Our neighbors advised us that Mercury was in a retrograde phase, putting a kind of curse on all communications systems, but we and Papa Legba prevailed.
It was Tuesday before we had a stable backhaul.
Commnet Wireless unexpectedly came online on Thursday afternoon. Even more unexpectedly, they did so in the license block for which we had previously been cleared by Verizon. So we lost half a day double checking our license and re-coordinating spectrum with the Commnet NOC.
We had congestion on Friday and Saturday. We were running nearly twice the capacity as in 2009, but there seemed to be at least twice as many phones in the environment, maybe more.
We had about 4,000 autoprovisioned users, connected about 7,000 phone calls and processed about 50,000 text messages.

Heros of the Burn for 2010:

Arturo Mayorga Cerda, for setting up the tower with nothing but hand tools.
Jessica Burgess, for climbing the tower on the last day to connect the crane.
Mark Petersen, for staying late to help pack up the camp, even though he was ill and feeling badly and probably should have been in bed resting.

And general thanks for 2010:

DPW for driving and removing our anchors and for sending the boom truck to take down the tower.
John Gilmore for supporting smqueue and loaning us a switch, Christmas lights and lots of other goodies. And for bringing some interesting people to the camp.
Donald Kirker for on-site e-mail and SMS hacking.
Mark Petersen again for Asterisk support and camp photography.
Glenn Edens for staying home, manning the office and dealing with the thousands of e-mails we have been receiving as a result of recent press coverage.
Lisa Hyde for the great 3-minute warning message and our BMIR PSA, which I would like to find a copy of. (If anyone has a copy, that would be great.)
Jessica Burgess again for rounding up the groceries, organizing the bar and generally keeping the camp organized while we played with computers and radios.
Tim Bowden for loaning us the solar panels.
Ralf Muehlen for IP support and general moral support.
All of the well-wishers who stopped by the camp.

I'll add some photos later, but Donald posted a good set on the Flickr that will do nicely for now. We are already looking forward to next year.

Taking it on the Road

2010-08-24T22:15:00.000-07:00

So my garage is full of groceries and in front of Harvind's house there's a 16' moving van loaded up with oversized tents, radio tower segments and jugs of water. It must be time for Burning Man.

So what's new with OpenBTS at Burning Man this year?

First, we are moving into multi-ARFCN systems. Last year, we ran a 3-sector system with 1 ARFCN per sector. This year, we will run two sectors with 3 ARFCNs per sector. So with 33% less equipment and about half as much power, we will double our capacity.

Second, we are using new hardware packaging. This will be our first real-world deployment of our second generation radio. In bench testing so far, this new radio is giving considerably better performance than the old USRP + RFX hardware, and at considerably smaller size, power consumption and cost. The BTS packaging is also more compact than 2009, using a 4U rack-mount chassis similar to the one used in Niue, placed in a weatherproof travel rack recycled from thePfarrkirchen workshop. And we are running a 90' tower instead of a 70' tower, using LMR-600 instead of LMR-400. Those changes should give an effective improvement of about 3 dB in overall performance.

(Harvind running one of the BTS units though its paces with the CMD-57.)

Third, we are hoping to have the air to ourselves. We know that Commnet will not be running a portable site from Frog Pond again. We have heard that they have a fulltime site in Gerlach, but hopeful that the coverage and capacity of that site will be too small to cause a lot of the complications we experienced last year. Here's to hoping. We'll see what happens when we get there.

Fourth, we will actually and deliberately route calls to the outside world. We have not done that before but it will probably make the system a lot more useful. We will, however, time-limit calls to prevent abuses. We will not be routing inbound calls, which will do a lot to preserve the social nature of the event.

We hit the road tomorrow and arrive on the playa at 4:30 & G on Thursday morning. Drop by for a visit if you are in the neighborhood. Just look for the tower.

A First Taste of Peru

2010-08-24T20:40:00.000-07:00

I recently returned from the Latin American Workshop on Wireless Networks for Rural Areas in Ica, Peru. Due to a very tight travel schedule this month, I spent more time on airplanes and buses than at the actual workshop, but I am very glad I went. The workshop brought together researchers, regulators and carriers to discuss current barriers to rural telecommunications service and review the activities of some exciting projects, especially in Peru and Brazil.

My stay in Peru was short but enjoyable and the workshop organizers demonstrated remarkable hospitality. My first impression is that Peru is a fascinating land, inhabited by hard-working people who are rightfully proud of their heritage. I hope to visit again.

Dieter Spaar Goes Public

2010-08-03T10:43:00.000-07:00

Dieter Spaar has started a blog. For those who don't know about Hr Spaar, he is a consulting engineer in Germany and a serious contributor to the OpenBSC and OsmocomBB projects. Using tools from those projects, along with some of his private work, Dieter sometimes does very useful experiments that reveal security shortcomings in cellular handsets and network equipment. Dieter is a modest, quiet man and sometimes other people go out and grandstand and get a lot of attention showing off his work. I am glad to see him speaking up.

The Man Burns in 35 Days

2010-08-01T15:42:00.000-07:00

It's that time of year again and preparations are under way for Burning Man 2010. The enclosures are at the machine shop, groceries are stacking up in the garage and we have our early arrival passes. We are posting the official public information here and I'm sure I will have more to say as the project develops.

Pfarrkirchen

2010-07-08T08:45:00.000-07:00

I returned from the Pfarrkirchen OpenBTS workshop just over week ago and am finally sitting down to write about it. I hope everyone enjoyed themselves and learned a lot, which certainly appeared to be the case. Dieter Spaar offered his farm for event and made excellent preparations. If you want to see some photos, Ali Gündüz posted some good ones on his blog. We also discovered a new way to test the robustness of our mechanical packaging: Check a BTS unit onto United Airlines and see what's left of it when you get it back. Their baggage handlers managed to slam one of our cases against the ground so hard that they broke the rack-mounting tabs on our Elma 2U boxes and even damaged the internal elements of a cavity filter.

The general form of the workshop was to divide the class into small teams and assign each team a "pet" OpenBTS unit to configure and hack over the next three days. The material was broken into topic-oriented sessions, starting with an overview of the physical layer and ending with a multi-BTS network carrying phone calls and text messages. The presentation style was usually 30-45 minutes of theory followed by a live demonstration or class exercise on each team's BTS. The operating area of the network was a little smaller than hoped, mostly due to mud, but there's not much than can be done to control the weather and I think everyone was understanding about that.

(Workshop class session: SMS handling. Thanks to Mark P. for this photo.)

(We ate a hearty Bavarian menu, catered to the farm. Thanks to Mark P. for this photo.)

One of the things I appreciated most about this workshop was the collection of attendees. They came from a range of backgrounds: network operators, academics, development specialists, security specialists -- a very broad mix. Together, this group produced some really great discussions during the breaks and meals. I felt very fortunate to have met them all in person and in one place. Plus, Harald Welte and Karsten Nohl came out to the farm as well to participate in the discussions and present some short demos of OpenBSC and OsmocomBB. Harald even (unintentionally) fuzzed our systems with malformed messages from one of his test phones, something that I now plan to make part of our regular testing.

(Downlink signal strength from the BTS unit used by Ali's team. See his blog for a photo of the actual unit. Thanks to Dieter for taking these measurements and making this map.)

Did Anyone Else Notice That?

2010-04-26T19:02:00.000-07:00

There are 4 variants of GSM's A5 stream cipher, the algorithm used to secure subscriber connections over the radio link:

A5/0. No encryption at all.
A5/1. "Stronger" encryption. This option is supposedly provided only on basestation equipment delivered in North America and Europe.
A5/2. Weak encryption, exported to the rest of the world. The GSMA declared A5/2 obsolete in 2006 and mandated that it be phased out in networks and not supported in new handsets. I think.
A5/3. True strong encryption, being phased in for UMTS systems and proposed as an upgrade for GSM systems.

My first question for anyone who happens to be reading this: What, specifically, was the GSMA's 2006 mandate regarding A5/2? Can anyone point me to an authoritative document?

My second question is more vexing. Let's assume for a minute that I really do have my facts right. Suppose you are using a post-2006 GSM phone somewhere outside of North America and Europe. The carrier's equipment isn't supposed to support A5/1. Your handset isn't supposed to support A5/2. A5/3 isn't available yet in non-UMTS networks. So what A5 variant are you using?

Catch a Demo at eComm

2010-04-09T18:41:00.000-07:00

The eComm conference starts on April 19 in San Francisco. I'm speaking. Last year was my first year there and it was a great conference for anyone interested in future directions and new thinking in telecommunications. I met a lot of people there who turned out to be very important for the project over the next year, like Tim Panton, who was our initial contact for Niue, or Rob Ullens from Voxbone who helped sponsor our Burning Man 2009 project. I will be in more familiar company this time around, and very much looking forward to it.

I made the first public demo of an OpenBTS node at last year's conference. But that's old news, the speakers' slots are short and I have a lot of new things to talk about. I'm bringing the equipment, so if you are at eComm and want to see an OpenBTS demo, hunt me down and we can arrange something.

Niue #11: Don't Get Me Wrong...

2010-03-26T19:00:00.000-07:00

Don't get me wrong. Despite all the whinging about IUSN, Niue was a good experience over all. Niue is one of the few unspoiled places left on Earth and it was a privilege to go there. For the most part the installation is a success. Most of the people we dealt with were competent, friendly and supportive of the project. The system is up and running. I logged in remotely this morning (with Telecom's permission) and saw some control channel activity:

OpenBTS> chans

TN chan transaction UPFER RSSI TXPWR TXTA DNLEV DNBER

TN type id pct dB dBm sym dBm pct

0 SDCCH/4-0 1804303614 0.00 -61 33 15 -102 0.00

0 SDCCH/4-1 1804303619 0.00 -58 33 8 ----- ------

So at that moment there were two handsets on control channels at distances of roughly 7.5 km and 4.0 km, they were both transmitting at 2 W and the channels were error-free. Not bad for a prototype. We will try to do a software update later this evening to make the task of provisioning a little easier for the Telecom staff and continue to monitor performance as conditions allow.

And Tim's Xorcom box finally arrived, even though Tim himself is back in the UK. Installation has been a little more hairy than expected, but it is happening. When that is complete, Telecom Niue will be able to connect calls between OpenBTS and their wireline switch, which is one more step toward a public mobile network.

Yes, there are still problems and loose ends. This is a test network and nobody expects everything to be perfect at this point. We understand the problems. We have a plan. It might take a few weeks for everything to come together, but it will happen.

There are plenty of technical details that I'm leaving out for now. All of that will be released when it is ready. For now, I want to thank the people who have supported this project, especially Taiichi, Frank and the people at Telecom Niue and in the government. We look forward to working with you all as this project moves forward. And I thank those people of Niue who have show patience and offered kind words, because I know you outnumber those few who were cursing and blaming. Faka'aue lahi. We will do our best for you.

Niue #10: Settling this Nonsense Once and for All

2010-03-24T18:33:00.000-07:00

Thursday was our last full day in Niue. Our equipment was turned off the afternoon before because, as best I could tell, a public disinformation campaign from IUSN's operators had lead their subscribers to blame us for widespread WISP outages. I had had my fill of the whole mess and took comfort in the fact that I would be on the next flight out.

That morning, Frank came to guesthouse and ask what my plans were. I said Jessica and I would probably go snorkeling again at Limu and maybe have a picnic, but didn't have any technical work planned. Since everyone was blaming us for IUSN's outages, I would not turn on the equipment unless someone specifically asked me to do so. Frank's response was clear: We were acting with cabinet authorization, at the request of the acting Premier, to test a mobile phone system. There was no higher authority in the country. I should do whatever I thought was reasonable to advance that testing.

I went to the Telecom office and spoke with the Director. We were turning on the GSM system again, but would not announce it yet. We wanted to determine if we were really the cause of the outages. The process would take about an hour.

Sitting at the guesthouse in North Alofi, Harvind started a ping to gatech.edu, a server in Atlanta. 800 ms, no packet loss. I turned on the NS5 at telecom. 800 ms, no packet loss. Harvind turned on the NS5 at the guesthouse. 800 ms, no packet loss.

I drove up to Sekena and called Harvind from the AMPS phone. Still 800 ms, no packet loss. I turned on the NS5 at the tower site. Harvind could get the web interface on the access point. Our whole backhaul network was running. Altanta was still 800 ms, no packet loss. I booted the the BTS and turned on the power amp. Still 800 ms, no packet loss. We waiting another ten minutes. No change.

I drove down to the internet cafe in Alofi, IUSN's retail outlet. I asked, "Now that the mobile stuff is shut down, is everything working again?"

"Yes, just fine."

I asked to be sure, "Is it working right now?" They looked over at a screen and said it was.

We left the system on all day. The Telecom Director called around to people who had been complaining of service outages. There were no problems. We had discovered how to prevent our equipment from interfering with IUSN: Just don't tell them it's on. By the end of the day, we were sitting on the porch in Alofi, making cheap GSM calls overseas and using the internet at the same time. Later that day, the Director sent out an e-mail explaining that we had determined that the GSM system was not causing internet outages. Thanks to IUSN's misinformation campaign, we probably lost a full day of testing and some die-hards out there are still blaming us for everything bad that happens to their internet service.

(Next door to the IUSN/RockET internet cafe, there is a combination bakery and pool hall run by a Kenyan man who lost his passport (in red). The bread he makes is very good for breakfast toast, but molds over fast in the tropical climate. He was enthusiastic about the GSM project and I hope he eventually gets good use of it. This photo has little to do with the blog post, other than proximity, but I'm tossing it in here anyway just to help give a sense of the place.)

Niue #9: Up and Running, for a Little While

2010-03-24T12:14:00.000-07:00

On our second Tuesday in Niue, we were finally going to fix our antenna, using the parts fabricated the day before at the government's marine repair shop. We met the Telecom techs at the tower, turned off the BTS PA and broadcast equipment for safety, and got to work.

About half an hour later, we got a call on the AMPS phone at the tower site. It was BCN asking if we had turned off their FM transmitter. It turns out that there had been a miscommunication about which morning we would be working, so this was an unscheduled outage. We explained that the techs were already up the tower and everyone agreed that the safest move was just to let them finish their work. A couple of hours later, the BTS antenna was fixed and everyone was back on the air.

(James Mataele (upper) and Kone Magatogia (lower) installing the antenna mount on the "Chinese TV tower" at Sekena, about 53 meters up. Photos courtesy of Toki Talagi.)

We spent the next afternoon and morning doing some coverage tests. We were still loosing range because of interference from IUSN's US-stye 900 MHz network, but we could use downlink RSSI to estimate what the uplink coverage would look like were IUSN to stop jamming us. It looked like we would have good outdoor coverage in Alofi and all along Alofi Bay down to Halagigie, about 6.5 km from the tower site. Indoor coverage would probably be good in North Alfoi and marginal in most of the rest of town. We had marginal coverage in the hospital parking lot, but none at the airport. There was room for improvement and solid coverage of the populated areas of Niue will definitely require additional sites. That was all in line with the Hata suburban propagation model. The rural model did not apply; the bush vegetation was too dense. Still, in a lot of places in and around Alofi, signals were strong enough for the system to work, even with the interference.

(Harvind at Opaahi Reef, 4.5 km from site, "talking to Allison" at a very strong -65 dBm.)

Back at the Telecom building, Tim was trying to connect OpenBTS to the rest of the world, despite the missing Xorcom analog gateway. Using the new Asterisk-Skype interface, he provisioned a few specific handsets to support calls to a few specific international numbers, just to prove it could be done. For about 2 days, a lucky few of us where making international calls from mobile handsets in Niue at about US$0.03 per minute. Ironically, it was easier (and much cheaper) to call the UK and Japan than to call a wired phone in the same room.

During all of this testing, IUSN's WISP was still just as broken as it had been the week before, except now they had someone to blame. On Wednesday morning, IUSN forwarded me a sample complaint:

"Morning all,

To my surprise my internet is working this morning at 5:30am. I've had no internet connection since Thursday last week. Thanks to the GSM mobile people whom are here on the island doing testing and in the process ...... blocking internet to all users north of the NDB bank and Telecom NIUE. Apparently they put up a machine at Telecom NIUE with the signal beamed at the Makapu tower using the same frequency as IUSN is using. No notification whatsoever - how rude!! They even deny that their machine is blocking internet for some people.... and yesterday even turn off the radio to parts of NIUE without letting the general public and BCN know. Anyway, I hope this will be sorted out today!!"

The reported days and times of the service outage did not correlate with our activities, but IUSN didn't let ignorance and bad facts get in the way of good finger-pointing. By Wednesday afternoon, I was literally getting stopped in the street by angry old men shaking their fists at me and yelling, "Mr. St. Clair says you cut the internet!! Internet very important for this island!!" Disgruntled IUSN subscribers were showing up at the guesthouse to harass us in person. I suspected that there was an active campaign of blame and defamation going on somewhere. (If you wonder why I have no kind words for IUSN's operators...) My suspicions were confirmed when saw this:

To: All Users

Re: Wifi Interference

IUS-N has learned only recently that technical consultants have been on the island for the past two weeks and have been testing a wireless GSM phone system which may have been interrupting your ability to connect with IUS-N's WiFi services over the past few days, in particular, in the Alofi North area. We have learned they will continue to do those tests, sporadically, with no warning, today and possibly in the future.

IUS-N is not able to control the timing of the consultants' tests, nor are the consultants informing IUS-N of the dates or times of these tests.

This email is our warning to users that you likely can expect more unannounced WiFi interference in the Alofi North area today, and possibly in the future, without warning.

This interference may cause connection problems from your location.

If you do experience any problems connecting, or you have in the past few days, please email support@niue.nu with detailed information, to help us keep track of these events.

Regards,

Richard StClair

Technical Manager, IUSN

Around 14:00 Wednesday, to satisfy public complaints, the Director of Telecommunications asked that we shut down all of our equipment. By 14:20 everything was powered off. Around 18:00, internet service was restored in Alofi. So what happened in those three and a half hours? We didn't know. I was still confident that we were not the cause of this week-long internet outage, but open minded enough to want a serious investigation. The problem is that the afternoon's sequence of events didn't provide any solid information about anything.

(While all of this was going on, a cruise ship anchored in Alofi Bay and tended 100 or so German tourists into town. In my best broken German, I greeted them "Guten Tag! Willkommen bei schönes Niue. And you should really be wearing a hat in this sun." Surreal for all involved.)

Niue #8: A Kick in the Pants

2010-03-22T19:11:00.001-07:00

Monday started with a meeting with the Minister (acting head of state while the Premier is out of the country) and the principals in the project. What's the status? I told him the system was installed and running, but there were problems:

The antenna was at an odd angle because of hardware problems.
IUSN's 900 MHz network was interfering with us and limiting our range.
Something (probably IUSN) was interfering with our 5 GHz link to the tower.
Tim's analog gateway box was still in New Zealand.

We agreed there wasn't much we could do about IUSN or New Zealand's customs office on short notice, but we might fix the antenna problem and still make some calls to the outside world via a VoIP carrier and Telecom's satellite link.

The Minister called the Director of Agriculture and Fisheries. The Director called their marine repair shop and told them to expect us later in the morning. The marine repair shop had welding equipment and good stocks of stainless steel plate and threaded rod. Surely, they could build us an antenna mount. Next stop, Telecom. We wanted to talk to the technicians about the antenna mount to be sure we were building the right thing. This time, Toki and Kone drew on the whiteboard the picture I wish I had seen back in January: a detailed, dimensioned drawing of the TV tower hand rail and their preferred antenna mounting technique. It was a revelation. We went to the fisheries shop and showed the mechanics what we needed. They got to work. A little later, the Minister stopped by to check the progress.

(Dept. of Fisheries marine repair shop. On a remote island, you learn to work with what you have.)

While the shop worked, we played "telephone". Tim provisioned a few handsets with 2xxx numbers, including "2009", a woman with a Fijian SIM who got accidentally included in the test group. (She was amazed when one of us dialed her on a wrong number.) Speech calls were spotty in Alofi, but SMS was working reasonably well and we were texting just because we could.

That evening, we went to the BCN studios for a radio interview and call-in program. One of the signs on the wall said "Less English, More Niuean", so Frank did most of the talking.

The interviewer was kind enough to make notes for me in English summarizing the calls to the program. One stuck out as particularly important for OpenBTS. It was something like, "We have the old AMPS system and we can't fix it when it breaks. We have the Chinese TV system and we can't fix it when it breaks. How will the new mobile system be any different?" That's a darn good question, and the answer, I hope, is a good argument for open designs: Telecom Niue has a full bill of materials for their BTS unit, a complete description of the electronics and all of the source code to the software that is running in it. Telecom Niue has all of the information they need to build another BTS just like the one we left behind, even the names of the vendors who supplied the components to us. And I would hope that if they post to openbts-disucss, people there will help them even if we are not around.

Niue #7: Day of Rest

2010-03-22T16:28:00.000-07:00

Sunday started with an IUSN technician coming to the guest house to return the key to the TV tower site and complain that we "blew out" their Trango power supplies by cycling them the day before. That was surprising and I apologized, although I still doubt that's what actually happened. He then went on to complain that he had to work on a weekend and even on his birthday. That was annoying, so I assured him that however inconvenienced he was by us, we where much more inconvenienced by IUSN.

After that little spat, Frank and Taiichi took us on a driving tour of the coastline. Niue is a big chunk of limestone surrounded by a narrow shelf. There are no real beaches, but there are amazing caves and chasms and tide pools 100 yards long and 20-30 feet deep.

In the middle of all that, we stopped at Taiichi's for lunch. We had green coconuts straight from the tree, Cookie grilled some pork and lamb and Frank brought a collection of local foods cooked in an "umu", an underground oven.

Niue #6: Installation

2010-03-18T18:49:00.000-07:00

On our second Saturday morning, we met the Telecom technicians at Sekena. We were finally going to install the GSM equipment.

The first immediate glitch was that the u-bolts and pipe we had scavenged would not work. The pipe was too short and the u-bolts were too small. (It would really have been nice to have a mechanical drawing of that safety rail back in January...) The techs said they could probably bolt the antenna directly to the railing, with no pipe or u-bolts at all. At the very least, they could install the antenna cable and reposition the NS5 to shorten its CAT5 run.

(Toki spooling out the cable.)

(Sam hoisting the antenna.)

(Looking up the tower.)

While the Telecom guys worked the tower, we installed the BTS in the rack we had stripped a week earlier. By dumb luck, the 200' LMR-600 cable we brought was exactly the right length.

(Harvind and the newly-installed BTS unit.)

Once the techs cleared the tower, we engaged the power amp. A few handsets started beeping and buzzing as OpenBTS pushed the welcome message into them, but something was wrong. Even right under the tower, speech calls barely worked. Harvind started poking around in the radio layer and announced that we were getting hit with serious interference in the high-end of the downlink spectrum, probably from IUSN's 900 MHz gear. He made some gain and IF adjustments to get the best performance we could manage under the circumstances, but we were still loosing 6 dB of our noise floor. That's a factor of 2 in range and a factor of 4 in subscriber battery life. And the tower techs told us the antenna is at a funky angle because it didn't fit the hand rail very well without the pipe, so we might not even had coverage in Alofi. Arg!!

At this point we wondered: was the interference from IUSN equipment on the tower, or from remote sites beaming at the tower. There was a cabinet on the wall with two POE injectors in it. We had four good work days left on the island and not a lot of time for dodgy e-mails with IUSN's remote managers. Their service was completely unusable that morning, so it seemed unlikely that anyone would even notice a 5-minute link outage. I unplugged the IUSN equipment. The interference want away. I plugged it back in. The interference returned. It wasn't the most prudent thing I've ever done, but now we knew exactly where our problems were coming from.

Despite the interference, on the waterfront in Alofi, about 5 km away, we could get cellular coverage and LOS wifi to the tower site at the same time, so we set up an evening work session in a seaside park to play around with the first real timing advance we'd seen since Burning Man. It's not the worst place I've had to work.

Niue #5: The Gear Arrives!

2010-03-18T16:37:00.000-07:00

Friday is airplane day in Niue. The one weekly flight from Auckland arrives a little after noon and about 1/4 of the country turns out to meet it. This is the off-season for tourists, so the flights are running about 1/2 full, mostly shuttling the 30,000-strong Niuean diaspora to and from their ancestral home. My wife, Jessica, was arriving on this flight. So was Taiichi's girlfriend, Cookie. So was our equipment. It was a big day and we were at the airport waiting with everyone else.

After meeting the ladies, we had lunch at Mr. Lee's house, near the airport. Mr. Lee was a Chinese chef who had been stranded in Niue in part of a work-permit scam (long story). He had been fishing the night before and presented us with a wonderful meal assembled from local ingredients. We couldn't stay as long as we liked, though, because our cargo was ready for pick-up and the customs office closed at 16:00, just like everything else.

That evening, we set up the BTS at the guest house to be sure nothing was damaged in shipping. After a visual inspection, we connected the unit to a battery, booted the CPU and then... not much. Asterisk was hanging on DNS because there was no real network to connect it to, so anything that relied on a SIP transaction was timing out. But unlike at Burning Man, we had Tim there and he knew how to fix it. A few minutes later, we placed the first-ever GSM call in Niue. Later that night, Tim made his own blog post, with some photos, too.

Niue #4: License?! We Don't Need No Stinkin' License!

2010-03-16T11:09:00.000-07:00

By Wednesday afternoon, pings to California through IUSN showed 95% packet loss. They also showed 10-20 second latency, a sign of some serious network management problems. But by getting up really early, Tim managed to make the project's first blog post from Niue. A few hours later, via Telecom's private network, we got our first e-mail from IUSN telling us that they run a lot of US-style 900 MHz broadband equipment all over Niue and it was too bad that we didn't coordinate with IUSN and New Zealand's infrastructure consultant before we arrived. Their attitude seemed to be that Niue's spectrum is unregulated and they grabbed it first. They claimed that they needed no license, but still took issue with us referring to their system as "unlicensed". At that point, I gave up trying to make sense of their communications.

Silly us. We had been told our client had a license for GSM900. We had coordinated with the government regulator and with the state-owned telco. How foolish of us, not getting permission from some US-based non-profit we had never heard of, not consulting with some advisor from some other government and not expecting to see a ton of US-market ISM-900 gear in ITU region 3, stomping all over the GSM uplink band. We told IUSN that we were surprised to see ISM-900 equipment in Niue, since it is generally illegal outside of North America, and that we were surprised, license or not, that the official regulator had no specific technical information about what they were doing.

So here's the 900 MHz spectrum plan for Niue, as we now understand it:

880-890: Telecom Niue's AMPS WLL downlink, 7 kW EIRP, 2 sites

890-915: GSM900 uplink , up to 1 W EIRP, potentially hundreds of sites

903-928: IUSN's 900 MHz network, 6 W EIRP, ~20 sites

935-960: GSM900 downlink, up to 50 W EIRP, 1 site

IUSN was worried that GSM900 downlink would interfere with their ISM-900 stuff. That was unlikely from the start. ISM-900 radios are designed for unlicensed operation, so they need to tolerate high-power near-band interferers, like public service radios and cell towers. The fact that IUSN's network could coexist with Telecom's scorching hot AMPS sites meant that we were unlike to cause any new problems. But we also knew that ISM-900 gear jams the GSM900 uplink, since they overlap in the 902-915 MHz range ... which is why that stuff is illegal in most of the world. And we could see that cell phones themselves might disrupt IUSN's network, if there were ever enough of them out there. We recommended, as an immediate measure, that IUSN retune their links to avoid operating below 915 MHz within 3 km of the GSM site, at least to the greatest degree possible, and that GSM avoid the 902-915 MHz range. To our knowledge the Niue GSM system does avoid 902-915 MHz, but we never got a response from IUSN. And we don't need one now, since we turned over operation of the GSM site to Telecom Niue when we left.

On Thursday evening, IUSN was already reporting 900 MHz link failures. Our GSM gear was still in Auckland. Powerful stuff, huh?

Niue Episode 3: Linking the Site

2010-03-14T10:04:00.000-07:00

Monday morning started with a meeting at Telecom Niue's switching room. All of the administrators had stepped out of the way at this point, so it was us, senior engineer Carlos Tukutama and a handful of technicians and junior engineers. Among them was Toki, who we had met in the airport in Auckland a few days earlier. The switch manager was out sick and couldn't make the meeting, but for the most part, this was the Telecom technical staff. It was time to actually start doing something.

The plan for the week was

Establish an IP link between Sekena and the Telecom switch room.
Set up an Asterisk box in the switch room.
Strip the equipment rack and set up our power supply.
Put the antenna mounting hardware in place on the TV tower.

Now, here are two important parts of the story that become the foundation for a lot of what happened over the next week and a half:

First, there is a group called the Internet Users' Society Niue (IUSN) that runs a public WISP on the island, the "free" one that costs NZ$25 to join and blocks outgoing mail and UDP applications. The history of the relationship between IUSN and the Niue gov't is messy, but that's tangential to our story here. We knew they were running a lot of 2.4 GHz gear in Alofi because we could see it on our laptops, but we didn't know much beyond that. As it turns out, neither did anyone else on the island.

Second, we were told that our client had a license for the GSM-900 band. We were also told several times that we were acting with "cabinet authority". According to IUSN's reading of Niue law, "cabinet authority" means that we don't actually need radio licenses. IUSN also claims that since Niue is not a member of the ITU that there is no spectrum regulation, but we were told that Telecom Niue's senior engineer, Carlos, was the official spectrum regulator for the country. I know that Telecom Niue does issue amateur radio licenses; their telephone book says so. It seemed prudent and respectful to coordinate our radio activities through Carlos, whether we had a legal obligation to do so or not. We presumed that IUSN were doing the same.

Back to the narrative...

I showed Carlos our Nanastations and told him that we would need spectrum in the 5.2, 5.3, 5.7 or 5.8 GHz band and that we would prefer the 5.2 GHz band. He said he was not aware of anyone else in the country using that band and we were free to do so. He asked what our fading margin would be. We said we were expecting 20 dB. He said that sounded OK. Easy enough, right? So the first order of business Monday morning was to put up the Nanostations, one on the TV tower and one on the utility mast at the Telecom office.

(James Mataele and Kone Magatogia on Telecom's utility tower.)

Word of the project was spreading. When we got to Sekena Monday afternoon to install the other Nanostation, the BCN TV crew was not far behind.

The Telecom guys put the NS5 on the tower and routed us a cable. We used an unshielded cable because our shielded cable was in a box in Auckland and would have been too short anyway. Tim started some connectivity tests.

(Tim, Frank and Taiichi Fox, the private investor in the project.)

The link was flakey as hell. The first problem was cable length. We fixed that later in the day by cutting out a lot of excess line. The second problem appeared to be interference. One minute we'd have our expected 20 dB margin, the next minute the link would disappear completely. A band scan didn't show any 5 GHz 802.11a systems, but there are plenty of possibilities beyond 802.11a. We tried lots of different frequencies in the 5 GHz band, but there were drop-outs on every one of them. We turned off the NS5s for a while.

We figured that if IUSN were running 5 GHz, surely they would have told Carlos. So maybe the interferer was a non-comms system, something outside of Niue's control, like a mobile radar. We went to the Dept. of Fisheries and asked about marine radars, but they said there were no ships in the area that day. What the heck? Were our radios just broken? Was there a configuration problem? Some resonance from the broadcasting equipment in our unshielded ethernet cable?

We were also on the hunt for u-bolts. We could not get mechanical data on the TV tower before we got here and from what we now understood, we would need some large galvanized u-bolts and a 1.5 meter section of 5 cm pipe. After a day of driving all over the island and collecting several plumbing samples, Taiichi and I found a spare fence post at the airport that looked perfect for a pipe, but the u-bolts were a problem. And everywhere we went, we got the same two questions, "When will my phone work?" and "How much will it cost to call New Zealand?"

Meanwhile, the IUSN WISP was failing badly in Alofi. We considered the possibility that IUSN had 5 GHz gear after all and just never bothered to tell anyone, but the WISP failures did not correlate with the state of our NS5s. By Tuesday, the IUSN service was just unusable, regardless of what we were doing. But to be safe, we stopped by the IUSN ground station and asked the technician there about 5 GHz equipment. He said he had no idea what kind of equipment was out there. Everyone who knew was out of the country. He also said they were having problems with their satellite equipment.

(IUSN's ground station near Avacele.)

Niue Episode 2: Site Prep

2010-03-13T08:51:00.000-08:00

After meeting with the government, we took a look at the Telecom Niue switching room, home of a big Redcomm analog switch that serves all of the wireline phones in the country. Tim's mission, should the Xorcom box arrive, is to connect OpenBTS/Asterisk to this switch. In the meantime, Tim is thinking of what else he might connect us to. Telecom's attitude about the Redcomm seems to be that it is big and it is old and it works OK, so don't screw with it. We respect that point of view. When we suggested direct VoIP connections for international calls out of the GSM system, Telecom was skeptical. They were decidedly against anything that would connect their private IP network to the island's public ISP. We respect that, too, and respected it more and more as we started to understand the country's connectivity situation.

Our next stop was the installation site, a hilltop called Sekena, about 1 km south of Makefu village. The Chinese had put a TV tower there as part of a reconstruction package following cyclone Heta in 2004 and we were co-siting with the Broadcasting Company of Niue (BCN). Sekena is the second highest point on the island and an excellent site for a radio systems, so 240 meters to the north, there is a 700 Watt AMPS-850 system being used for WLL service to the north part of the island.

(lat -19.018, long -169.918)

About 2/3 of the way up the tower, 53 meters up, there is a platform with a sturdy handrail. That's where our antenna will go, alongside some existing VHF police radios.

At the tower base there is concrete shed with grid power and air conditioning. BCN gave us use of an old rack that housed a defunct TV repeater. Strip the rack and it's ours. We were feeling a little better about making progress without our cargo, but it was late on Friday and Niueans take their weekends seriously, so we told Telecom that we will have work for their technicians on Monday and we headed back to the guesthouse to get some showers and clean clothes. Then we went to Alofi and paid NZ$25 each to get our laptops provisioned in the "free" wifi system. Ping time to California was about 800 ms, packet loss about 5% and most ports were blocked, including outbound SMTP and every UDP-based application we could think of. (And by Monday, we would think that was really good.)

(Project funder Taiichi Fox helps strip the old TV repeater rack.)

On Saturday, we went back out to Sekena to strip the equipment rack. On Sunday, in proper Niuean style, we took a day off. Our government contact, Frank Sioneholo, took us to his village of Mutalau to see a "hair-cutting" ceremony and the sea cave where his ancestors welcomed the first missionaries to the island.

(Two pickup trucks of slaughtered pigs to celebrate a little boy's first haircut.)

(Frank Sioneholo at the sea cave where his ancestors greeted the first missionaries to Niue.)

Niue Episode 1: A Rough Start

2010-03-09T17:26:00.000-08:00

The first hard step of doing anything in Niue is that of actually getting there. It is not near anything. There is one flight per week, from Auckland. Harvind and I flew into Auckland a day early, just to be safe, booked our GSM gear through Air New Zealand's cargo office at the highest priority and met Tim Panton when he arrived a few hours later. Tim had been in transit for over 24 hours already but managed to be in good spirits anyway.

(Auckland)

Our first snag was that some VoIP hardware that was supposed to be waiting for Tim at the hotel wasn't there. Since that gear wasn't absolutely critical to the project and since we had another week to get it through, we didn't worry too much just yet. We left Auckland on a Saturday morning and arrived at Hanan International Airport in Alofi on a Friday afternoon. (Hanan's terminal building is a lot like Black Rock City's terminal building, just with pavement.)

About an hour later we hit our second snag: our GSM cargo, the BTS, antenna and cables, had not been on the airplane. It would be at least a week before we installed a BTS. At least we had a couple of Nanostation-5 radios and plenty of time for site prep, right?

After verifying that our cargo really was stuck in Auckland for another week, we went to a meeting with the acting Premier, the Director of Telecommunications, the project's private funder, the Director of Economic Development and an infrastructure consultant from the New Zealand Commissioner's office. We proposed our newly-improvised schedule for the week, a "prep" schedule intended to allow fast installation of the BTS as soon as it arrives. We talked about risks: the risk of our cargo missing next week's flight, Tim's ideas about how to connect the BTS to anything else if his Xorcom box never shows up, and some concerns we had about the mechanical details the installation site. We had not changed clothes since arriving, so we met the acting head of state in blue jeans and golf shirts. As we walked out someone said we "didn't need to dress up next time".

It was a bad start, but not a disaster. If we could have the installation site, backhaul and PBX ready by Friday and then work through the weekend, we might still have a working GSM system by the next Monday.

FAKALOFA LAHI ATU

2010-03-07T09:47:00.000-08:00

"FAKALOFA LAHI ATU! Please respond with your provisioning code..."

There is now an OpenBTS pilot site in Niue, installed with the cooperation of Telecom Niue under a license from the government. The system is still in a closed evaluation, but when the evaluation phase ends the Niue system will probably be the first OpenBTS installation to provide common-carrier service to the general public. This is a very big step for the project and will bring a much-missed service to the residents, many of whom already own GSM handsets when they travel in New Zealand. It will be a learning process for everyone involved.

Installation took two weeks and is still incomplete, mostly due to customs delays in New Zealand and incomplete documentation on the installation site. We also had serious problems coordinating spectrum with a large public wifi system who's operators seem to think that they can use whatever spectrum they want without consulting the regulators. I would have blogged about all of this on the spot, but the public internet service was unusable most of the time we were there. (Naturally, they blamed us. More on that later. UPDATED BONUS: They are STILL BLAMING US.) If you need a blog fix right away though, Tim Panton managed to squeeze a posting out.

The short status summary is this: Telecom Niue's technicians put a 13 dBi sector antenna about 53 meters up on a platform. From there, we should be able to get reasonably good coverage over Alofi, 3-5 km away, once the wifi people quit jamming our uplink with their unlicensed 900 MHz gear. We managed to make a few international calls from cellphones in Alofi and we sent a lot of text messages among ourselves around the island. We look forward to working with Telecom Niue over the next few weeks to get the system better configured and tied-in to their existing wireline switch. The details will follow over the next few days.

I also want to say that most of the people we encountered in Niue were remarkably nice to us and that the natural beauty of the island's coastline is stunning ... even for someone who lives in California.

(Kone Magatogia setting the antenna, 53 meters AGL. Thanks to Toki Talagi for this photo.)

Still Spinning

2010-01-08T09:03:00.000-08:00

(This post disappeared from the blog a couple of days ago. I don't know why. It's back now.)

A little over a week ago, I posted some comments on Karsten Nohl's public A5/1 attack and the GSMA's response. Reuters & MSNBC think that carrying what is essentially a GSMA press release should somehow pass for journalism. (Thanks, Foneswitch, for bringing this to my attention.) I thought that the links to ads for cellular service were a particularly ironic touch. I hope a real person actually thought to put them there. Irony aside, two things about this press release strike me.

First, the GSMA is admitting that there are simple things that carriers can do to make their networks more secure and that they just haven't bothered doing those things yet. If that's really the case, then Dr. Nohl's public announcements are already having their intended effect of forcing the GSMA to pay attention the security security practices of its members. I'd really love to hear the specifics of these improvements, though, since traces of transactions in most EU and US networks show that they are already just about as secure as their current equipment allows. There's not much left to do.

Second, many GSM networks in other parts of the world, outside the US and EU, are not encrypted at all, not even with A5/2. Many governments simply do not allow their publics to use encrypted cellular networks. This is an issue that is not addressed by Dr. Nohl's project, by the GSMA or by the various media outlets who are providing free publicity for the GSMA's damage control campaign. For nearly half of the world's GSM users, all this bickering and posturing over A5 cracking is irrelevant because they are not even allowed that level of security. Worse yet, most of them probably don't even know that. That's a much bigger story that doesn't get covered.

Given all the recent spin, I think we should probably stop listening to the GSMA and just find out for ourselves how these networks are configured. In that spirit, I would like to propose a new crowd-sourcing project on GSM security: a public catalog of the security and encryption policies of all of the world's public cellular networks. A good place to put this might be somewhere in Wikipedia, where it will be very hard to censor once published. If anyone is interested in organizing something like that, please let me know.

Fun Facts about Copyright Law

2010-01-06T08:00:00.000-08:00

I am not a lawyer and this is not legal advice. The point of this post is to help you understand your rights well enough to know when to call a lawyer and to help avoid ambiguities that will lead to wasteful conflict later. The topic here is US copyright law, specifically as it applies to computer software written by independent contractors.

Under US law, copyright originates with the author. If you write something, you hold the copyright until it expires or until you transfer it to another party. This transfer can happen in two ways, either through an explicit transfer agreement or through a "work for hire" agreement.

The explicit transfer is easy to understand, but can suffer complications. Suppose you are an independent contractor who writes software for a living. A client hires you for a project. You have a consulting contract that states that all copyrights for the commissioned work are transferred to the client. Done. Right? Maybe, maybe not.

For example, what if the software you deliver includes a library that you wrote several years ago for which you hold copyrights? You can save yourself a lot of grief if you make it clear to your client, in the agreement and in copyright notices on the code itself, that the client will not receive copyrights to that previous work. Sure, if there's a lawsuit, you can present evidence that the prior work existed before the agreement and therefore was not commissioned by the client. That will take several months and cost several tens of thousands of dollars and even then you risk a bad outcome, even if you are in the right, since the legal process has a lot more to do with how fast you can write checks than where the facts actually fall. It is much better to make the history and status of that prior work clear before you even start. And that's just one example of the kind of thing that can go wrong.

The work-for-hire transfer is a little trickier. It really has no place in the software world. The work-for-hire concept is based on the idea that work performed as part of a master-servant relationship belongs to the master, so copyrights to works produced in such a relationship transfer automatically. In the situation of the independent software contractor there are two problems with the work-for-hire concept.

First, in today's world a master-servant relationship is usually interpreted to mean regular employment, not contracting. If you are an independent contractor or consultant, then you are probably not performing work for hire, regardless of what any contract might say. Just calling something "work for hire" in a contract does not automatically make it so.

Second, the automatic copyright assignment of a work for hire relationship does not apply to software. US copyright law transfers copyrights for certain specific classes of works if they are commission as part of a work for hire agreement. The important point, though, is that computer software is not in that list of specific classes of work. Therefore, even under a work for hire agreement, copyrights to computer software remain with the author in the absence of an explicit copyright transfer. Let's pause for a minute while all the software developers (and their managers) go look at their employment agreements.

The chances are very good that any client who is asking you to sign a consulting agreement will just assume that paying you money automatically creates a work-for-hire relationship and that such a relationship automatically applies to software. Neither of these assumptions is true and the facts of the matter, as seen by a court, may include a number of factors, such as your degree of independence in your work, how you bill the client, etc. You will probably prevail, but it would be far better to avoid the inhumane machinery of the legal process by starting with clear contract language defining the copyright status of your work.

Kestrel's standing rule for work on cellular projects is that we will hold all copyrights to all source code that we produce. That is not negotiable; either the client agrees or we refuse the job. We make this clear in our communications with clients, put it in every statement of work and print it on our quotes and invoices. Much of that code goes into OpenBTS and those copyrights are reassigned to the Free Software Foundation, which offers a whole raft of advantages that will be fodder for a future post.

Spin It All You Want

2009-12-30T21:30:00.000-08:00

I started this blog about a year ago with a posting on my visit to 25C3, the Chaos Computer Club's annual congress in Berlin. This year, I missed 26C3. I spent most of the month of November traveling and I will spend nearly half of January on the road. I'm spending December at home.

One of the many things I missed this year was Karsten Nohl's presentation of a public-participation attack on A5/1, the "better" encryption used in GSM systems. I've seen this reported in dozens of different venues in the last few days, including the AP wire story in my own local print newspaper. Most of them offer a confused, half-assed description of what Dr. Nohl did followed by some bit of unchallenged spin from the GSMA. It was typical mass-media coverage, where every story has exactly two sides and the reporter understands neither. (There were are few exceptions, but not many.)

For those who are confused, here's the basic story. An efficient A5/1 cracking algorithm has been known for some time, based on a large pre-computed table called a "rainbow table". What Dr. Nohl did was organize a public-participation project to generate a rainbow table and make it publicly available. He also described a hypothetical intercept device based on readily available hardware and software components, one that could probably be built for $20k-$30k.

The GSMA's response to this, to first say that Dr. Nohl's work is irrelevant and then to say that it is illegal, shows us is that the marketeers and MBAs who make up the GSMA don't understand much about technology, security or the law. Conveniently, someone else has already blogged most of what I am thinking right here. Furthermore, describing a hypothetical system is not illegal. Lots of governments distribute documents that will tell you how to build all manner of intelligence equipment. They are called "patents". What Dr. Nohl did, in patent terminology, was offer a "preferred embodiment", which is not the same as actually building something, using it, or offering it to the public.

Lots of people have cracked A5/1 in private and most of them, you can be sure, never announced it publicly. What Dr. Nohl did was crack it out in the open, largely to prove just how easy it was. You can be sure that if any large organization, criminal or otherwise, wanted to crack A5/1, it has been within their ability for several years now and you can wager that many already have. The GSMA finds it more convenient to ignore that, since fixing the problem would require the cellular carriers who make up that organization to spend billions of dollars upgrading their equipment. It's a lot cheaper to try to intimidate a young mathematician by calling him a criminal in the newspapers. Well played, GSMA.

Rest Octets are an Abomination

2009-12-01T17:21:00.000-08:00

In Harald Welte's recent blog, I see that he is adding rest octets to the system information messages in OpenBSC. We will eventually do the same for OpenBTS. I have implemented GSM rest octets in other systems in the past, but I am eager to look at Harald's code to see if his approach is cleaner than any of my old schemes.

And rest octets are an abomination. I just have to get that on the record: an abomination.

The Democratization of Communications Warfare

2009-11-21T20:01:00.000-08:00

So I am in Berlin for the second time in two weeks, fresh back from DeepSec in Vienna, lounging on the main deck of c-base and looking back on the week. I'm waiting for a new passport and the consulate isn't open on weekends. Harald Welte has been kind enough again to let me use his living room and I'm looking forward to seeing some Cold War museums with him tomorrow.

For many years, the telecommunications industry has relied on the cost and complexity of network equipment to achieve many of its security goals. Sure, the standards had big security holes, but you needed really expensive equipment and a lot of expertise to exploit those holes. The problem, though, is that cost and complexity were often the only security measures. If you had network equipment, network exploitation was usually just a question of how you configure that equipment, and the attack configurations were usually obvious.

Right now, there are genuinely bad people using the public communications networks to plan genuinely bad things. There are state actors using network exploits to monitor or track these bad people. There are state actors using network exploits to abuse the privacy of their citizens. There are criminals using network exploits to commit fraud. There are targets using knowledge of network exploits to confound the state actors who are targeting them. When we see this cycle of measures and countermeasures in the world of radar systems, we call it "electronic warfare". To describe this cycle of exploits and counter-exploits in telecom networks, I'll introduce a new phrase: "communications warfare". The weapons in this type of warfare are IMSI-catchers, jammers and hacked handsets. Thanks to cost and complexity, communications warfare in the cellular networks has largely been the domain of large, well-funded organizations. Even hackers usually stayed out of this game because the equipment and know-how are at a premium, so much so that some mistake the most basic techniques for trade secrets.

Moore's Law and the open source movement are removing the cost and complexity of network equipment. VoIP projects have been doing that for wireline networks for several years now, but projects like OpenBTS and OpenBSC are starting to do the same for cellular. These projects remove barriers that prevent people from experimenting with cellular technologies in their homes and classrooms. They demystify the systems. They have the potential to democratize cellular communications, but thanks to the inherent failings of cellular security, these projects also have the potential to democratize cellular communications warfare.

I don't think that democratizing communications warfare is a good thing, but I think that democratizing cellular is a very good thing. I have spent some time this week wondering if it is possible to achieve the first without unleashing the second.

Astricon 2009

2009-10-21T19:06:00.000-07:00

We took OpenBTS on the road again last week, to Astricon in Glendale, Az. I brought one of the Burning Man test units for show and tell and a desktop kit in case anyone wanted to see a demo.

My first contact at the conference was with Adrian Georgescu. He has some great tools for scalable, reliable VoIP and media networks. If we ever build a large core network for OpenBTS, his approach would be perfect. We also had a long discussion about OpenBTS and he offered some good comments about how it might fit into commercial markets and the difficulties of approaching some of those markets.

I was scheduled to speak in the last session of the conference. I did not originally plan to give a demo in the talk, but everyone who knew me kept saying they would be there for the "demo". So on the morning before my talk I rearranged the slides to make room for a demo, lest I disappoint. (Here are the slides, all 11.4 MB of glorious PDF.) The talk was well-received. I did pretty much the same demo as I had done at eComm a few months back. (I won't be making to next eComm in Amsterdam, but these conferences are very good and Lee Dryburgh does a great service by organizing them.) Anyway, it's a phone system. I make a call. Go figure. The demo took a fun new turn, though, when audience members whose phones had attached to the cell started calling one of my test the handsets on the podium. Audience participation was a nice touch. It made the system real, not just a demo trick. That evening I was walking down a sidewalk and a bunch of guys sitting outside a sushi bar called me over, "Hey! OpenBTS! We want to get you a beer!" Well, that's OK.

I also ran into people from small rural VoIP carriers. They had never taken the idea of entering the cellular market seriously, first because of really stupid spectrum regulation and second because the minimum cost of rolling out service is something like $200k for core network equipment, even for a really small network. Since OpenBTS doesn't need core network equipment, we can drop that entry cost to something like $50k, maybe a lot less if you are already in the wifi business. Can getting rid of the cost barrier change the politics of spectrum regulation? We will do our best to find out.

But for now we're patching bugs and cleaning up a the next release. Then it's back on the road for Vienna. Later.

Burning Man 2009: Days 10 & 11

2009-10-11T14:56:00.000-07:00

Ok, so I'm a month late with this...

Monday, 7 September, was day 10 on the Playa. Black Rock City was packing up and it was time to "leave no trace". The biggest task was to pull down out 70-foot tower. Pavel had started stripping the tower the afternoon before by removing the WD9XSP lighted sign. He started this new morning by removing the antennas.

(He's about 60 feet up there. We took this through a telephoto lens.)

While Pavel did that, most of the rest of the campers packed up the shelter and I bicycled down to Heavy Equipment Camp, part of the Department of Public Works (DPW), with a half liter of tequila. I was going to see if I could get a crane to help pull down the tower. When I got to Heavy Equipment, I walked in through the gate, set my donation bottle on a table next to the coffee pot and took a seat with a few other would-be customers in the "lobby": a bunch of old sofas drawn around a low table under a shade tarp. This is how you borrow a crane in Black Rock City. We were waiting for Chaos, the dispatcher. I was informed that the weekend burns had been very busy for DPW and that Chaos was sleeping in. No problem. On the coffee table there was an overflowing ashtray and can labeled "Tactical Bacon" with an outline of an assault rifle on the wrapper. After a while, an older gentleman came out from a shipping container with 3 or 4 VHF radios clipped on his vest. It was Chaos, the same guy who had inspected our tower installation a week earlier. He walked into the lobby area, talked to each of us and took a few notes. He told me they would be there when they got there. I went back to camp.

We kept packing. The neighbors loaned us a leaf-blower for clearing the dust out of the back of the truck. John came by and we opened a bottle of peach lambic left by a supporter a few days earlier to toast all the people who had helped us so much over the last month. Then we took down the shelter and started contemplating how we were going to remove the guy-line stake from the rock-hard ground, a task we could not safely start until the tower was down. John got impatient waiting for the crane, climbed up the tower with the gin-pole and started pulling it down himself.

John took down the top segment (#7) of the tower himself and was starting to remove segment #6 when the crane arrived. He connected the crane hook to the tower and climbed down. We detached all of the guy-lines and watched the crane lift the whole tower and lay it on its side. It took about 5 minutes.

We spent the next two hours pounding the guy-line stakes out of the ground with sledgehammers. The trick, we found, was to swing at just the right angle to hit the inside of the eye. It was more important to hit accurately than to hit hard. It just took patience.

By the time we were ready to drive out, the sun was already setting. We said our goodbyes, mounted up and went to sit in the 2-hour traffic jam on the way out the gate. We ended up spending the night in Reno due to car problems, finally unloaded most of the camp into a rented storage locker on Wednesday and spent the rest of the week blowing dust out of our equipment. And we are already scheming for next year.

Burning Man 2009: Service

2009-10-04T21:39:00.000-07:00

So we were out in the Playa running a GSM network, but to what end? What were we doing with it? We had planned to offer two services: SMS and speech. We planned for SMS to be our primary service, with half of the network bandwidth set aside for text messaging. Speech would be a secondary service and we would stop provisioning users for speech once we saw congestion in the speech channel assignments.

A key element of our plan was auto-provisioning via SMS. It works like this:

The handset comes to register with us. We check the SIP registry to see if the subscribed is provisioned.
If the subscriber is already provisioned, we just accept the registration.
If the subscriber is not already provisioned, we accept the registration anyway and push across a text message, something like: "Please respond with your telephone number to subscribe to our experimental network. NO EMERGENCY CALLS." The return address of this message is a short code in our SMS server that is assigned to the provisioning application. (We call this the "invitation" message.)
The user replies to the invitation message with a text to the provisioning short code. The text message and the sender's IMSI get passed into the provisioning application.
The provisioning application creates new entries in Asterisk's sip.conf and extensions.conf and forces Asterisk to reload these configuration files.
The provisioning application sends a status text message back to the user.

This process actually worked in the field. We registered 1,300 users via SMS, most of them on Tuesday and Wednesday. The significant problem was that we kept the record of who had already received the invitation message in RAM, not on disk, so any time we rebooted a BTS every handset was likely to receive (yet) another copy of that message. This was an oversight that we did not catch until late in the week, so if you were at Burning Man and got a lot of copies of the invitation message, we apologize. We know how to fix the problem and it won't happen again. (And that kind of thing is part of what makes us an experimental service.)

But the presence of CommNet Wireless was a problem. Whenever a handset lost the CommNet signal, it would try to camp to us. But people don't pay a lot of attention to what network they are on, so they would attempt to send SMS and make calls to numbers not defined in our PBX. For phone calls, we could still connect them through Link2Voip, but for text messages there was nothing to do but bounce the message back to the sender. We bounced 227 such messages during the test. But we also delivered 271 messages between handsets within Black Rock City. (And we had 658 SMS submission attempts fail because Asterisk was non-responsive.)

For speech calls, we placed about 1000 calls to the outside world and had about 900 call attempts fail because the user didn't dial "1" first. But we only connected 39 calls to on-playa handsets and those were almost all our own test calls. This tells us two things about the Burner population. First, they don't make a lot of calls. Second, nearly all of the calls they do make are to the outside world. We talked to a few users and talked to the users who came to our camp to use the PAP2 phones. The calls were very practical: bring more water, bring the spare radiator hose, send a locksmith, etc. (We also had about 2,800 call attempts fail, some because of bugs in our call control state machine and some due to Asterisk problems. We're still studying those.)

So, in short, we saw enough activity to know how the system works and how it fails, but did not provide service anywhere near the capacity we could have. And we saw enough failures to keep us busy for the next few weeks, forcing a lot of progress to a more stable system.

Burning Man 2009: Coverage

2009-09-27T16:42:00.000-07:00

On Monday (day 3), we had made our first coverage estimates, sending SMS to ourselves or to short codes that returned status information. We did this a few more times over the course of the week. We had hoped to use RRLP to automate the coverage estimation process, but so few phones were returning useful RRPL results that we had to do it by hand, with a bicycle, a test phone and a note pad. This is a rough estimate of our -90 dBm coverage within BRC:

"A", "B" and "C" are our three sectors, arranged to maximize coverage over the city and insure coverage at the front gate, the airport and the greeters' station. The "x" at 4:30 marks our camp and tower. The "0" at 3:00 marks Fusion Valley, the first link in our backhaul chain. These coverage estimates match those predicted by the Hata suburban model, the same propagation characteristic observed in BRC in our 2008 test.

Despite the failure of RRLP to produce data, we did get some automatic coverage information from measurements of the timing errors of RACH bursts arriving at each BTS. In a GSM system, the timing error of an arriving RACH burst relative to the signal framing yields a rough estimate of the round-trip propagation time from the BTS to the handset and back. We logged every one of the 2,018,321 validated RACH bursts we received and finally got around to sorting through some of that data late last week.

Analysis of RACH burst timing indicated that most of the arriving RACH bursts came from a distance of about 1 km (mode of 1 km and mean of 1.3 km) and nearly all of them (>99%) came from a distance of 3 km or less. That makes sense given our coverage, our tower location, the geometry of the city and the fact that BRC accounts for >90% of the area's population during the festival.

Outside BRC, though, we predicted that coverage would follow the Hata open rural model, meaning that once a handset got out of the shadow of all of the RVs, shipping containers and metal-framed domes, our range could be 15 km or more. There is some evidence of that in our logs. A small fraction of RACH (~0.5%) bursts arrived on sector A from clustered distances of 11 km and 15 km. Gerlach, NV was 15 km away in the direction of sector A, has a population of about 500 and could account for the cluster of access attempts at that distance. We still don't have a good explanation for the cluster of access attempts at 11 km.

Burning Man 2009: Days 4-6 (part 2)

2009-09-25T18:40:00.000-07:00

Sorry for the posting delay. Back to the blog. For now.

Apart from excessive registration activity, we had a few other technical challenges, most related to backhaul and power.

Our backhaul was a point-to-point WiFi link, running from a Nanostation 5 on our tower to a 30' tower in a camp called Fusion Valley, over at the 3:00 plaza. Fusion Valley, in turn, linked us to Center Camp where we got onto a 10 MB/s microwave relay back to an ISP in the "real world". The people running this network had been very supportive of our project and their system worked most of the time, but keeping anything working reliably on the Playa is hard and they were not granted any magical protection against that. Our static IP number changed and sometimes just quit working. A few times we lost backhaul completely, like the time someone at Fusion Valley plugged some power tools into the same circuit as the network gear, tripped the breaker and then just walked away for a while. Well, it's Burning Man. Stuff happens.

We were running a mostly-local service. Occasional loss of backhaul should not have been a serious problem, but it was, because Asterisk kept trying to run DNS and reverse-DNS queries. Asterisk would just lock up when it couldn't reach the internet. We tried replacing every hostname we could find with a numeric address, we tried filling out /etc/hosts by hand, we tried setting up a local DNS cache, we really tried our best to understand and trim down our Asterisk configuration, but we just could not stop Asterisk from freezing when the backhaul was down. Since we were using the Asterisk SIP registry has our HLR for location updating and SMS address resolution, a frozen Asterisk server also shut down everything else we were doing.

Another technical problem was power. Here, we did something dumb: we assumed that new batteries would be topped off, and so we didn't bother checking the acid levels. We had enough battery capacity to run the full system for at least 10 hours but the first time we tried to leave the system on overnight we woke up to a low voltage alarm at 4:00. Since we didn't want to run a generator right next to the tent at 4:00, we shut everything down and went back to bed more than a little concerned about the batteries. The next morning we started the generator but the batteries were not responding well, not taking a charge. By the afternoon, though, we thought to check the acid levels and found they were very low. We topped off the batteries with water and had no more problems that week, except for a little boil-over from over-filling. BTW: Playa dust is excellent for neutralizing battery acid spills.

There were also a few loose ends for John to tie up in the SMS server, like saving and reloading the message queue and "bouncing" undeliverable message back to their senders. He also added some test features, like a "411" short code that would return system status information, that were really handy for coverage testing. These weren't really problems, though, just straightforward development.

By day 6, Thursday, we had fixed the registration loads, had decent power and backhaul, and a good feature set on the SMS server. We were (finally) starting to have a stable system.

Burning Man 2009: Days 4-6 (part 1)

2009-09-18T19:04:00.000-07:00

OK, this is where the blog gets technical.

Day 4 (Tuesday) started with fried eggs, cornbread pancakes and Turkish coffee, followed by the departure of the tower crew. We were all disappointed that they could not stay for the full week. Hopefully, they will next year. After we said our goodbyes and Martin's SUV disappeared into the dust, we turned our attention back the the BTS units.

When we first turned on the system at full power, we got flooded with location updating (GSM registration) requests. We had expected this. We had seen it the year before. We had heard about it from network engineers from big carriers. I had seen it in IMSI catchers. We expected it would die down after a couple of hours, too, but there we were wrong. Now what? We had to determine the cause of the excessive load, we had to slow it down, and we had to make whatever changes we could to allow the system to serve provisioned test users even in the face of this load.

Over the next two days of experiments and analysis, we found multiple causes for the excessive registration load.

We were using the "IMSI attach" protocol, meaning that any time a phone slipped out and service and came back it would attempt to register again. That's great for keeping up-to-date presence information in small networks, but it was a disaster in a large network with spotty coverage along the edges.
We were trying to do RRLP queries during each registration. RRLP is the protocol used to communicate with the GPS receivers inside most US handsets. We had hoped to make maps of coverage in real time and Alon had put a lot of work into that. But without GPS assistance data the useful RRLP response rate was less than 1% and a lot of the requests were timing out, causing phones to spend more time on the radio channel and exacerbating the loading problem.
We had the cell reselect hysteresis set too low, causing phones to change cells (and thus request registration) too frequently.
We had the registration timer (T3212) set too low. Again, a small T3212 was good for presence tracking in small networks, but a poor choice in this environment.
We were using a random exponential back-off for access requests but were resetting the back-off timer value (T3122) any time there was a successful channel grant. This was too permissive.
There was another GSM network on the playa, Commnet Wireless, operating from Frog Pond, a hot spring on private property a few miles away. Any time a phone passed into Commnet Wireless' service and back out again, it would request another registration.

Most of these problems could be fixed with configuration changes and software modifications, however:

We could not set T3212 above one hour because asterisk would not allow us to set the SIP registry timeout to more than an hour.
Commnet was just something we would have to live with.

(Aerial photo of the Commnet site at Frog Pond.)

Fixing those problems greatly reduced the registration problem. We made two additional changes, though, to insure smooth operation:

We reserved a few control channels for non-registration activities. This insured that provisioned users would have a chance to get service even during surges of registration activity.
We added a control loop that would automatically adjust downlink power to limit congestion. This was a huge improvement in terms of overall system behavior, since it would automatically ramp up power at just the right rate whenever we had to restart a BTS unit.

Of course, congestion management was not the only problem we had to solve. More on that in part2.

Burning Man 2009: Day 3

2009-09-18T13:24:00.000-07:00

On the morning of day 3 (Monday) the tower was up but not fully wired. Martin, the sponsor who had actually provided the tower, climbed up to have a look around. Our neighborhood was getting built out quickly on this first day of public access.

The first order of the morning was for Arturo to mount the third cell sector antenna, run the RF cabling, and set up our wifi link. Once that was complete, Arturo took a break to use Bill's spiffy field shower and Star, a visitor to the camp and friend of the project, volunteered to install the light-up call sign that Bill had brought.

By the late morning, we had the BTS units laid out on a trailer and wired-up under a shade tarp. We were ready to start testing.

First, we started up the BTS units without the power amplifiers and made a few test calls with handsets in the tent. Things looked normal, so we turned on the power amps. We got an immediate flood of registration attempts, which was expected. The flood continued, though, for well over two hours, which was not expected. By the third hour we turned off the power amplifiers, took a look at the logs, and started to think about why the registration load was so high and what we might to do manage it. (More on that the next post.) Meanwhile, the tower guys (Arturo, Victor and Martin) sat out on the back of the truck playing guitar and signing Mexican tunes, which was a nice touch on a windy afternoon in the high desert.

That evening, we made beef stew with the leftover meat and bones from Sunday's steaks. The tower crew got a much-deserved night out on the town in Black Rock City, starting with a tour of the city from Ranger Velveeta (Annie) and ending with a visit to Vamp Camp in the 7:30 plaza. We met back together at 3am and talked about plans for the next morning.

Burning Man 2009: Days 1 & 2

2009-09-17T14:01:00.000-07:00

So I'm finally getting around to writing up some of our experiences at Burning Man.

Let's start with the first two days. We came in with early arrival passes on Saturday 29 August and most of us arrived in the afternoon. The first order of business was to set up the shelter before dark. John Gilmore dropped by to lend a hand and say hello. Then we set up the kitchen. The tower crew arrived that evening and we made all had a hot dinner together in the cool desert night.

The plan for the next morning was to start early and have the tower erected by noon, but no plan survives contact with the Playa. The first complication was that the ground was much harder than the tower crew had expected. They were expecting sand and loose rock, typical of their other desert installations in northern Mexico and the southwestern US. The Playa was different: 20-30 cm of brittle gypsum laying over dense, damp clay. They had made a set of screw-in anchors, but the screw pitch was too steep and anchors were impossible to drive in this hard ground. They considered borrowing some welding equipment from DPW to re-engineer the screws, but in the end we just stripped off the threads and drove the bare stakes (each about 1.7 m long) directly into the playa with a sledgehammer. Dealing with the anchors blew the whole morning and by noon it was starting to get windy.

We spent the rest of the day erecting the tower, which meant a lot of sitting around waiting for short breaks in the wind, especially as we got closer to the final height of 21 m. There was also a brief distraction when a gust of wind nearly took our shelter, ripping most of the grommets out of our tarps and breaking several of the PVC ribs. Mr. Gilmore literally saved our camp with a box of spare nylon webbing and clips left over from his own shade structure.

Meanwhile, Bill was busy rigging up our shower and greywater evaporation pond so that we could clean up properly when all this backbreaking work was complete.

By late afternoon, our main tower technician, Arturo, was hoisting the final tower section, with two antennas attached. He wrestled them into place in a 40-kt wind and a small crowd on the ground applauded. Somewhere in all that drama Cameragirl and Chaos came by to "inspect our erection" and critique our guy lines. We contemplated putting a wind turbine on the top of the tower, but decided against it. Given the wind conditions, that wind turbine could have supplied most of our power most of the time, but we were concerned that the installation would have been just too dangerous, so we would be relying on the gasoline generator all week.

That evening there were 12 for dinner in the camp. We had steak and mashed potatoes and everyone toasted Arturo for his tower work. He was definitely the man of the evening. Even though the tower itself was erected by sundown, the electronics were still not fully installed, meaning that we were at least half a day behind schedule already. But we had set up a 21 m tower in a windstorm without any injuries, so we saw the day as a success. Such is life in BRC: you frequently adjust your expectations to match reality.

We're Back

2009-09-09T19:32:00.000-07:00

We cleaned out and returned the truck today, marking the end of our 11-day site test in Black Rock City, Nevada, site of the Burning Man festival. Like every other project in BRC, we had our ups and downs, but the trip was a success in the sense that we learned an awful lot about operating a full-range cell site under continuous load from thousands of handsets.

There were a lot of successes. We were happy with the hardware performance and packaging. We rigged a 70' tower in high winds with no injuries. Operating range matched our predictions. We discovered and fixed several bugs under conditions that would have been hard to simulate in a lab. Backhaul integration worked, both for +1 NANP and +883 iNum.

There were also problems. We had problems with Asterisk configuration and performance. We lost IP connectivity a few times every day. The presence of the Commnet Wireless GSM network complicated our operations in unexpected ways. Test users did not follow even the simplest of instructions. We will better understand what really happened as we sort through the hundreds of megabytes of logs in the flash drives of the BTS units and the CDRs and logs of Asterisk and our SMS sever.

There were delights, too. We had visits from a lot of well-wishers and supporters, many of whom volunteered their efforts and brought us gifts. We had good neighbors (the "popcorn guys") and shared dinner with them on a few nights. And we had a lot of fun, both in the hacking sessions and out on the playa. (Alon even chipped a tooth fighting in Thunderdome at the Goth camp, part of the complete Burning Man experience.)

We'll write this all up over the next few days as we sort through the data, but I'll close with some photos of our BRC neighborhood (4:30 & H) under the full moon, me and Alon in our network "ops center" (three laptops and a couple of PAP2 phones in a tent) and us standing around looking concerned while Pavel climbs the tower to strip it:

We Loaded the Truck Today

2009-08-28T21:50:00.000-07:00

We are ready to take open source, open access cellular to Burning Man. Some photos from the last few days...

Kitting out the parts at the beginning of what turned out to be a very long week:

Harvind leaving Sam's with steaks, cheese, wine, canned fruit and ... RV batteries:

(Yes, we plan to eat well.)

Acrylic sheets that cracked overnight:

Laying it out again on aluminum:

Starting from the upper left and going clockwise, that's the power amp, the USRP, the single-board computer, the uplink cavity filter, the duplexer, the pre-amp and the power buses.

Finally, we have the three main-site BTS units and some test equipment in the truck:

(Check out the classic Tektronix 7000 mainframe on the left. Nearly as old as me and tough enough for the Playa. There's a R&S CMD-57 in the box. Not quite tough enough for the Playa, so it will live in the box until needed.)

Time for bed. We have a long drive tomorrow.

We Load the Truck Tomorrow

2009-08-27T08:39:00.000-07:00

We are still building the hardware for the Burning Man GSM test network. We should have finished that by Monday, but:

We used acrylic sheets for mounting radio components the first assembly. Big mistake. The sheets cracked overnight from the strain of the screws. We rebuilt them with aluminum yesterday.
A soldering technician disappeared with our power amps and half of our USRPs. We'll deal with him when we get home. Or maybe have our lawyer deal with him next week. In the meantime, we have a system to build and are expecting replacements today.
Some of our cavity filters got damaged in shipment and had to be returned for service. We got them back yesterday. I want to thank Anatech Microwave for their excellent customer support on that emergency repair.
The USPS appears to have lost our wifi gear. Hopefully, a second shipment arrives tomorrow. Thanks to PCBAY.com for doing their best to clean that up.

The good news is that we've had some volunteers step forward to help, especially Alon Levy, John Gilmore and Donald Kirker, and their help is saving this project. We still plan to be on the Playa by Saturday afternoon and running the main site by Sunday afternoon. It has just meant a few long nights this week.

GSM Security Workshop

2009-08-12T15:41:00.000-07:00

On November 17 & 18, at the DeepSec conference in Vienna, Harald Welte and I will present a workshop on GSM security. Because I was under an injunction at the time the original workshop description was drafted, the material on the official schedule is very limited, which harms me and DeepSec by limiting advertising. Now that my speech rights have been restored, I'd like to use this blog for a shameless plug.

The DeepSec GSM security workshop will begin with an overview of the GSM air interface, Um, sufficient for those not yet familiar with cellular protocols to follow the subsequent material. We will then describe standard Um security mechanisms, their fundamental flaws, common operational mistakes and known techniques for exploiting these flaws and mistakes. We will describe the mechanisms, capabilities and limitations of passive interception, jamming, active attacks on Um and the use of other public networks for higher-layer attacks. More importantly, we will describe best security practices, means of identifying various attacks and the countermeasures available to carriers and to individual subscribers. Going beyond theory, we will demonstrate many of the attacks and countermeasures using a private GSM network built with commercially available components, software from the OpenBTS and OpenBSC projects, and additional software components not found in the public distributions of those projects. We will also take this opportunity to blow away a lot of the trade secret claims that typically surround this field by reviewing publicly available sources, including patents, academic papers and even the court records of intellectual property disputes, that describe these attacks and countermeasures in sufficient detail to allow their recreation by engineers of ordinary skill.

Of course, that's assuming we get at least three people to sign up for the workshop, which is the minimum number to justify the cost to the conference. For more information, see the conference registration page. Early bird registration ends September 7.

The Man Burns in 27 Days

2009-08-09T11:27:00.000-07:00

Plans to run a cellular system at Burning Man are well under way. We have an FCC license and spectrum coordination with Verizon in the GSM850 band. We have most of our equipment in hand and are starting final assembly this week. We have a store-and-forward SIP/SIMPLE server, thanks to John Gilmore. We have official camp placement and early access to the site. We have a block of 10,000 iNum phone numbers in country code +883, thanks to Voxbone. We have a 70' tower and an installation crew, thanks to Martin Pelayo. Things are on schedule and a lot of people have stepped forward to help us and we thank them all.

We heard a story that Larry Harvey was very concerned when he got wind of our plans, thinking we'd turn the Playa into some kind of chatfest. I'd like to assure Mr. Harvey and anyone else that we don't have the bandwidth to provide normal calling service to 50,000 people, nor do we have roaming and settlement agreements with cellular carriers. We couldn't light up BRC with normal speech service if we wanted to, and we don't want to. Want we can do is provide speech service for about 1,000 early subscribers, which we assume will mostly be early arrivals, BM staff, Rangers, DPW, perimeter patrols, etc. After that, we will have enough bandwidth left to run about 1,500 SMS transfers per minute, allowing us to provide text service to pretty much anyone who wants it. We are hoping participants will find this service useful, as a means of locating friends, meeting new people and getting information about Playa events.

We'll post more on the details of using the service soon, after we have made a few final decisions on network configuration and policies.

Three Quotes

2009-07-15T19:08:00.000-07:00

"No state has ever benefited from protracted war."

"Massimiliano Martone and Martone Radio Technology, Inc. and David Burgess, Kestrel Signal Processing, Inc. and Range Networks, Inc. have resolved all disputes between them and all litigation between them has been dismissed. Each of the parties is pursuing their own business interests."

"We have done so much for so long with so little that we are now qualified to do anything with nothing."

Pre-Paid, Revisited

2009-06-24T21:27:00.000-07:00

In a previous post, I talked about a Net10 Nokia 1600 that appeared to be SIM-locked and have some special firmware that made it nearly useless for anything but Net10's prepaid service.

For the latest experiment, I found an AT&T "Go Phone" Nokia 2610. I turned it on right next to a running OpenBTS system. It powered up, registered with OpenBTS and then tried to send an SMS to the private ISDN address 1111340002 via an SMSC at + 14047259800. Here is the raw TPDU of the message.

If anyone has immediate ideas on the meaning of that 69-byte payload or what the handset is expecting to see in response, let me know. The known parameters are:

IMSI 310410250887606
MSISDN +1 707 386 8928
PIN 8928
ICCID 8901 4104 2125 0887 6088

Unfortunately, the phone itself has a power supply problem, so I will need to find another one. And then I can post a second example for comparison.

A Big, Dangerous Assumption

2009-06-21T00:42:00.000-07:00

Lately, I've been exchanging thoughts with people in the OpenBSC project about a specific class of DOS attacks against cellular networks. We discussed GSM vulnerabilities specifically, and tried and failed to think of ways to harden our systems against them.

The DOS attacks we discussed would made from the subscriber side of the cellular air interface. These "rogue handset" attacks have a fundamental commonality with false-basestation attacks: the key to performing either type of attack is having a GSM device that allows you to control layer 3 (L3), the layer where most of the resource management and call signaling actually happen. This observation touches on a huge shortcoming of many ISDN/SS7 systems, that they are built with the assumption that any entity in L3 can be trusted to follow the protcol. (I had a related conversation with Jacob Appelbaum a couple of weeks earlier where he made a broader comment about the error of "trusting the infrastructure".) The ugly truth is that if you can take control of an L3 entity you can make a lot of networks do a lot of strange things.

In a recent appeals case in England, MMI v. CellXion, the UK high court upheld a ruling that the function of an IMSI-catcher was sufficiently non-obvious to justify patent protection. Part of that decision was based on testimony from so-called experts that GSM security was once thought to be "unbreakable". It is unfortunate that the high court was mislead by such testimony. To be blunt, anyone who ever thought that GSM security was unbreakable must not have tried. Heck, you can build an IMSI catcher by accident just by misconfiguring certain cellular equipment. But the important point here is that the representations of these so-called experts reflect the long-standing assumption that rogue parties cannot get their hands on the equipment they need to spoof elements of the system. That assumption may have been reasonable in early days of SS7, when these technologies were new, the equipment was expensive and all of the networks were run by governments and megacorporations. Even then, though, breaking the security was merely expensive, far from impossible. The cost is down now. Today anyone with a few hundred dollars can get their hands on a trace phone, a surplus micro-BTS, a SIM kit, a used cellular network test set or an account with a commerical VoIP-PSTN gateway. All of these products can be used to attack cellular and PSTN networks in various ways, ranging from identity spoofing to shutting down whole cells. Most people are unaware of these risks, continue to trust the network and continue to carry potentially dangerous misconceptions about what is secure and what is not.

GSM Roaming

2009-05-27T09:41:00.000-07:00

I was having an e-mail exchange with John Todd about call routing between cellular and VoIP networks. He asked, "If am roaming with my AT&T phone in Germany and am on the T-Mobile network, and someone in Germany calls my +1-... E.164 number from their Deutsche Telecom land line, the call isn't routed via the US - it gets terminated locally because Deutsche Telecom passes the call to T-mobile directly. But is DT sending the call to my "base" E.164 address, or to the MSRN?"

Good question. I gave my best answer based on my reading of GSM 03.04 and GSM 04.08. John suggested that the answer might also be useful information for other VoIP people trying to get a handle on what goes on inside a cellular network. So here it is:

The originating Deutsche Telekom (DT) local exchange (LE) in Germany, acting on your MSISDN (mobile subscriber ISDN, your normal cellular telephone number), contacts an international switching center (ISC) in Germany, which in turn contacts an ISC in the US.
The US ISC, acting on your MSISDN, contacts AT&T's gateway mobile switching center (GMSC) which in turn contacts AT&T's HLR (home location register) to get your MSRN (mobile subscriber roaming number, the number where your call actually needs to terminate).
The HLR returns an MSRN in Germany that had previously been assigned to you by the German T-Mobile network.
The AT&T GMSC tells the US ISC to forward the call to the MSRN in Germany.
The US ISC tells the German ISC to forward the call to the MSRN in Germany.
The German ISC tells the DT LE to forward the call to the MSRN in Germany.
The DT LE, now using the MSRN, contacts a T-Mobile GMSC in Germany.
The T-Mobile GMSC looks up your MSRN in its visitor location register (VLR), where it finds your IMSI and sees that you are an AT&T subscriber, since that is encoded into the IMSI. The GMSC also gets the identity of the basestation controller (BSC) where you most recently registered.
The T-Mobile VLR contacts the AT&T HLR to verify your account. (Not absolutely sure on this step, but probably. We'll contact AT&T's HLR again in a few seconds, though, so they might defer this step.)
The T-Mobile GMSC contacts your serving BSC to initiate paging on the radio interface. The paging message, sent on the common control channel (CCCH) of every BTS controlled by that BSC, contains your IMSI or TMSI.
Your handset sees the paging message and responds on the random access channel (RACH).
The BTS/BSC sees the RACH message and responds with a channel assignment on your serving BTS through the CCCH.
You pick up the newly assigned dedicated control channel (DCCH) and establish LAPDm async balanced mode. At this point, you have effectively have an ISDN D-channel connection to the BSC.
On the new D-channel, you send a "paging response" message that identifies you, by IMSI or TMSI, to the BSC. (If you send a TMSI, the BSC resolves it to an IMSI at this point.)
The BSC (optionally) authenticates you with AT&T's HLR, (optionally) initiates encryption, and then sends you a message informing you that "connection mode" is established. You may also (optionally) get reassigned to a new radio channel at this point, or simply be told that the mode of your existing radio channel has changed. Either way, you now have an ISDN-like connection to T-Mobile's GMSC, with a D-channel for signaling and B-channel for media.
From this point forward, the signaling part is just like Q.931.

I would encourage any ISDN jockeys out there, especially from OpenBSC or Linux Call Router to correct anything I overlooked or got wrong in that.

Some Comments on IMSI-Catchers

2009-05-06T11:18:00.000-07:00

Update 30 Jan 2012: Since originally writing this post in mid-2009, there have been some noteworthy developments in this area, the biggest of which is that UK patent case MMI & CellXion, referenced below, went to appeal with the result that the IMSI-catcher was deemed too obvious to patent in light of known prior art. For anyone to claim secrecy here is, I think, becoming something of an intelligence test. Enjoy the original post.

I'm going to comment briefly on IMSI-catchers. These are devices that perform false-basestation attacks on cellular networks, including man-in-the-middle call interception. Here's an example of one.

First, I wrote software for IMSI-catchers in the past. That is now a matter of public record.

Second, the GSM protocol operations of an IMSI-catcher are not trade secrets. The IMSI-catcher was patented by Rohde & Schwarz (R&S) in 2003 under the name "virtual basestation" and the implementation of the device is explained in this patent to a degree sufficient to allow a cellular engineer of reasonable skill to construct one, as is the standard for patent applications worldwide. Most of the patent is in German, but this recent ruling from the UK high court summarizes the R&S patent in English for a non-engineering audience. Moreover, that MMI v. CellXion ruling references an earlier published patent application from Nokia, and although I cannot find a copy of that particular document on the web, the high court clearly accepts its existence. Either way, R&S or Nokia, once something is published in a patent application, it is no longer a trade secret. So here's a quick lesson on IP law as relates to 2G-2.5G GSM IMSI-catching:

It's no secret. There are public documents distributed by UK & EU governments that describe how to do it.
Even if it were a secret, that secret would belong to Nokia or R&S, because they appear to have started working on that problem not long after the GSM standard was published.
If you are selling IMSI-catchers in the UK or Europe without the blessing of R&S, you are setting yourself up for a lawsuit, with MMI v. CellXion as a precedent.

Third, I cannot build IMSI-catchers for anyone outside of a verified US government contract. So the next time some unauthorized party contacts me asking for one, I will publish your contact information in this blog.

Fourth, the most common way to build an IMSI-catcher comes directly from the R&S patent itself and is based entirely on off-the-shelf commercial equipment. Nearly any BTS or BTS simulator can be used as the basis of an IMSI-catcher.

Pre-Paid

2009-05-03T21:46:00.000-07:00

Last week I was in a close-out store and found a bunch of Net10 prepaid Nokia 1600s for $20 each. At first I thought I'd found a good source of cheap handsets for testing. I got one home and even though I provisioned it in my OpenBTS system, and even though it registered and showed service, it refused to place a call without any minutes in its "tank".

Here's what I did find, which may be of interest. First, the SIM was generic-looking, no corporate logo, just the letters "SIM" printed on it. Second, when the phone tried to register, the IMSI was from AT&T: 310410226242003. Third, the phone rejected other SIMs, including other AT&T SIMs. The handset appears to be keyed to a specific SIM, so to get this handset to act like a normal phone I'd need to get it rebranded, not just unlocked. Fourth, menus in the phone showed the IMSI, the IMEI, the phone number and a "random number". That was unusual, since a handset normally does not know its own phone number. I am also eager to see if that "random number" is really Ki.

So I won't be buying a big pile of Nokia 1600s at Big Lots, but I'm keeping this one phone because it will be a great opportunity to see how prepaid phones interact with the network. Hopefully, in a couple of weeks I'll have a chance to play with that, unless some other OpenBTS developer out there beats me to that. (Hint, hint...)

The Value of Knowing How Stuff Works

2009-05-02T16:21:00.000-07:00

I was in a thrift store yesterday and came across an old automatic fire alarm. It was a wind-up bell-clapping mechanism triggered by a thermostat. Just by holding it you hand, your could feel how it worked. There was a time when most equipment was like that. You could look at a device and get a pretty good idea of how worked, how to fix it and what its limitations where. You could even do this with electronic equipment once you learned to recognize a few basic component types. I am old enough to have grown up in a world that was mostly like that, but I may well have been in the last generation to do so. For example, I used to repair my cars myself, diagnosing problems by sound and smell. I haven't touched an engine in years though, partly because I can afford more reliable cars now, but partly because when I look under the hood of a modern automobile I can't find the engine. My best friend's dad was a TV repair man, who learned his trade as a radioman in the Marines. He know his craft was in its twilight the first time he saw a "gutless wonder", a unit with hardly anything in it but 2 big ICs and a high-voltage transformer.

Now, I don't mean to sound like some kind of old crabby guy here. I'm getting to a point. Today, most people are surrounded by world of gadgets and appliances of stunning complexity and haven't a clue as to how most of it works. And I say "how it works" instead of "how they work" because these gadgets are all working together, as a system. You punch a text message into your cell phone and hit send and a few minutes later a post appears on Twitter and chances are you literally have no idea what happened in between, or how much information you exposed about yourself in the process. Frankly, I think it's a little dangerous to be so dependent on an interconnected world most people don't understand. (James Burke talked about this kind of danger in his "Connections" program over 30 years ago, a program that made a strong impression on me as a child, but the world of 30 years ago just seems quaint now.) And it's more than a little dangerous when these people are regulating this world they don't understand, lawmakers who have never used e-mail, whose mental model of the internet is "a series of tubes" and who are constantly surrounded by paid lobbyists representing agendas that often run counter to public interest.

What does all of that have to do with OpenBTS? One of the motivations for releasing a GSM stack in open source is to help curious people understand how cellular technologies work, to demystify the GSM network by reducing it to a simple form. This is happening, to some degree, through students and "makers" who have built working OpenBTS nodes as class or club projects. I think there are about a dozen such systems out there now, not counting commercial development kits, and I love to hear from these people. Congratulations to everyone who has even tried to run OpenBTS, but especially to those who succeeded. That first phone call was pretty exciting, wasn't it? And it was very satisfying to know how it happened. Granted, we're not educating lawmakers yet, if that's even a meaningful goal, but it's a start.

The Man Burns in 130 Days

2009-04-28T09:51:00.000-07:00

We have cleared the legal hurdles to run a test network at Burning Man 2009. This network will probably operate in the PCS1900 band, making it compatible with nearly all AT&T and T-Mobile handsets currently used in the US, as well as with any tri-band or quad-band handsets sold anywhere else in the world.

The current plan is to deploy a system largely intended for local (BRC-only) text messaging. We will also support limited speech service, connecting on-playa calls through user-provided numbers and routing inbound calls through the +883 country code. As a practical matter, the Burning Man 2009 experimental network will be important for testing hardware and software designs for use in rural villages, remote facilities and disaster relief applications.

We will release more details as they come together.

Low-Power GSM in the UK

2009-03-21T15:08:00.000-07:00

Back in April 2006, the UK spectrum regulator, Ofcom, did something very rare these days: they auctioned off new spectrum in a standard GSM band. Specifically, Ofcom auctioned off 12 national licenses in the top 6.6 MHz of the DCS1800 band. The lucky winners and winning bids are listed on that link, but I'll repeat them here:

British Telecommunications PLC £275,112
Cable & Wireless UK £51,002
COLT Mobile Telecommunications Ltd £1,513,218
Cyberpress Ltd £151,999
FMS Solutions Ltd £113,000
Mapesbury Communications Ltd £76,660
O2 Ltd £209,888
Opal Telecom Ltd £155,555
PLDT Ltd £88,889
Shyam Telecom UK Ltd £101,011
Spring Mobil AB £50,110
Teleware PLC £1,001,880

A national DCS license for less than US$100k. Damn. (I bet COLT felt like chumps when they saw that they were spending something like 10x the typical winning bid, too.)

The catch is that transmitted power is limited to 200 mW and mast heights are limited to 10 meters AGL for outdoor installations. It seems to me that the 200 mW limitation seems overly conservative, given that a typical GSM handset can put out a full Watt, but Ofcom wrote up a report justifying these limitations on the grounds that they were required for limiting interference with other cells. Clearly, from the assumptions of the report, Ofcom expects these licenses to be used to provide high capacity over small areas, with each licensee having non-exclusive access to the full 6.6 MHz spectrum. Otherwise, it would have made more sense to give each licensee exclusive use of a more limited bandwidth at much higher power levels, even if that meant fewer licenses.

So I'm assuming this is all about fill-in pico-cells, but maybe I'm wrong. I'd love to hear reports of what the license holders are actually doing with this spectrum. I've also heard of similar low power cellular in the Netherlands. I can't find as much information on that, but welcome any reports of similar openings in other countries.

The Problem of Spectrum Granularity

2009-03-03T14:11:00.000-08:00

[BTW, Greetings from eComm 2009.]

One of the most serious challenges to providing low-cost cellular service in rural areas is the lack of available cellular spectrum. Just about everywhere in the world, all of the spectrum is already locked up by incumbent carriers. So, you might ask, if the spectrum is already held, why don't the people living under it have service? The problem is one of granularity.

Rural areas have lower population density and less infrastructure than urban areas. You need taller towers to get greater range. Your cell sites might not have grid power. The best sites may not be near paved roads. These factors make rural areas more expensive to serve. As the same time, perversely, the people who live in these rural areas have less income, and there are a lot less of them. So if you are a cellular carrier with licenses in both rural and urban areas, you have good motives to concentrate on urban service and ignore the rural areas.

Basic physics shows us that urban and rural areas might require different technical approaches. Basic demographics shows us that expectations of profitability are much lower in rural areas than in urban areas. So how do regulators deal with that? They make it nearly impossible to get a cellular license in a rural area without having to get a license in an urban area at the same time. No, that's not supposed to make sense, but it is true nearly everywhere in the world.

Here in the US, the FCC auctioned most cellular licenses by "metropolitan statistical area" (MSA) or "rural statistical area" (RSA). Despite those promising names, more often than not an MSA or RSA is just a county or group of counties. (Here's the map in PDF.) That's why I can't get a license for rural Solano County, California, which is mostly sheep pasture and marshes, without getting licenses for several cities totaling nearly 500,000 people at the same time. That's why I can't get a license for Gerlach, Nevada, an isolated town of about 200 people, without getting a license for Reno, a distant city of more than 200,000, in the bargain. What if you want to serve Gerlach but can't afford a license for Reno? TFB (too ... bad). No license for you!

It's bad enough to do business that way in the US, where even the country folk are affluent by world standards, but in developing countries, where the urban-rural disparity is even greater, most licenses are national. For example, if you want to provide cellular service anywhere in Kenya, you probably need a license for Nairobi. And since median income in Nairobi is around US$160/mo and the median income in the coutryside is less than US$30/mo, you can imagine what that does for the prospects of a small rural carrier ever happening.

If you wanted a licensing system to discourage rural service, it would be hard to design a more effective spectrum allocation policy. Some countries are making noises about changing these policies soon. Let's hope.

NDA and the Path to Servitude

2009-03-02T19:08:00.000-08:00

I have a friend with a consulting client (who will remain unnamed) who is using a digital radio receiver (that will remain unidentified). They are having a hell of a time getting the receiver to work for this client's application, but for a number of reasons that I won't detail here there's a strong motivation to use this particular receiver, regardless of the difficulties.

The problem appears to be in the receiver device driver. So the client runs a test, and the receiver interface fails, and they send the results to the radio vendor. The vendor sends back some questions about the test. They send some answers. The vendor recommends another test. They try it. The vendor asks more questions. Since the client and the vendor are in radically different time zones, every step of this little dance takes at least a day. This has been going on for weeks.

So I ask this friend, "Why wait for the vendor? Why don't you just look at the source code for the device driver and fix this problem yourself?" Damned good idea, but they don't have the source code because the interface to the radio is proprietary. Make a nasally whining noise when you say that: proprietary. No one is suggesting that the vendor should put everything under GPL and give it out to world in a free download. Hell, my friend can even sign an NDA.

Just show them the code. I seriously doubt there's anything there he hasn't seen before. He's worked with several of digital radio systems over the years and all of the interfaces look pretty much the same. You have packets or frames of baseband samples. The packets or frames are timecoded, maybe with sequence numbers, a sample clock, IRIG, SMPTE, whatever. You've seen the G.711 steam in RTP? Most digital radio interfaces look a lot like that. It's the only approach that makes any sense.

On second though, to hell with the NDA. Once anyone sees the code under NDA their careers are in mortal danger. What's the problem? Suppose you sign that NDA and see this proprietary interface and then go off and design another interface for another digital radio. And since there's really only one way to build that interface that makes any sense, it will inevitably have similarities to the design you received under NDA. You may well end up getting sued even though you've done nothing wrong. Legally, those similarities are justified under the "merger" doctrine, but it's not like you just go stand in front of a judge and say "It's just merger, your honor." Instead, there's a process, a process that takes many months and costs a frightful amount of money and has an uncertain outcome. And since your ability to participate in this process is directly related to available funds (and not much else), and since if you fail to participate in the process you lose by default, you can easily get railroaded into signing away your intellectual rights to avoid personal financial ruin. This can happen. I've seen it happen. No thanks. Let them fix their own driver.

Getting back to the original problem, though, what my friend has isn't really a technical problem with the radio interface so much as a psychological problem with the vendor. As an engineering consultant, maybe he should start charging double to deal with psychological problems.

Protocol Bugs in GSM Handsets

2009-02-27T12:00:00.000-08:00

[Mass-produced consumer goods with software faults?! Say it's not so!]

Nearly all GSM handsets have bugs in their protocol stacks. If your GSM network is sufficiently complete and correct, it will not exercise those bugs to any degree that will affect service. But the bugs are there and if you are experimenting with GSM network equipment (like these good people) you will see them. It would be useful, informative and (now) possible for us to start a public discussion of specific bugs in specific handset models. To that end, the old OpenBTS sourceforge wiki is offered for that purpose. It's very much a work in progress, but I would invite anyone with first-hand GSM implementation experience to use and contribute.

The most common handset bugs I have seen are in idle-mode behavior and the handling of the TMSI. These bugs normally do not affect service, but can represent security threats for high-profile individuals who carry the wrong phones. Here's one example it gross detail.

A GSM subscriber is ultimately identified by IMSI. (The IMEI, common in IS-95 and IS-136, is rarely used in GSM.) Subscriber identities are frequently sent across the air interface in unencrypted form. That's because encryption cannot be activated until the subscriber's identity is known. In fact, nearly every transaction in GSM L3 begins with an uplink message that carries the mobile identity, and the LAPDm contention resolution procedure causes the network to echo that first message back verbatim.

If an attacker knows the subscriber identity associated with a person and can intercept GSM control channels, the attacker can use these open identity exchanges to track that person's movement and calling activity from cell to cell through a GSM network. An attacker could also use the IMSI for direct toll fraud in networks that do not perform authentication, which is more common that you might think in some parts of the world. To mitigate these risks, the GSM specification introduces the TMSI, an arbitrary 32-bit tag that can be used in place of the IMSI for anonymizing transactions.

Exposure of the IMSI-TMSI relationship would make the TMSI useless, so in a well managed network these two identifiers never appear in the same unencrypted transaction. Typically, the handset will use the IMSI for an initial access, the network will then perform authentication and engage encryption, and then assign a TMSI through the encrypted channel. All future accesses will use the TMSI. That's what you'll see in most American and European networks, which tend to be well-managed from a security standpoint, A5/1 weaknesses aside.

Now, the TMSI is valid only in the "location area" (LA) in which it is assigned. The LA is normally the area served by a common base station controller (BSC). In American networks, this usually means an area with a population of 100,000 to 250,000 people. When a handset moves to a new location area it is supposed to invalidate its TMSI, forcing the use of the IMSI on the first access in the new LA. Why? Because if the phone initiates access with a TMSI, that TMSI will be useless in the new LA. The network will just have to turn around and request the IMSI in order to authenticate the handset. When that happens, the previous IMSI-TMSI relationship is exposed, unencrypted, on the air interface. An attacker who keeps good records of CCCH transactions over several cells can then go back and reconstruct the user's movement and pattern of calling activity in the previous LA.

So there's an example of a common handset bug with security implications. I welcome reports of others on the new wiki.

GSM WLLs and Carrier Acceptance

2009-02-25T15:00:00.000-08:00

The biggest challenge to the deployment of OpenBTS is that all of the world's cellular spectrum is already licensed, most of it to very big companies. These big companies don't have strong motivation to deploy low-cost services in rural areas. First, their actual cost of operation is fairly high in rural areas. Second, even if that cost of operation could be lowered dramatically, it would create a marketing problem. Solving the first problem will only magnify the second.

Suppose you're "Big Cellular" and you run a GSM network in the developing world. It costs you $4-$8 per subscriber per month to operate, costing less in urban areas and more in rural areas. But the people who actually live in rural areas can only afford about $2/month, so you mostly avoid those areas, unless a major road happens to pass through them, carrying your richer urban customers between cities. Government regulators may pressure you to serve the rural areas, but you can always just show them your balance sheets and argue (honestly, even) that you are already giving the broadest service that can reasonably be expected for a profitable network. Everyone's happy -- expect for the rural poor who, will never get telephone service under this model.

This is all cozy until a disruptive technology makes $2/month rural service a real possibility. If you're Big Cellular, that's not good news. You already have a legacy network that you're still paying for and the new technology is not directly compatible with it. Even if it were compatible, the new technology creates a marketing problem because your urban customers paying $12/month will soon be demanding to know why they can't get $2 service like their country cousins. You can try starting a second brand, but that's very expensive and you fear that your new, cheap brand will simply erode your existing market along the urban-rural edge.

The solution here is to make sure that the new service is not a viable substitute for normal cellular. I'm not saying give the rural poor broken service. I'm saying give them what they really need, which is reliable telephone service at a very low price, which is not the same thing as cellular, even if the "subscriber terminal" was built to be a cellphone.

The purpose of the new network is to provide basic telephone service in rural areas. You don't need full cellular functionality to do that. For example, maybe you don't implement handovers of active calls between cells. Maybe you don't allow your rural subscribers to roam into "real" cellular networks. If you are really cheap, maybe you even bind each SIM to a specific cell site, eliminating all of the mobility management functions. This functionality already has a name: wireless local loop (WLL). You use GSM like you might use DECT or WiFi, but with much larger service areas and much cheaper handsets.

Operating in WLL mode offers several advantages in this scenario. There is the technical advantage of a much simpler core network, although a carrier can still support roaming for conventional cellular subscribers if it chooses. There is the business advantage of no longer being a direct competitor to legacy cellular networks. And depending on what country you are in, there may be regulatory advantages as well.

If you are Big Cellular, this new low-cost WLL is not a particular threat to your existing business. It serves a market you would rather not deal with. Maybe you can open a new subsidiary to operate WLL networks, or, depending on your local regulations, you can lease your fallow rural spectrum to a WLL carrier. The WLL becomes a modest source of profit. Universal service can be someone else's problem while you, Big Cellular, can do what comes naturally: market ever more complex services to the cities and gouge tourists with crazy roaming fees. Everyone is happy again, and maybe this time we can spread it around a bit more.

GPL and Security Applications

2009-02-25T13:56:00.000-08:00

It is no great secret that many intelligence gathering processes rely on the ignorance or carelessness of their targets. That is why parties that engage in intelligence gathering are loathe to reveal the technical details of their tools. If potential intelligence targets know the tools, they can know the limitations of those tools and take appropriate countermeasures. Since law enforce and intelligence are (or at least should be) legitimate activities to preserve public safety, it is (arguably) in the public interest to protect information about "sources and methods."

So given that, is there a problem with using copyleft practices in an intelligence or security application? Not really, at least not if you can trust your own customers to behave responsibly. The key principle of copyleft open source software is that you must make your source code available to the customers who receive your products. That is not at all the same thing as making it available to the general public and even classified software can be copylefted if the license is drafted correctly.

For example, you could, in principle, produce classified software under a copyleft license and still be within the license and the law while delivering that software to a government customer within the same classified program. You could, in principle, produce law enforcement products, not sellable to the general public, do so under a copyleft license and make the source code available only the the law enforcement agencies that actually buy the products. Again, this can be fully legal and within the terms of the license. The key concept here is that even though the end customer is free, under the license, to redistribute the work, they will do not so because of other practical and legal constraints outside of the license. To be blunt, if you are being prosecuted for a national security violation, a lawsuit from a software vendor is the least of your worries. Civil intellectual property law is not an appropriate tool for protecting state secrets anyway.

Surprise Purge of OpenBTS from SourceForge

2009-02-05T15:43:00.000-08:00

[The SF site was restored a week later. I'd delete this, but there's a long comment thread.]

Back in December, when an injunction was issued against OpenBTS, I had put in a request to SourceForge to purge the project. Then I removed the non-compliant material myself. Then I forgot about the purge request. Now SourceForge finally got around to purging the project.

For anyone who was on the OpenBTS mailing list at SourceForge, I apologize. I'm seeing what can be done about restoring the mailing list and web pages.

Until then, I'll try to find another place to host the web pages.

What Stuff Costs, Part 2: CAPEX

2009-01-24T12:00:00.000-08:00

There are no "list prices" in the global telecom industry. Every purchase is a negotiated deal with the details covered by NDAs. Prices are arbitrary. How do the equipment providers get away with that? Let's take a look...

Consider the cost of installing a BTS in rural site. Something like this:

That costs $200k-$250k, depending on what part of the world you're in. Most of that money is for "civil installation": site prep, concrete pads, backup power, the mast, that little shack, etc. There's well over $150k worth of stuff there just to support the BTS. So what should the actual BTS cost? As long as it's a lot less than the infrastructure cost, the buyer doesn't care because it won't be a significant part of the total site cost. The baseband processors, transceivers, power supplies and amplifiers for a 3-sector 3-TRX ("1/1/1") kit typically run $20k-$50k, depending on the vendor, the buyer, the specific product and whatever side deals the vendor can offer. That will give 21 Bm channels at full rate. There's no point in going below $20k because the savings to the carrier are insignificant below that point. And the price can't go much above $50k before the BTS becomes significant in the total. Notice that this price range has nothing to do with the actual cost of producing a BTS, as long as that cost is well below $20k. The total installed cost is around $75k per fielded TRX, or around $11k per Bm channel.

That's the equipment in the field. You also need a core network. The core network gets installed carrier-grade data centers. As long as the equipment costs less than the data centers, prices just don't matter much. Together, the BSCs, MSCs and location registers in the core network can easily cost over $5k per fielded TRX, or about $700 per Bm channel. The civil part probably costs twice that, bringing to total to around $15k/TRX or $2,100 per Bm channel. The core network also creates a floor for a viable network size, since even a "small" MSC is built to support hundreds of cell sites and priced accordingly.

So the rollout cost is around $15k/TRX for electronics and totals around $90k/TRX for a low-density network when you include all of the civil infrastructure. One TRX can serve about 1,000 subscribers in the developing world so your rollout capital is least $90 per subscriber, not counting counting other costs ignored here. Note, though, that the dominant cost is civil infrastructure. Even if the electronics were free, the total capital would not change by more than about 25%.

The only way to dramatically change the cost of a cellular network is to simplify the infrastructure, something that the existing equipment providers have little motivation to do. For example, if the whole BTS package can be mounted directly onto the mast and left out in the weather, you can get rid of that air conditioned shack. If you cut the power requirements, you also cut the cost of the backup power systems. OpenBTS is radical, though, in its approach to the core network: get rid of it and run BTS units as peers. Don't just reduce the cost of equipment. Reduce the amount of equipment.

This is one way that OpenBTS hopes to change the economics of rural cellular service: reducing the capital requirements to build a network. The OpenBTS model can reduce the rollout capital from over $90/sub to around $25/sub, not by offering a "cheap BTS" but by eliminating most of the steel and concrete and generators that a conventional GSM network requires. OpenBTS can also reduce the minimum size of a viable network to something as small as a single cell site, allowing a carrier to start service with an initial capital investment of less than $30k. Will carriers go for it, though? Is there any spectrum available for this new kind of carrier to emerge? We're working on it...

What Stuff Costs, Part 1: OPEX

2009-01-23T20:00:00.000-08:00

Most African cellular carriers are partly owned by corporations like Millicom and Vodaphone that are traded on stock exchanges in Europe and America. They publish regular financial reports. From those reports we can tell that the typical 2007 African cellular subscriber paid $10-$12/month to talk on the phone for just over half an hour. That sounds like a rip-off until you do a little more math and realize that it actually cost the carrier about $6/month to provide the service, not counting the cost of internetworking. What the heck?

Let's say, for simplicity, that all of the traffic is compressed into 6 hours each day, so that you see a load of about 0.003 Erlang per subscriber during this peak traffic time. A minimum 3-sector GSM BTS site provides about 10.5 Erlangs at 2% blocking and thus serves 3,500 subscribers at your typical daily peak load. If your cost of operation is $6/sub/mo, that corresponds to a cost of about $252k/year per BTS site to run your network, with most of that cost in the BTS site itself: about $200k/year. (!) When we first estimated this, we though we'd misplaced a decimal point somewhere. Then we did we read this article in Balancing Act that put the cost of operating an off-grid BTS site in Africa at around $210k/year. Then we talked to some telecom people from Africa who said the cost was well over $150k/yr but they didn't know by how much. So it probably really is around $200k/yr. Why?

It's all about power. Suppose you have a BTS that draws 5 kW. And since it's in the tropics you have to cool it, which brings your power budget up to 7 kW. To supply that, you need a generator. And since a generator is a target for theft, you need security lighting and cameras, which drive up your power budget and add at least 1 Mb/s to your backhaul requirement, which requires yet more power. Before long, the site is drawing over 1o kW continuously and you are burning at least 25 gallons of diesel fuel every day. Now you need a crew with a truck to drive around fixing generators and fences and filling fuel tanks, which is complicated by the fact that most of these sites aren't even near roads. It starts looking like war logistics, where Sun Tzu tells us that every sack of rice at the front cost 10 more just to get there. By the time you have everything in place you're spending nearly $20k/mo to keep this beast running.

This matters a lot to the long term development of these countries, because most of the people who live out in the countryside cannot afford $6/mo for anything, meaning that they will never get telephone service, not even on a non-profit basis. To achieve universal service, someone will need to try something completely different.

So here's the good news: if you can keep site power consumption down to just a few hundred Watts, this all changes dramatically. Instead of a generator, you can run the whole site on solar panels or microturbines in many parts of the world. No more diesel fuel. No more crews in trucks. Every two years, you replace the batteries in the power system. That's all. That's why the design target for OpenBTS is 75 Watts per transceiver, a target that we are very near already just using off the shelf equipment.

Other other cost components in the subscriber rate are internetworking and capital amortization. Most connections between African carriers happen in Europe. That means that if you call from your MTN cell phone to a wired phone down the street that call may well get routed through France at French long distance rates. And the capital cost of rolling out a rural GSM network is at least $100/subscriber. But those are topics for other posts.

A Tale of Two Licenses

2009-01-19T22:01:00.000-08:00

OpenBTS includes a partial implementation of the GSM air interface, Um, the radio link between a GSM handset and its serving basestation. (By "partial", I mean it includes the minimum set of features to support speech telephony and, in release 2.0 and later, text messaging.)

Now, before I go any further, let me say this it not legal advice. This is my best understanding of a situation based on discussions with some strong authorities in the FOSS world.

Since late September 2008, after some convincing from John Gilmore, all distributions of OpenBTS have been under GPLv3. We like GPLv3. We'd do everything under GPLv3 if we could, but that's a story for a different blog. With OpenBTS there's a catch, though. Even though GSM is a publicly available specification, it is not"open". Many essential elements of GSM are covered by patents. These patents are held by companies like Ericsson, AT&T and Alcatel and are registered in the ETSI IPR database. The current GPL distributions of OpenBTS are offered for only private experimental use, which is generally exempt from patent licensing. Furthermore, OpenBTS is presently distributed as software, not an actual, usable end product. Anyone using OpenBTS is expected to comply with all applicable laws, including patent laws.

But let's say you use OpenBTS in a complete product that provides GSM service. You got the source code under GPLv3. And you have licenses for GSM patents. And you sell your GSM product to network operators. Section 6 of GPLv3 requires that you make the source code available to your customers and Section 11 requires that you extend your GSM patent licenses to anyone to whom you distribute the source code. Because these GSM patent licenses cost money and are granted under limited terms, these requirements appear to be in conflict. (This is not a hypothetical situation, BTW.)

Thankfully, there's a loophole of sorts. Look closely at Section 6. It does not say you must distribute the source code. It just says that you must make sure that people who have your product know where to get that source code. So the key to delivering a commercial GSM system under GPLv3 is to make sure that there is at least one party to distribute the source code who

does not hold GSM patent licenses,
has up-to-date copies and
did not receive that code under GPLv3 from anyone who does hold GSM patent licenses.

Note the compound in requirement (3). A party can meet (3) by receiving the code from a party with no GSM patent licenses or by receiving the code outside of GPLv3. There are two parties in the world who have complete releases of OpenBTS who didn't get it under GPLv3: the authors (Kestrel Signal Processing, Inc.) and the Free Software Foundation, who were granted copyrights on 24 October 2008. Either party could fill the role of the distributor as long as that party does not hold GSM patent licenses.

So, here are some scenarios for equipment providers wanting to use OpenBTS:

Kestrel distributes OpenBTS to an equipment provider under GPLv3 and then Kestrel makes the source code available to that equipment provider's customers in compliance with GPLv3. That works as long as Kestrel doesn't hold GSM patent licenses and the equipment provider does not want to add proprietary features.
Kestrel distributes OpenBTS to an equipment provider under GPLv3 and the FSF makes the source code available to that equipment provider's customers in compliance with GPLv3. That works as long as the equipment providers' software is identical to some public release of OpenBTS. It does not work if the equipment provider wants to add proprietary features.
Kestrel provides OpenBTS to an equipment provider under a different license that relieves the equipment provider of all GPL obligations. This would allow the equipment provider to add proprietary features to OpenBTS and operate with no ongoing reliance on Kestrel or the FSF. It is probably the option that most equipment providers would choose. This would cost money, though, since OpenBTS uses other GPL libraries that would need to be sub-licensed.

And here are some scenarios if the project founders want to get into the equipment business:

Kestrel starts producing a complete turn-key GSM box, not just source code. Now Kestrel needs GSM patent licenses, so Kestrel no longer meets requirement (1) and can no longer distribute OpenBTS under GPLv3. But Kestrel could transfer source code to FSF let them distribute it under GPLv3 to preserve the open source project.
Kestrels' owners set up a new, distinct company to produce and sell GSMequipment and hold the GSM patent licenses. Kestrel extends a non-GPL license to that new company. This last option preserves Kestrel's ability to release under GPLv3, but still allows the project founders to pursue the equipment business.

Is that complicated enough for everyone?

Putting SMS in the SIP World

2009-01-14T12:52:00.000-08:00

SIP/SIMPLE defines two models for messaging. In the "pager model" (RFC-3428), the message is simply transferred with the MESSAGE method and the receiver responds with "ok" or "accepted". In the "session model", the two SIP endpoints establish a session with the INVITE procedure, transfer messages within the session, and then close the session with BYE.

SMS naturally follows the paged model. Radio channels are scarce resources and cannot be held idle for long periods in any practical system. A typical SMS transaction occurs outside of an established call and the radio channel is held just long enough to move a single message. Store-and-forward handling is an absolute requirement, too. When a short message is submitted to the network by a handset, the payload part (the TPDU) must be held until a delivery channel can be established to the destination . Even if the destination handset has service, this can take several seconds. And if the handset does not have service this can take hours or days.

When we first contemplated SMS for a SIP-oriented core network, we thought we could use a messaging server like Jabber to replace the SMSC. That doesn't work. Most messaging servers follow the session model. And why not? IM applications are session oriented and maintaining an open session in the IP world is very cheap. These messaging servers are built for chat sessions and presence, not for storing and forwarding one-off messages to intermittently connected handsets.

So the OpenBTS project is off to build its own store-and-forward page-mode messaging server, probably from MySQL and some simple scripts, using Kannel as a gateway to conventional GSM SMSCs. We'll post more as it happens and welcome any suggestions.

Free As In Freedom

2009-01-07T11:02:00.000-08:00

I never used to have strong opinions about open source development. Then I got sued.

I started working as an independent consultant in late 2001. One reason I "went indy" was that I refused to ever again sign a standard engineering employment agreement that would give my someone else ownership of everything I do or think. I would never again submit to servitude.

In my consulting work, I was careful to read NDAs. I knew not to sign "derivative works" agreements. I knew that most non-compete agreements were illegal in California. I was clear with my clients: "Your project is not the only thing I'm working on and you don't own those other things." I thought I was savvy enough to protect my intellectual freedom. I was wrong.

[...]

To avoid this situation in the future, Harvind and I now insist that any software work based on a publicly available specification be delivered under an open source license. And if a potential client refuses to do business that way? We walk away because those people are not worth the trouble they will cause down the road.

A Lie Agreed Upon: Getting Hybrid Cellular into the Field

2009-01-02T09:52:00.001-08:00

I just got back from 25C3 in Berlin. Thanks to Delta, I had a miserable time getting there, but I'm still glad I went. I met a lot of good people. I tried Club-Mate. I tried my best mangled German with the Reisegepäck staff at Flughafen Tegel. On the last night, Harald Welte treated me to a steak dinner, one of the few proper sit-down meals I had on the whole trip.

I met a lot of telecommunications professionals and we discussed the problem of carrier acceptance of the OpenBTS approach: providing a simple set of services at minimal cost and replacing the GSM "core network" with collection of peer-to-peer SIP applications.

There was a general consensus that the OpenBTS approach was technically feasible, even on large scales, and could be integrated into existing GSM core networks if needed. There was also a general consensus that most incumbent carriers would reject the technology, even if integration into core networks is easy, largely on economic grounds. The simple truth is that nobody in the telecommunications industry is really interested in making a modest profit by serving large numbers of very poor people. The typical cellular executive would rather talk about extending 3G or 4G networks into rural Africa and then not do it. Talking about bringing scorching fast networks to the poor is much more exciting than actually building a 2G system that they can afford to use.

This consensus is not new to me. I have had nearly the same conversation several times over the last two years with a handful of ex-employees from African cellular carriers. The big carriers will continue to concentrate on squeezing more revenue from their more affluent customers by offering more complex services. In the meantime, these big carriers will continue to sit on spectrum that is completely unavailable to the people who live under it, or, in the case of South Africa, cover the whole country with services that only a small minority can actually afford to access. To borrow a phrase from Mark Twain, universal service is "a lie agreed upon" for the telecommunications industry. It won't happen, even at modestly profitable levels, unless regulators force it.

This does not mean that OpenBTS will not find a commercial market. It just means that it will not find a market with incumbent commercial carriers until regulators force them to get serious about universal service. That won't happen until early adopters, most likely small rural carriers, use OpenBTS to demonstrate that self-sustaining universal service really is possible. So we're looking for early adopters.