Rest Octets are an Abomination

In Harald Welte's recent blog, I see that he is adding rest octets to the system information messages in OpenBSC. We will eventually do the same for OpenBTS. I have implemented GSM rest octets in other systems in the past, but I am eager to look at Harald's code to see if his approach is cleaner than any of my old schemes.

And rest octets are an abomination. I just have to get that on the record: an abomination.

The Democratization of Communications Warfare

So I am in Berlin for the second time in two weeks, fresh back from DeepSec in Vienna, lounging on the main deck of c-base and looking back on the week. I'm waiting for a new passport and the consulate isn't open on weekends. Harald Welte has been kind enough again to let me use his living room and I'm looking forward to seeing some Cold War museums with him tomorrow.

For many years, the telecommunications industry has relied on the cost and complexity of network equipment to achieve many of its security goals. Sure, the standards had big security holes, but you needed really expensive equipment and a lot of expertise to exploit those holes. The problem, though, is that cost and complexity were often the only security measures. If you had network equipment, network exploitation was usually just a question of how you configure that equipment, and the attack configurations were usually obvious.

Right now, there are genuinely bad people using the public communications networks to plan genuinely bad things. There are state actors using network exploits to monitor or track these bad people. There are state actors using network exploits to abuse the privacy of their citizens. There are criminals using network exploits to commit fraud. There are targets using knowledge of network exploits to confound the state actors who are targeting them. When we see this cycle of measures and countermeasures in the world of radar systems, we call it "electronic warfare". To describe this cycle of exploits and counter-exploits in telecom networks, I'll introduce a new phrase: "communications warfare". The weapons in this type of warfare are IMSI-catchers, jammers and hacked handsets. Thanks to cost and complexity, communications warfare in the cellular networks has largely been the domain of large, well-funded organizations. Even hackers usually stayed out of this game because the equipment and know-how are at a premium, so much so that some mistake the most basic techniques for trade secrets.

Moore's Law and the open source movement are removing the cost and complexity of network equipment. VoIP projects have been doing that for wireline networks for several years now, but projects like OpenBTS and OpenBSC are starting to do the same for cellular. These projects remove barriers that prevent people from experimenting with cellular technologies in their homes and classrooms. They demystify the systems. They have the potential to democratize cellular communications, but thanks to the inherent failings of cellular security, these projects also have the potential to democratize cellular communications warfare.

I don't think that democratizing communications warfare is a good thing, but I think that democratizing cellular is a very good thing. I have spent some time this week wondering if it is possible to achieve the first without unleashing the second.

Astricon 2009

We took OpenBTS on the road again last week, to Astricon in Glendale, Az. I brought one of the Burning Man test units for show and tell and a desktop kit in case anyone wanted to see a demo.

My first contact at the conference was with Adrian Georgescu. He has some great tools for scalable, reliable VoIP and media networks. If we ever build a large core network for OpenBTS, his approach would be perfect. We also had a long discussion about OpenBTS and he offered some good comments about how it might fit into commercial markets and the difficulties of approaching some of those markets.

I was scheduled to speak in the last session of the conference. I did not originally plan to give a demo in the talk, but everyone who knew me kept saying they would be there for the "demo". So on the morning before my talk I rearranged the slides to make room for a demo, lest I disappoint. (Here are the slides, all 11.4 MB of glorious PDF.) The talk was well-received. I did pretty much the same demo as I had done at eComm a few months back. (I won't be making to next eComm in Amsterdam, but these conferences are very good and Lee Dryburgh does a great service by organizing them.) Anyway, it's a phone system. I make a call. Go figure. The demo took a fun new turn, though, when audience members whose phones had attached to the cell started calling one of my test the handsets on the podium. Audience participation was a nice touch. It made the system real, not just a demo trick. That evening I was walking down a sidewalk and a bunch of guys sitting outside a sushi bar called me over, "Hey! OpenBTS! We want to get you a beer!" Well, that's OK.

I also ran into people from small rural VoIP carriers. They had never taken the idea of entering the cellular market seriously, first because of really stupid spectrum regulation and second because the minimum cost of rolling out service is something like $200k for core network equipment, even for a really small network. Since OpenBTS doesn't need core network equipment, we can drop that entry cost to something like $50k, maybe a lot less if you are already in the wifi business. Can getting rid of the cost barrier change the politics of spectrum regulation? We will do our best to find out.

But for now we're patching bugs and cleaning up a the next release. Then it's back on the road for Vienna. Later.

Burning Man 2009: Days 10 & 11

Ok, so I'm a month late with this...

Monday, 7 September, was day 10 on the Playa. Black Rock City was packing up and it was time to "leave no trace". The biggest task was to pull down out 70-foot tower. Pavel had started stripping the tower the afternoon before by removing the WD9XSP lighted sign. He started this new morning by removing the antennas.

(He's about 60 feet up there. We took this through a telephoto lens.)

While Pavel did that, most of the rest of the campers packed up the shelter and I bicycled down to Heavy Equipment Camp, part of the Department of Public Works (DPW), with a half liter of tequila. I was going to see if I could get a crane to help pull down the tower. When I got to Heavy Equipment, I walked in through the gate, set my donation bottle on a table next to the coffee pot and took a seat with a few other would-be customers in the "lobby": a bunch of old sofas drawn around a low table under a shade tarp. This is how you borrow a crane in Black Rock City. We were waiting for Chaos, the dispatcher. I was informed that the weekend burns had been very busy for DPW and that Chaos was sleeping in. No problem. On the coffee table there was an overflowing ashtray and can labeled "Tactical Bacon" with an outline of an assault rifle on the wrapper. After a while, an older gentleman came out from a shipping container with 3 or 4 VHF radios clipped on his vest. It was Chaos, the same guy who had inspected our tower installation a week earlier. He walked into the lobby area, talked to each of us and took a few notes. He told me they would be there when they got there. I went back to camp.

We kept packing. The neighbors loaned us a leaf-blower for clearing the dust out of the back of the truck. John came by and we opened a bottle of peach lambic left by a supporter a few days earlier to toast all the people who had helped us so much over the last month. Then we took down the shelter and started contemplating how we were going to remove the guy-line stake from the rock-hard ground, a task we could not safely start until the tower was down. John got impatient waiting for the crane, climbed up the tower with the gin-pole and started pulling it down himself.

John took down the top segment (#7) of the tower himself and was starting to remove segment #6 when the crane arrived. He connected the crane hook to the tower and climbed down. We detached all of the guy-lines and watched the crane lift the whole tower and lay it on its side. It took about 5 minutes.

We spent the next two hours pounding the guy-line stakes out of the ground with sledgehammers. The trick, we found, was to swing at just the right angle to hit the inside of the eye. It was more important to hit accurately than to hit hard. It just took patience.

By the time we were ready to drive out, the sun was already setting. We said our goodbyes, mounted up and went to sit in the 2-hour traffic jam on the way out the gate. We ended up spending the night in Reno due to car problems, finally unloaded most of the camp into a rented storage locker on Wednesday and spent the rest of the week blowing dust out of our equipment. And we are already scheming for next year.

Burning Man 2009: Service

So we were out in the Playa running a GSM network, but to what end? What were we doing with it? We had planned to offer two services: SMS and speech. We planned for SMS to be our primary service, with half of the network bandwidth set aside for text messaging. Speech would be a secondary service and we would stop provisioning users for speech once we saw congestion in the speech channel assignments.

A key element of our plan was auto-provisioning via SMS. It works like this:

• The handset comes to register with us. We check the SIP registry to see if the subscribed is provisioned.
• If the subscriber is already provisioned, we just accept the registration.
• If the subscriber is not already provisioned, we accept the registration anyway and push across a text message, something like: "Please respond with your telephone number to subscribe to our experimental network. NO EMERGENCY CALLS." The return address of this message is a short code in our SMS server that is assigned to the provisioning application. (We call this the "invitation" message.)
• The user replies to the invitation message with a text to the provisioning short code. The text message and the sender's IMSI get passed into the provisioning application.
• The provisioning application creates new entries in Asterisk's sip.conf and extensions.conf and forces Asterisk to reload these configuration files.
• The provisioning application sends a status text message back to the user.

This process actually worked in the field. We registered 1,300 users via SMS, most of them on Tuesday and Wednesday. The significant problem was that we kept the record of who had already received the invitation message in RAM, not on disk, so any time we rebooted a BTS every handset was likely to receive (yet) another copy of that message. This was an oversight that we did not catch until late in the week, so if you were at Burning Man and got a lot of copies of the invitation message, we apologize. We know how to fix the problem and it won't happen again. (And that kind of thing is part of what makes us an experimental service.)

But the presence of CommNet Wireless was a problem. Whenever a handset lost the CommNet signal, it would try to camp to us. But people don't pay a lot of attention to what network they are on, so they would attempt to send SMS and make calls to numbers not defined in our PBX. For phone calls, we could still connect them through Link2Voip, but for text messages there was nothing to do but bounce the message back to the sender. We bounced 227 such messages during the test. But we also delivered 271 messages between handsets within Black Rock City. (And we had 658 SMS submission attempts fail because Asterisk was non-responsive.)

For speech calls, we placed about 1000 calls to the outside world and had about 900 call attempts fail because the user didn't dial "1" first. But we only connected 39 calls to on-playa handsets and those were almost all our own test calls. This tells us two things about the Burner population. First, they don't make a lot of calls. Second, nearly all of the calls they do make are to the outside world. We talked to a few users and talked to the users who came to our camp to use the PAP2 phones. The calls were very practical: bring more water, bring the spare radiator hose, send a locksmith, etc. (We also had about 2,800 call attempts fail, some because of bugs in our call control state machine and some due to Asterisk problems. We're still studying those.)

So, in short, we saw enough activity to know how the system works and how it fails, but did not provide service anywhere near the capacity we could have. And we saw enough failures to keep us busy for the next few weeks, forcing a lot of progress to a more stable system.

Burning Man 2009: Coverage

On Monday (day 3), we had made our first coverage estimates, sending SMS to ourselves or to short codes that returned status information. We did this a few more times over the course of the week. We had hoped to use RRLP to automate the coverage estimation process, but so few phones were returning useful RRPL results that we had to do it by hand, with a bicycle, a test phone and a note pad. This is a rough estimate of our -90 dBm coverage within BRC:

"A", "B" and "C" are our three sectors, arranged to maximize coverage over the city and insure coverage at the front gate, the airport and the greeters' station. The "x" at 4:30 marks our camp and tower. The "0" at 3:00 marks Fusion Valley, the first link in our backhaul chain. These coverage estimates match those predicted by the Hata suburban model, the same propagation characteristic observed in BRC in our 2008 test.

Despite the failure of RRLP to produce data, we did get some automatic coverage information from measurements of the timing errors of RACH bursts arriving at each BTS. In a GSM system, the timing error of an arriving RACH burst relative to the signal framing yields a rough estimate of the round-trip propagation time from the BTS to the handset and back. We logged every one of the 2,018,321 validated RACH bursts we received and finally got around to sorting through some of that data late last week.

Analysis of RACH burst timing indicated that most of the arriving RACH bursts came from a distance of about 1 km (mode of 1 km and mean of 1.3 km) and nearly all of them (>99%) came from a distance of 3 km or less. That makes sense given our coverage, our tower location, the geometry of the city and the fact that BRC accounts for >90% of the area's population during the festival.

Outside BRC, though, we predicted that coverage would follow the Hata open rural model, meaning that once a handset got out of the shadow of all of the RVs, shipping containers and metal-framed domes, our range could be 15 km or more. There is some evidence of that in our logs. A small fraction of RACH (~0.5%) bursts arrived on sector A from clustered distances of 11 km and 15 km. Gerlach, NV was 15 km away in the direction of sector A, has a population of about 500 and could account for the cluster of access attempts at that distance. We still don't have a good explanation for the cluster of access attempts at 11 km.

Burning Man 2009: Days 4-6 (part 2)

Sorry for the posting delay. Back to the blog. For now.

Apart from excessive registration activity, we had a few other technical challenges, most related to backhaul and power.

Our backhaul was a point-to-point WiFi link, running from a Nanostation 5 on our tower to a 30' tower in a camp called Fusion Valley, over at the 3:00 plaza. Fusion Valley, in turn, linked us to Center Camp where we got onto a 10 MB/s microwave relay back to an ISP in the "real world". The people running this network had been very supportive of our project and their system worked most of the time, but keeping anything working reliably on the Playa is hard and they were not granted any magical protection against that. Our static IP number changed and sometimes just quit working. A few times we lost backhaul completely, like the time someone at Fusion Valley plugged some power tools into the same circuit as the network gear, tripped the breaker and then just walked away for a while. Well, it's Burning Man. Stuff happens.

We were running a mostly-local service. Occasional loss of backhaul should not have been a serious problem, but it was, because Asterisk kept trying to run DNS and reverse-DNS queries. Asterisk would just lock up when it couldn't reach the internet. We tried replacing every hostname we could find with a numeric address, we tried filling out /etc/hosts by hand, we tried setting up a local DNS cache, we really tried our best to understand and trim down our Asterisk configuration, but we just could not stop Asterisk from freezing when the backhaul was down. Since we were using the Asterisk SIP registry has our HLR for location updating and SMS address resolution, a frozen Asterisk server also shut down everything else we were doing.

Another technical problem was power. Here, we did something dumb: we assumed that new batteries would be topped off, and so we didn't bother checking the acid levels. We had enough battery capacity to run the full system for at least 10 hours but the first time we tried to leave the system on overnight we woke up to a low voltage alarm at 4:00. Since we didn't want to run a generator right next to the tent at 4:00, we shut everything down and went back to bed more than a little concerned about the batteries. The next morning we started the generator but the batteries were not responding well, not taking a charge. By the afternoon, though, we thought to check the acid levels and found they were very low. We topped off the batteries with water and had no more problems that week, except for a little boil-over from over-filling. BTW: Playa dust is excellent for neutralizing battery acid spills.

There were also a few loose ends for John to tie up in the SMS server, like saving and reloading the message queue and "bouncing" undeliverable message back to their senders. He also added some test features, like a "411" short code that would return system status information, that were really handy for coverage testing. These weren't really problems, though, just straightforward development.

By day 6, Thursday, we had fixed the registration loads, had decent power and backhaul, and a good feature set on the SMS server. We were (finally) starting to have a stable system.

Burning Man 2009: Days 4-6 (part 1)

OK, this is where the blog gets technical.

Day 4 (Tuesday) started with fried eggs, cornbread pancakes and Turkish coffee, followed by the departure of the tower crew. We were all disappointed that they could not stay for the full week. Hopefully, they will next year. After we said our goodbyes and Martin's SUV disappeared into the dust, we turned our attention back the the BTS units.

When we first turned on the system at full power, we got flooded with location updating (GSM registration) requests. We had expected this. We had seen it the year before. We had heard about it from network engineers from big carriers. I had seen it in IMSI catchers. We expected it would die down after a couple of hours, too, but there we were wrong. Now what? We had to determine the cause of the excessive load, we had to slow it down, and we had to make whatever changes we could to allow the system to serve provisioned test users even in the face of this load.

Over the next two days of experiments and analysis, we found multiple causes for the excessive registration load.

• We were using the "IMSI attach" protocol, meaning that any time a phone slipped out and service and came back it would attempt to register again. That's great for keeping up-to-date presence information in small networks, but it was a disaster in a large network with spotty coverage along the edges.
• We were trying to do RRLP queries during each registration. RRLP is the protocol used to communicate with the GPS receivers inside most US handsets. We had hoped to make maps of coverage in real time and Alon had put a lot of work into that. But without GPS assistance data the useful RRLP response rate was less than 1% and a lot of the requests were timing out, causing phones to spend more time on the radio channel and exacerbating the loading problem.
• We had the cell reselect hysteresis set too low, causing phones to change cells (and thus request registration) too frequently.
• We had the registration timer (T3212) set too low. Again, a small T3212 was good for presence tracking in small networks, but a poor choice in this environment.
• We were using a random exponential back-off for access requests but were resetting the back-off timer value (T3122) any time there was a successful channel grant. This was too permissive.
• There was another GSM network on the playa, Commnet Wireless, operating from Frog Pond, a hot spring on private property a few miles away. Any time a phone passed into Commnet Wireless' service and back out again, it would request another registration.

Most of these problems could be fixed with configuration changes and software modifications, however:

• We could not set T3212 above one hour because asterisk would not allow us to set the SIP registry timeout to more than an hour.
• Commnet was just something we would have to live with.

(Aerial photo of the Commnet site at Frog Pond.)

Fixing those problems greatly reduced the registration problem. We made two additional changes, though, to insure smooth operation:

• We reserved a few control channels for non-registration activities. This insured that provisioned users would have a chance to get service even during surges of registration activity.
• We added a control loop that would automatically adjust downlink power to limit congestion. This was a huge improvement in terms of overall system behavior, since it would automatically ramp up power at just the right rate whenever we had to restart a BTS unit.

Of course, congestion management was not the only problem we had to solve. More on that in part2.

Burning Man 2009: Day 3

On the morning of day 3 (Monday) the tower was up but not fully wired. Martin, the sponsor who had actually provided the tower, climbed up to have a look around. Our neighborhood was getting built out quickly on this first day of public access.

The first order of the morning was for Arturo to mount the third cell sector antenna, run the RF cabling, and set up our wifi link. Once that was complete, Arturo took a break to use Bill's spiffy field shower and Star, a visitor to the camp and friend of the project, volunteered to install the light-up call sign that Bill had brought.

By the late morning, we had the BTS units laid out on a trailer and wired-up under a shade tarp. We were ready to start testing.

First, we started up the BTS units without the power amplifiers and made a few test calls with handsets in the tent. Things looked normal, so we turned on the power amps. We got an immediate flood of registration attempts, which was expected. The flood continued, though, for well over two hours, which was not expected. By the third hour we turned off the power amplifiers, took a look at the logs, and started to think about why the registration load was so high and what we might to do manage it. (More on that the next post.) Meanwhile, the tower guys (Arturo, Victor and Martin) sat out on the back of the truck playing guitar and signing Mexican tunes, which was a nice touch on a windy afternoon in the high desert.

That evening, we made beef stew with the leftover meat and bones from Sunday's steaks. The tower crew got a much-deserved night out on the town in Black Rock City, starting with a tour of the city from Ranger Velveeta (Annie) and ending with a visit to Vamp Camp in the 7:30 plaza. We met back together at 3am and talked about plans for the next morning.

Burning Man 2009: Days 1 & 2

So I'm finally getting around to writing up some of our experiences at Burning Man.

Let's start with the first two days. We came in with early arrival passes on Saturday 29 August and most of us arrived in the afternoon. The first order of business was to set up the shelter before dark. John Gilmore dropped by to lend a hand and say hello. Then we set up the kitchen. The tower crew arrived that evening and we made all had a hot dinner together in the cool desert night.

The plan for the next morning was to start early and have the tower erected by noon, but no plan survives contact with the Playa. The first complication was that the ground was much harder than the tower crew had expected. They were expecting sand and loose rock, typical of their other desert installations in northern Mexico and the southwestern US. The Playa was different: 20-30 cm of brittle gypsum laying over dense, damp clay. They had made a set of screw-in anchors, but the screw pitch was too steep and anchors were impossible to drive in this hard ground. They considered borrowing some welding equipment from DPW to re-engineer the screws, but in the end we just stripped off the threads and drove the bare stakes (each about 1.7 m long) directly into the playa with a sledgehammer. Dealing with the anchors blew the whole morning and by noon it was starting to get windy.

We spent the rest of the day erecting the tower, which meant a lot of sitting around waiting for short breaks in the wind, especially as we got closer to the final height of 21 m. There was also a brief distraction when a gust of wind nearly took our shelter, ripping most of the grommets out of our tarps and breaking several of the PVC ribs. Mr. Gilmore literally saved our camp with a box of spare nylon webbing and clips left over from his own shade structure.

Meanwhile, Bill was busy rigging up our shower and greywater evaporation pond so that we could clean up properly when all this backbreaking work was complete.

By late afternoon, our main tower technician, Arturo, was hoisting the final tower section, with two antennas attached. He wrestled them into place in a 40-kt wind and a small crowd on the ground applauded. Somewhere in all that drama Cameragirl and Chaos came by to "inspect our erection" and critique our guy lines. We contemplated putting a wind turbine on the top of the tower, but decided against it. Given the wind conditions, that wind turbine could have supplied most of our power most of the time, but we were concerned that the installation would have been just too dangerous, so we would be relying on the gasoline generator all week.

That evening there were 12 for dinner in the camp. We had steak and mashed potatoes and everyone toasted Arturo for his tower work. He was definitely the man of the evening. Even though the tower itself was erected by sundown, the electronics were still not fully installed, meaning that we were at least half a day behind schedule already. But we had set up a 21 m tower in a windstorm without any injuries, so we saw the day as a success. Such is life in BRC: you frequently adjust your expectations to match reality.

We're Back

We cleaned out and returned the truck today, marking the end of our 11-day site test in Black Rock City, Nevada, site of the Burning Man festival. Like every other project in BRC, we had our ups and downs, but the trip was a success in the sense that we learned an awful lot about operating a full-range cell site under continuous load from thousands of handsets.

There were a lot of successes. We were happy with the hardware performance and packaging. We rigged a 70' tower in high winds with no injuries. Operating range matched our predictions. We discovered and fixed several bugs under conditions that would have been hard to simulate in a lab. Backhaul integration worked, both for +1 NANP and +883 iNum.

There were also problems. We had problems with Asterisk configuration and performance. We lost IP connectivity a few times every day. The presence of the Commnet Wireless GSM network complicated our operations in unexpected ways. Test users did not follow even the simplest of instructions. We will better understand what really happened as we sort through the hundreds of megabytes of logs in the flash drives of the BTS units and the CDRs and logs of Asterisk and our SMS sever.

There were delights, too. We had visits from a lot of well-wishers and supporters, many of whom volunteered their efforts and brought us gifts. We had good neighbors (the "popcorn guys") and shared dinner with them on a few nights. And we had a lot of fun, both in the hacking sessions and out on the playa. (Alon even chipped a tooth fighting in Thunderdome at the Goth camp, part of the complete Burning Man experience.)

We'll write this all up over the next few days as we sort through the data, but I'll close with some photos of our BRC neighborhood (4:30 & H) under the full moon, me and Alon in our network "ops center" (three laptops and a couple of PAP2 phones in a tent) and us standing around looking concerned while Pavel climbs the tower to strip it:

We Loaded the Truck Today

We are ready to take open source, open access cellular to Burning Man. Some photos from the last few days...

Kitting out the parts at the beginning of what turned out to be a very long week:

Harvind leaving Sam's with steaks, cheese, wine, canned fruit and ... RV batteries:

(Yes, we plan to eat well.)

Acrylic sheets that cracked overnight:

Laying it out again on aluminum:

Starting from the upper left and going clockwise, that's the power amp, the USRP, the single-board computer, the uplink cavity filter, the duplexer, the pre-amp and the power buses.

Finally, we have the three main-site BTS units and some test equipment in the truck:

(Check out the classic Tektronix 7000 mainframe on the left. Nearly as old as me and tough enough for the Playa. There's a R&S CMD-57 in the box. Not quite tough enough for the Playa, so it will live in the box until needed.)

Time for bed. We have a long drive tomorrow.

We Load the Truck Tomorrow

We are still building the hardware for the Burning Man GSM test network. We should have finished that by Monday, but:

• We used acrylic sheets for mounting radio components the first assembly. Big mistake. The sheets cracked overnight from the strain of the screws. We rebuilt them with aluminum yesterday.
• A soldering technician disappeared with our power amps and half of our USRPs. We'll deal with him when we get home. Or maybe have our lawyer deal with him next week. In the meantime, we have a system to build and are expecting replacements today.
• Some of our cavity filters got damaged in shipment and had to be returned for service. We got them back yesterday. I want to thank Anatech Microwave for their excellent customer support on that emergency repair.
• The USPS appears to have lost our wifi gear. Hopefully, a second shipment arrives tomorrow. Thanks to PCBAY.com for doing their best to clean that up.

The good news is that we've had some volunteers step forward to help, especially Alon Levy, John Gilmore and Donald Kirker, and their help is saving this project. We still plan to be on the Playa by Saturday afternoon and running the main site by Sunday afternoon. It has just meant a few long nights this week.

GSM Security Workshop

On November 17 & 18, at the DeepSec conference in Vienna, Harald Welte and I will present a workshop on GSM security. Because I was under an injunction at the time the original workshop description was drafted, the material on the official schedule is very limited, which harms me and DeepSec by limiting advertising. Now that my speech rights have been restored, I'd like to use this blog for a shameless plug.

The DeepSec GSM security workshop will begin with an overview of the GSM air interface, Um, sufficient for those not yet familiar with cellular protocols to follow the subsequent material. We will then describe standard Um security mechanisms, their fundamental flaws, common operational mistakes and known techniques for exploiting these flaws and mistakes. We will describe the mechanisms, capabilities and limitations of passive interception, jamming, active attacks on Um and the use of other public networks for higher-layer attacks. More importantly, we will describe best security practices, means of identifying various attacks and the countermeasures available to carriers and to individual subscribers. Going beyond theory, we will demonstrate many of the attacks and countermeasures using a private GSM network built with commercially available components, software from the OpenBTS and OpenBSC projects, and additional software components not found in the public distributions of those projects. We will also take this opportunity to blow away a lot of the trade secret claims that typically surround this field by reviewing publicly available sources, including patents, academic papers and even the court records of intellectual property disputes, that describe these attacks and countermeasures in sufficient detail to allow their recreation by engineers of ordinary skill.

Of course, that's assuming we get at least three people to sign up for the workshop, which is the minimum number to justify the cost to the conference. For more information, see the conference registration page. Early bird registration ends September 7.

The Man Burns in 27 Days

Plans to run a cellular system at Burning Man are well under way. We have an FCC license and spectrum coordination with Verizon in the GSM850 band. We have most of our equipment in hand and are starting final assembly this week. We have a store-and-forward SIP/SIMPLE server, thanks to John Gilmore. We have official camp placement and early access to the site. We have a block of 10,000 iNum phone numbers in country code +883, thanks to Voxbone. We have a 70' tower and an installation crew, thanks to Martin Pelayo. Things are on schedule and a lot of people have stepped forward to help us and we thank them all.

We heard a story that Larry Harvey was very concerned when he got wind of our plans, thinking we'd turn the Playa into some kind of chatfest. I'd like to assure Mr. Harvey and anyone else that we don't have the bandwidth to provide normal calling service to 50,000 people, nor do we have roaming and settlement agreements with cellular carriers. We couldn't light up BRC with normal speech service if we wanted to, and we don't want to. Want we can do is provide speech service for about 1,000 early subscribers, which we assume will mostly be early arrivals, BM staff, Rangers, DPW, perimeter patrols, etc. After that, we will have enough bandwidth left to run about 1,500 SMS transfers per minute, allowing us to provide text service to pretty much anyone who wants it. We are hoping participants will find this service useful, as a means of locating friends, meeting new people and getting information about Playa events.

We'll post more on the details of using the service soon, after we have made a few final decisions on network configuration and policies.

Three Quotes

"No state has ever benefited from protracted war."

"Massimiliano Martone and Martone Radio Technology, Inc. and David Burgess, Kestrel Signal Processing, Inc. and Range Networks, Inc. have resolved all disputes between them and all litigation between them has been dismissed. Each of the parties is pursuing their own business interests."

"We have done so much for so long with so little that we are now qualified to do anything with nothing."

Pre-Paid, Revisited

In a previous post, I talked about a Net10 Nokia 1600 that appeared to be SIM-locked and have some special firmware that made it nearly useless for anything but Net10's prepaid service.

For the latest experiment, I found an AT&T "Go Phone" Nokia 2610.  I turned it on right next to a running OpenBTS system.  It powered up, registered with OpenBTS and then tried to send an SMS to the private ISDN address 1111340002 via an SMSC at + 14047259800.  Here is the raw TPDU of the message.

If anyone has immediate ideas on the meaning of that 69-byte payload or what the handset is expecting to see in response, let me know.  The known parameters are:

• IMSI 310410250887606
• MSISDN +1 707 386 8928
• PIN 8928
• ICCID 8901 4104 2125 0887 6088

Unfortunately, the phone itself has a power supply problem, so I will need to find another one.  And then I can post a second example for comparison.

A Big, Dangerous Assumption

Lately, I've been exchanging thoughts with people in the OpenBSC project about a specific class of DOS attacks against cellular networks. We discussed GSM vulnerabilities specifically, and tried and failed to think of ways to harden our systems against them.

The DOS attacks we discussed would made from the subscriber side of the cellular air interface. These "rogue handset" attacks have a fundamental commonality with false-basestation attacks: the key to performing either type of attack is having a GSM device that allows you to control layer 3 (L3), the layer where most of the resource management and call signaling actually happen. This observation touches on a huge shortcoming of many ISDN/SS7 systems, that they are built with the assumption that any entity in L3 can be trusted to follow the protcol. (I had a related conversation with Jacob Appelbaum a couple of weeks earlier where he made a broader comment about the error of "trusting the infrastructure".) The ugly truth is that if you can take control of an L3 entity you can make a lot of networks do a lot of strange things.

In a recent appeals case in England, MMI v. CellXion, the UK high court upheld a ruling that the function of an IMSI-catcher was sufficiently non-obvious to justify patent protection. Part of that decision was based on testimony from so-called experts that GSM security was once thought to be "unbreakable". It is unfortunate that the high court was mislead by such testimony. To be blunt, anyone who ever thought that GSM security was unbreakable must not have tried. Heck, you can build an IMSI catcher by accident just by misconfiguring certain cellular equipment. But the important point here is that the representations of these so-called experts reflect the long-standing assumption that rogue parties cannot get their hands on the equipment they need to spoof elements of the system. That assumption may have been reasonable in early days of SS7, when these technologies were new, the equipment was expensive and all of the networks were run by governments and megacorporations. Even then, though, breaking the security was merely expensive, far from impossible. The cost is down now. Today anyone with a few hundred dollars can get their hands on a trace phone, a surplus micro-BTS, a SIM kit, a used cellular network test set or an account with a commerical VoIP-PSTN gateway. All of these products can be used to attack cellular and PSTN networks in various ways, ranging from identity spoofing to shutting down whole cells. Most people are unaware of these risks, continue to trust the network and continue to carry potentially dangerous misconceptions about what is secure and what is not.

GSM Roaming

I was having an e-mail exchange with John Todd about call routing between cellular and VoIP networks.  He asked, "If am roaming with my AT&T phone in Germany and am on the T-Mobile network, and someone in Germany calls my +1-... E.164 number from their Deutsche Telecom land line, the call isn't routed via the US - it gets terminated locally because Deutsche Telecom passes the call to T-mobile directly.  But is DT sending the call to my "base" E.164 address, or to the MSRN?"

Good question.  I gave my best answer based on my reading of GSM 03.04 and GSM 04.08.  John suggested that the answer might also be useful information for other VoIP people trying to get a handle on what goes on inside a cellular network.  So here it is:

• The originating Deutsche Telekom (DT) local exchange (LE) in Germany, acting on your MSISDN (mobile subscriber ISDN, your normal cellular telephone number), contacts an international switching center (ISC) in Germany, which in turn contacts an ISC in the US.
• The US ISC, acting on your MSISDN, contacts AT&T's gateway mobile switching center (GMSC) which in turn contacts AT&T's HLR (home location register) to get your MSRN (mobile subscriber roaming number, the number where your call actually needs to terminate).
• The HLR returns an MSRN in Germany that had previously been assigned to you by the German T-Mobile network.
  
• The AT&T GMSC tells the US ISC to forward the call to the MSRN in Germany.
  
• The US ISC tells the German ISC to forward the call to the MSRN in Germany.
  
• The German ISC tells the DT LE to forward the call to the MSRN in Germany.
  
• The DT LE, now using the MSRN, contacts a T-Mobile GMSC in Germany.
• The T-Mobile GMSC looks up your MSRN in its visitor location register (VLR), where it finds your IMSI and sees that you are an AT&T subscriber, since that is encoded into the IMSI.  The GMSC also gets the identity of the basestation controller (BSC) where you most recently registered.
• The T-Mobile VLR contacts the AT&T HLR to verify your account.  (Not absolutely sure on this step, but probably.  We'll contact AT&T's HLR again in a few seconds, though, so they might defer this step.)
• The T-Mobile GMSC contacts your serving BSC to initiate paging on the radio interface.  The paging message, sent on the common control channel (CCCH) of every BTS controlled by that BSC, contains your IMSI or TMSI.
• Your handset sees the paging message and responds on the random access channel (RACH).
• The BTS/BSC sees the RACH message and responds with a channel assignment on your serving BTS through the CCCH.
  
• You pick up the newly assigned dedicated control channel (DCCH) and establish LAPDm async balanced mode.  At this point, you have effectively have an ISDN D-channel connection to the BSC.
  
• On the new D-channel, you send a "paging response" message that identifies you, by IMSI or TMSI, to the BSC.  (If you send a TMSI, the BSC resolves it to an IMSI at this point.)
• The BSC (optionally) authenticates you with AT&T's HLR, (optionally) initiates encryption, and then sends you a message informing you that "connection mode" is established.  You may also (optionally) get reassigned to a new radio channel at this point, or simply be told that the mode of your existing radio channel has changed.  Either way, you now have an ISDN-like connection to T-Mobile's GMSC, with a D-channel for signaling and B-channel for media.
• From this point forward, the signaling part is just like Q.931.
  

I would encourage any ISDN jockeys out there, especially from OpenBSC or Linux Call Router to correct anything I overlooked or got wrong in that.

Some Comments on IMSI-Catchers

I'm going to comment briefly on IMSI-catchers. These are devices that perform false-basestation attacks on cellular networks, including man-in-the-middle call interception. Here's an example of one.

First, I wrote software for IMSI-catchers in the past. That is now a matter of public record.

Second, the GSM protocol operations of an IMSI-catcher are not trade secrets. The IMSI-catcher was patented by Rohde & Schwarz (R&S) in 2003 under the name "virtual basestation" and the implementation of the device is explained in the text of this patent to a degree sufficient to allow a cellular engineer of reasonable skill to construct one, as is the standard for patent applications worldwide. Most of the patent is in German, but this recent ruling from the UK high court summarizes the R&S patent in English for a non-engineering audience. Moreover, that MMI v. CellXion ruling references an earlier published patent application from Nokia, and although I cannot find a copy of that particular document on the web, the high court clearly accepts its existence. Either way, R&S or Nokia, once something is published in a patent application, it is no longer a trade secret. So here's a quick lesson on IP law as relates to 2G-2.5G GSM IMSI-catching:

• It's no secret. There are public documents distributed by UK & EU governments that describe how to do it.
• Even if it were a secret, that secret would belong to Nokia or R&S, because they appear to have started working on that problem not long after the GSM standard was published.
• If you are selling IMSI-catchers in the UK or Europe without the blessing of R&S, you are setting yourself up for a lawsuit, with MMI v. CellXion as a precedent.

Third, I cannot build IMSI-catchers for anyone outside of a verified US government contract. So the next time some unauthorized party contacts me asking for one, I will publish your contact information in this blog.

Fourth, the most common way to build an IMSI-catcher comes directly from the R&S patent itself and is based entirely on off-the-shelf commercial equipment. Nearly any BTS or BTS simulator can be used as the basis of an IMSI-catcher.

Pre-Paid

Last week I was in a close-out store and found a bunch of Net10 prepaid Nokia 1600s for $20 each.  At first I thought I'd found a good source of cheap handsets for testing.  I got one home and even though I provisioned it in my OpenBTS system, and even though it registered and showed service, it refused to place a call without any minutes in its "tank".

Here's what I did find, which may be of interest.  First, the SIM was generic-looking, no corporate logo, just the letters "SIM" printed on it.  Second, when the phone tried to register, the IMSI was from AT&T: 310410226242003.  Third, the phone rejected other SIMs, including other AT&T SIMs.  The handset appears to be keyed to a specific SIM, so to get this handset to act like a normal phone I'd need to get it rebranded, not just unlocked.  Fourth, menus in the phone showed the IMSI, the IMEI, the phone number and a "random number".  That was unusual, since a handset normally does not know its own phone number.  I am also eager to see if that "random number" is really Ki.

So I won't be buying a big pile of Nokia 1600s at Big Lots, but I'm keeping this one phone because it will be a great opportunity to see how prepaid phones interact with the network.  Hopefully, in a couple of weeks I'll have a chance to play with that, unless some other OpenBTS developer out there beats me to that.  (Hint, hint...)

The Value of Knowing How Stuff Works

I was in a thrift store yesterday and came across an old automatic fire alarm.  It was a wind-up bell-clapping mechanism triggered by a thermostat.  Just by holding it you hand, your could feel how it worked.  There was a time when most equipment was like that.  You could look at a device and get a pretty good idea of how worked, how to fix it and what its limitations where.  You could even do this with electronic equipment once you learned to recognize a few basic component types.  I am old enough to have grown up in a world that was mostly like that, but I may well have been in the last generation to do so.  For example, I used to repair my cars myself, diagnosing problems by sound and smell.  I haven't touched an engine in years though, partly because I can afford more reliable cars now, but partly because when I look under the hood of a modern automobile I can't find the engine.  My best friend's dad was a TV repair man, who learned his trade as a radioman in the Marines.  He know his craft was in its twilight the first time he saw a "gutless wonder", a unit with hardly anything in it but 2  big ICs and a high-voltage transformer.

Now, I don't mean to sound like some kind of old crabby guy here.  I'm getting to a point.  Today, most people are surrounded by world of gadgets and appliances of stunning complexity and haven't a clue as to how most of it works.  And I say "how it works" instead of "how they work" because these gadgets are all working together, as a system.  You punch a text message into your cell phone and hit send and a few minutes later a post appears on Twitter and chances are you literally have no idea what happened in between, or how much information you exposed about yourself in the process.  Frankly, I think it's a little dangerous to be so dependent on an interconnected world most people don't understand.  (James Burke talked about this kind of danger in his "Connections" program over 30 years ago, a program that made a strong impression on me as a child, but the world of 30 years ago just seems quaint now.)  And it's more than a little dangerous when these people are regulating this world they don't understand, lawmakers who have never used e-mail, whose mental model of the internet is "a series of tubes" and who are constantly surrounded by paid lobbyists representing agendas that often run counter to public interest.

What does all of that have to do with OpenBTS?  One of the motivations for releasing a GSM stack in open source is to help curious people understand how cellular technologies work, to demystify the GSM network by reducing it to a simple form.  This is happening, to some degree, through students and "makers" who have built working OpenBTS nodes as class or club projects.  I think there are about a dozen such systems out there now, not counting commercial development kits, and I love to hear from these people.  Congratulations to everyone who has even tried to run OpenBTS, but especially to those who succeeded.  That first phone call was pretty exciting, wasn't it?  And it was very satisfying to know how it happened.  Granted, we're not educating lawmakers yet, if that's even a meaningful goal, but it's a start.

The Man Burns in 130 Days

We have cleared the legal hurdles to run a test network at Burning Man 2009. This network will probably operate in the PCS1900 band, making it compatible with nearly all AT&T and T-Mobile handsets currently used in the US, as well as with any tri-band or quad-band handsets sold anywhere else in the world.

The current plan is to deploy a system largely intended for local (BRC-only) text messaging. We will also support limited speech service, connecting on-playa calls through user-provided numbers and routing inbound calls through the +883 country code. As a practical matter, the Burning Man 2009 experimental network will be important for testing hardware and software designs for use in rural villages, remote facilities and disaster relief applications.

We will release more details as they come together.

Low-Power GSM in the UK

Back in April 2006, the UK spectrum regulator, Ofcom, did something very rare these days: they auctioned off new spectrum in a standard GSM band.  Specifically, Ofcom auctioned off 12 national licenses in the top 6.6 MHz of the DCS1800 band.  The lucky winners and winning bids are listed on that link, but I'll repeat them here:

• British Telecommunications PLC £275,112
  
• Cable & Wireless UK £51,002
  
• COLT Mobile Telecommunications Ltd £1,513,218
  
• Cyberpress Ltd £151,999
  
• FMS Solutions Ltd £113,000
  
• Mapesbury Communications Ltd £76,660
  
• O2 Ltd £209,888
  
• Opal Telecom Ltd £155,555
  
• PLDT Ltd £88,889
  
• Shyam Telecom UK Ltd £101,011
  
• Spring Mobil AB £50,110
  
• Teleware PLC £1,001,880

A national DCS license for less than US$100k.  Damn.  (I bet COLT felt like chumps when they saw that they were spending something like 10x the typical winning bid, too.)

The catch is that transmitted power is limited to 200 mW and mast heights are limited to 10 meters AGL for outdoor installations.  It seems to me that the 200 mW limitation seems overly conservative, given that a typical GSM handset can put out a full Watt, but Ofcom wrote up a report justifying these limitations on the grounds that they were required for limiting interference with other cells.  Clearly, from the assumptions of the report, Ofcom expects these licenses to be used to provide high capacity over small areas, with each licensee having non-exclusive access to the full 6.6 MHz spectrum.  Otherwise, it would have made more sense to give each licensee exclusive use of a more limited bandwidth at much higher power levels, even if that meant fewer licenses.

So I'm assuming this is all about fill-in pico-cells, but maybe I'm wrong.  I'd love to hear reports of what the license holders are actually doing with this spectrum.  I've also heard of similar low power cellular in the Netherlands.  I can't find as much information on that, but welcome any reports of similar openings in other countries.

The Problem of Spectrum Granularity

[BTW, Greetings from eComm 2009.]

One of the most serious challenges to providing low-cost cellular service in rural areas is the lack of available cellular spectrum. Just about everywhere in the world, all of the spectrum is already locked up by incumbent carriers. So, you might ask, if the spectrum is already held, why don't the people living under it have service? The problem is one of granularity.

Rural areas have lower population density and less infrastructure than urban areas. You need taller towers to get greater range. Your cell sites might not have grid power. The best sites may not be near paved roads. These factors make rural areas more expensive to serve. As the same time, perversely, the people who live in these rural areas have less income, and there are a lot less of them. So if you are a cellular carrier with licenses in both rural and urban areas, you have good motives to concentrate on urban service and ignore the rural areas.

Basic physics shows us that urban and rural areas might require different technical approaches. Basic demographics shows us that expectations of profitability are much lower in rural areas than in urban areas. So how do regulators deal with that? They make it nearly impossible to get a cellular license in a rural area without having to get a license in an urban area at the same time. No, that's not supposed to make sense, but it is true nearly everywhere in the world.

Here in the US, the FCC auctioned most cellular licenses by "metropolitan statistical area" (MSA) or "rural statistical area" (RSA). Despite those promising names, more often than not an MSA or RSA is just a county or group of counties. (Here's the map in PDF.)  That's why I can't get a license for rural Solano County, California, which is mostly sheep pasture and marshes, without getting licenses for several cities totaling nearly 500,000 people at the same time. That's why I can't get a license for Gerlach, Nevada, an isolated town of about 200 people, without getting a license for Reno, a distant city of more than 200,000, in the bargain. What if you want to serve Gerlach but can't afford a license for Reno? TFB (too ... bad). No license for you!

It's bad enough to do business that way in the US, where even the country folk are affluent by world standards, but in developing countries, where the urban-rural disparity is even greater, most licenses are national.  For example, if you want to provide cellular service anywhere in Kenya, you probably need a license for Nairobi.  And since median income in Nairobi is around US$160/mo and the median income in the coutryside is less than US$30/mo, you can imagine what that does for the prospects of a small rural carrier ever happening.

If you wanted a licensing system to discourage rural service, it would be hard to design a more effective spectrum allocation policy.  Some countries are making noises about changing these policies soon.  Let's hope.

NDA and the Path to Servitude

I have a friend with a consulting client (who will remain unnamed) who is using a digital radio receiver (that will remain unidentified).  They are having a hell of a time getting the receiver to work for this client's application, but for a number of reasons that I won't detail here there's a strong motivation to use this particular receiver, regardless of the difficulties.

The problem appears to be in the receiver device driver.  So the client runs a test, and the receiver interface fails, and they send the results to the radio vendor.  The vendor sends back some questions about the test.  They send some answers.  The vendor recommends another test.  They try it.  The vendor asks more questions.  Since the client and the vendor are in radically different time zones, every step of this little dance takes at least a day.  This has been going on for weeks.

So I ask this friend, "Why wait for the vendor?  Why don't you just look at the source code for the device driver and fix this problem yourself?"  Damned good idea, but they don't have the source code because the interface to the radio is proprietary.  Make a nasally whining noise when you say that: proprietary.  No one is suggesting that the vendor should put everything under GPL and give it out to world in a free download.  Hell, my friend can even sign an NDA. 

Just show them the code.  I seriously doubt there's anything there he hasn't seen before.  He's worked with several of digital radio systems over the years and all of the interfaces look pretty much the same.  You have packets or frames of baseband samples.  The packets or frames are timecoded, maybe with sequence numbers, a sample clock, IRIG, SMPTE, whatever.  You've seen the G.711 steam in RTP?  Most digital radio interfaces look a lot like that.  It's the only approach that makes any sense.

On second though, to hell with the NDA.  Once anyone sees the code under NDA their careers are in mortal danger.  What's the problem?  Suppose you sign that NDA and see this proprietary interface and then go off and design another interface for another digital radio.  And since there's really only one way to build that interface that makes any sense, it will inevitably have similarities to the design you received under NDA.  You may well end up getting sued even though you've done nothing wrong.  Legally, those similarities are justified under the "merger" doctrine, but it's not like you just go stand in front of a judge and say "It's just merger, your honor."  Instead, there's a process, a process that takes many months and costs a frightful amount of money and has an uncertain outcome.  And since your ability to participate in this process is directly related to available funds (and not much else), and since if you fail to participate in the process you lose by default, you can easily get railroaded into signing away your intellectual rights to avoid personal financial ruin.  This can happen.  I've seen it happen.  No thanks.  Let them fix their own driver.

Getting back to the original problem, though, what my friend has isn't really a technical problem with the radio interface so much as a psychological problem with the vendor.  As an engineering consultant, maybe he should start charging double to deal with psychological problems.

Protocol Bugs in GSM Handsets

[Mass-produced consumer goods with software faults?!  Say it's not so!]

Nearly all GSM handsets have bugs in their protocol stacks.  If your GSM network is sufficiently complete and correct, it will not exercise those bugs to any degree that will affect service.  But the bugs are there and if you are experimenting with GSM network equipment (like these good people) you will see them.  It would be useful, informative and (now) possible for us to start a public discussion of specific bugs in specific handset models.  To that end, the old OpenBTS sourceforge wiki is offered for that purpose.  It's very much a work in progress, but I would invite anyone with first-hand GSM implementation experience to use and contribute.

The most common handset bugs I have seen are in idle-mode behavior and the handling of the TMSI.  These bugs normally do not affect service, but can represent security threats for high-profile individuals who carry the wrong phones.  Here's one example it gross detail.

A GSM subscriber is ultimately identified by IMSI.  (The IMEI, common in IS-95 and IS-136, is rarely used in GSM.)  Subscriber identities are frequently sent across the air interface in unencrypted form.  That's because encryption cannot be activated until the subscriber's identity is known.  In fact, nearly every transaction in GSM L3 begins with an uplink message that carries the mobile identity, and the LAPDm contention resolution procedure causes the network to echo that first message back verbatim.

If an attacker knows the subscriber identity associated with a person and can intercept GSM control channels, the attacker can use these open identity exchanges to track that person's movement and calling activity from cell to cell through a GSM network. An attacker could also use the IMSI for direct toll fraud in networks that do not perform authentication, which is more common that you might think in some parts of the world. To mitigate these risks, the GSM specification introduces the TMSI, an arbitrary 32-bit tag that can be used in place of the IMSI for anonymizing transactions.

Exposure of the IMSI-TMSI relationship would make the TMSI useless, so in a well managed network these two identifiers never appear in the same unencrypted transaction.  Typically, the handset will use the IMSI for an initial access, the network will then perform authentication and engage encryption, and then assign a TMSI through the encrypted channel.  All future accesses will use the TMSI.  That's what you'll see in most American and European networks, which tend to be well-managed from a security standpoint, A5/1 weaknesses aside.

Now, the TMSI is valid only in the "location area" (LA) in which it is assigned.  The LA is normally the area served by a common base station controller (BSC).  In American networks, this usually means an area with a population of 100,000 to 250,000 people.  When a handset moves to a new location area it is supposed to invalidate its TMSI, forcing the use of the IMSI on the first access in the new LA.  Why?  Because if the phone initiates access with a TMSI, that TMSI will be useless in the new LA.  The network will just have to turn around and request the IMSI in order to authenticate the handset.  When that happens, the previous IMSI-TMSI relationship is exposed, unencrypted, on the air interface.  An attacker who keeps good records of CCCH transactions over several cells can then go back and reconstruct the user's movement and pattern of calling activity in the previous LA.

So there's an example of a common handset bug with security implications.  I welcome reports of others on the new wiki.

GSM WLLs and Carrier Acceptance

The biggest challenge to the deployment of OpenBTS is that all of the world's cellular spectrum is already licensed, most of it to very big companies.  These big companies don't have strong motivation to deploy low-cost services in rural areas.  First, their actual cost of operation is fairly high in rural areas.  Second, even if that cost of operation could be lowered dramatically, it would create a marketing problem.  Solving the first problem will only magnify the second.

Suppose you're "Big Cellular" and you run a GSM network in the developing world.  It costs you $4-$8 per subscriber per month to operate, costing less in urban areas and more in rural areas.  But the people who actually live in rural areas can only afford about $2/month, so you mostly avoid those areas, unless a major road happens to pass through them, carrying your richer urban customers between cities.  Government regulators may pressure you to serve the rural areas, but you can always just show them your balance sheets and argue (honestly, even) that you are already giving the broadest service that can reasonably be expected for a profitable network.  Everyone's happy -- expect for the rural poor who, will never get telephone service under this model.

This is all cozy until a disruptive technology makes $2/month rural service a real possibility.  If you're Big Cellular, that's not good news.  You already have a legacy network that you're still paying for and the new technology is not directly compatible with it.  Even if it were compatible, the new technology creates a marketing problem because your urban customers paying $12/month will soon be demanding to know why they can't get $2 service like their country cousins.  You can try starting a second brand, but that's very expensive and you fear that your new, cheap brand will simply erode your existing market along the urban-rural edge.

The solution here is to make sure that the new service is not a viable substitute for normal cellular.  I'm not saying give the rural poor broken service.  I'm saying give them what they really need, which is reliable telephone service at a very low price, which is not the same thing as cellular, even if the "subscriber terminal" was built to be a cellphone.

The purpose of the new network is to provide basic telephone service in rural areas.  You don't need full cellular functionality to do that.  For example, maybe you don't implement handovers of active calls between cells.  Maybe you don't allow your rural subscribers to roam into "real" cellular networks.  If you are really cheap, maybe you even bind each SIM to a specific cell site, eliminating all of the mobility management functions.  This functionality already has a name: wireless local loop (WLL).  You use GSM like you might use DECT or WiFi, but with much larger service areas and much cheaper handsets.

Operating in WLL mode offers several advantages in this scenario.  There is the technical advantage of a much simpler core network, although a carrier can still support roaming for conventional cellular subscribers if it chooses.  There is the business advantage of no longer being a direct competitor to legacy cellular networks.  And depending on what country you are in, there may be regulatory advantages as well.

If you are Big Cellular, this new low-cost WLL is not a particular threat to your existing business.  It serves a market you would rather not deal with.  Maybe you can open a new subsidiary to operate WLL networks, or, depending on your local regulations, you can lease your fallow rural spectrum to a WLL carrier.  The WLL becomes a modest source of profit.  Universal service can be someone else's problem while you, Big Cellular, can do what comes naturally: market ever more complex services to the cities and gouge tourists with crazy roaming fees.  Everyone is happy again, and maybe this time we can spread it around a bit more.

GPL and Security Applications

It is no great secret that many intelligence gathering processes rely on the ignorance or carelessness of their targets.  That is why parties that engage in intelligence gathering are loathe to reveal the technical details of their tools.  If potential intelligence targets know the tools, they can know the limitations of those tools and take appropriate countermeasures.  Since law enforce and intelligence are (or at least should be) legitimate activities to preserve public safety, it is (arguably) in the public interest to protect information about "sources and methods."

So given that, is there a problem with using copyleft practices in an intelligence or security application?  Not really, at least not if you can trust your own customers to behave responsibly.  The key principle of copyleft open source software is that you must make your source code  available to the customers who receive your products.  That is not at all the same thing as making it available to the general public and even classified software can be copylefted if the license is drafted correctly.

For example, you could, in principle, produce classified software under a copyleft license and still be within the license and the law while delivering that software to a government customer within the same classified program.  You could, in principle, produce law enforcement products, not sellable to the general public, do so under a copyleft license and make the source code available only the the law enforcement agencies that actually buy the products.  Again, this can be fully legal and within the terms of the license.  The key concept here is that even though the end customer is free, under the license, to redistribute the work, they will do not so because of other practical and legal constraints outside of the license.  To be blunt, if you are being prosecuted for a national security violation, a lawsuit from a software vendor is the least of your worries.  Civil intellectual property law is not an appropriate tool for protecting state secrets anyway.

Surprise Purge of OpenBTS from SourceForge

[The SF site was restored a week later.  I'd delete this, but there's a long comment thread.]

Back in December, when an injunction was issued against OpenBTS, I had put in a request to SourceForge to purge the project.  Then I removed the non-compliant material myself.  Then I forgot about the purge request.  Now SourceForge finally got around to purging the project.

For anyone who was on the OpenBTS mailing list at SourceForge, I apologize.  I'm seeing what can be done about restoring the mailing list and web pages.

Until then, I'll try to find another place to host the web pages.

What Stuff Costs, Part 2: CAPEX

There are no "list prices" in the global telecom industry. Every purchase is a negotiated deal with the details covered by NDAs. Prices are arbitrary.  How do the equipment providers get away with that? Let's take a look...

Consider the cost of installing a BTS in rural site.  Something like this:

That costs $200k-$250k, depending on what part of the world you're in.  Most of that money is for "civil installation": site prep, concrete pads, backup power, the mast, that little shack, etc.  There's well over $150k worth of stuff there just to support the BTS.  So what should the actual BTS cost?  As long as it's a lot less than the infrastructure cost, the buyer doesn't care because it won't be a significant part of the total site cost. The baseband processors, transceivers, power supplies and amplifiers for a 3-sector 3-TRX ("1/1/1") kit typically run $20k-$50k, depending on the vendor, the buyer, the specific product and whatever side deals the vendor can offer.  That will give 21 Bm channels at full rate.  There's no point in going below $20k because the savings to the carrier are insignificant below that point.  And the price can't go much above $50k before the BTS becomes significant in the total.  Notice that this price range has nothing to do with the actual cost of producing a BTS, as long as that cost is well below $20k.  The total installed cost is around $75k per fielded TRX, or around $11k per Bm channel.

That's the equipment in the field.  You also need a core network.  The core network gets installed carrier-grade data centers.  As long as the equipment costs less than the data centers, prices just don't matter much.  Together, the BSCs, MSCs and location registers in the core network can easily cost over $5k per fielded TRX, or about $700 per Bm channel.  The civil part probably costs twice that, bringing to total to around $15k/TRX or $2,100 per Bm channel.  The core network also creates a floor for a viable network size, since even a "small" MSC is built to support hundreds of cell sites and priced accordingly.

So the rollout cost is around $15k/TRX for electronics and totals around $90k/TRX for a low-density network when you include all of the civil infrastructure.  One TRX can serve about 1,000 subscribers in the developing world so your rollout capital is least $90 per subscriber, not counting counting other costs ignored here.  Note, though, that the dominant cost is civil infrastructure.  Even if the electronics were free, the total capital would not change by more than about 25%.

The only way to dramatically change the cost of a cellular network is to simplify the infrastructure, something that the existing equipment providers have little motivation to do.  For example, if the whole BTS package can be mounted directly onto the mast and left out in the weather, you can get rid of that air conditioned shack.  If you cut the power requirements, you also cut the cost of the backup power systems.  OpenBTS is radical, though, in its approach to the core network: get rid of it and run BTS units as peers.  Don't just reduce the cost of equipment.  Reduce the amount of equipment.

This is one way that OpenBTS hopes to change the economics of rural cellular service: reducing the capital requirements to build a network. The OpenBTS model can reduce the rollout capital from over $90/sub to around $25/sub, not by offering a "cheap BTS" but by eliminating most of the steel and concrete and generators that a conventional GSM network requires.  OpenBTS can also reduce the minimum size of a viable network to something as small as a single cell site, allowing a carrier to start service with an initial capital investment of less than $30k.  Will carriers go for it, though?  Is there any spectrum available for this new kind of carrier to emerge?  We're working on it...

What Stuff Costs, Part 1: OPEX

Most African cellular carriers are partly owned by corporations like Millicom and Vodaphone that are traded on stock exchanges in Europe and America.  They publish regular financial reports.  From those reports we can tell that the typical 2007 African cellular subscriber paid $10-$12/month to talk on the phone for just over half an hour.  That sounds like a rip-off until you do a little more math and realize that it actually cost the carrier about $6/month to provide the service, not counting the cost of internetworking.  What the heck?

Let's say, for simplicity, that all of the traffic is compressed into 6 hours each day, so that you see a load of about 0.003 Erlang per subscriber during this peak traffic time.  A minimum 3-sector GSM BTS site provides about 10.5 Erlangs at 2% blocking and thus serves 3,500 subscribers at your typical daily peak load.  If your cost of operation is $6/sub/mo, that corresponds to a cost of about $252k/year per BTS site to run your network, with most of that cost in the BTS site itself: about $200k/year. (!)  When we first estimated this, we though we'd misplaced a decimal point somewhere.  Then we did we read this article in Balancing Act that put the cost of operating an off-grid BTS site in Africa at around $210k/year.  Then we talked to some telecom people from Africa who said the cost was well over $150k/yr but they didn't know by how much.  So it probably really is around $200k/yr.  Why?

It's all about power.  Suppose you have a BTS that draws 5 kW.  And since it's in the tropics you have to cool it, which brings your power budget up to 7 kW.  To supply that, you need a generator.  And since a generator is a target for theft, you need security lighting and cameras, which drive up your power budget and add at least 1 Mb/s to your backhaul requirement, which requires yet more power.  Before long, the site is drawing over 1o kW continuously and you are burning at least 25 gallons of diesel fuel every day.  Now you need a crew with a truck to drive around fixing generators and fences and filling fuel tanks, which is complicated by the fact that most of these sites aren't even near roads.  It starts looking like war logistics, where Sun Tzu tells us that every sack of rice at the front cost 10 more just to get there.  By the time you have everything in place you're spending nearly $20k/mo to keep this beast running.

This matters a lot to the long term development of these countries, because most of the people who live out in the countryside cannot afford $6/mo for anything, meaning that they will never get telephone service, not even on a non-profit basis.  To achieve universal service, someone will need to try something completely different.

So here's the good news: if you can keep site power consumption down to just a few hundred Watts, this all changes dramatically.  Instead of a generator, you can run the whole site on solar panels or microturbines in many parts of the world.  No more diesel fuel.  No more crews in trucks.  Every two years, you replace the batteries in the power system.  That's all.  That's why the design target for OpenBTS is 75 Watts per transceiver, a target that we are very near already just using off the shelf equipment.

Other other cost components in the subscriber rate are internetworking and capital amortization. Most connections between African carriers happen in Europe. That means that if you call from your MTN cell phone to a wired phone down the street that call may well get routed through France at French long distance rates. And the capital cost of rolling out a rural GSM network is at least $100/subscriber.  But those are topics for other posts.

A Tale of Two Licenses

OpenBTS includes a partial implementation of the GSM air interface, Um, the radio link between a GSM handset and its serving basestation. (By "partial", I mean it includes the minimum set of features to support speech telephony and, in release 2.0 and later, text messaging.)

Now, before I go any further, let me say this it not legal advice. This is my best understanding of a situation based on discussions with some strong authorities in the FOSS world.

Since late September 2008, after some convincing from John Gilmore, all distributions of OpenBTS have been under GPLv3. We like GPLv3. We'd do everything under GPLv3 if we could, but that's a story for a different blog. With OpenBTS there's a catch, though. Even though GSM is a publicly available specification, it is not"open". Many essential elements of GSM are covered by patents. These patents are held by companies like Ericsson, AT&T and Alcatel and are registered in the ETSI IPR database. The current GPL distributions of OpenBTS are offered for only private experimental use, which is generally exempt from patent licensing. Furthermore, OpenBTS is presently distributed as software, not an actual, usable end product. Anyone using OpenBTS is expected to comply with all applicable laws, including patent laws.

But let's say you use OpenBTS in a complete product that provides GSM service. You got the source code under GPLv3. And you have licenses for GSM patents. And you sell your GSM product to network operators. Section 6 of GPLv3 requires that you make the source code available to your customers and Section 11 requires that you extend your GSM patent licenses to anyone to whom you distribute the source code. Because these GSM patent licenses cost money and are granted under limited terms, these requirements appear to be in conflict. (This is not a hypothetical situation, BTW.)

Thankfully, there's a loophole of sorts. Look closely at Section 6. It does not say you must distribute the source code. It just says that you must make sure that people who have your product know where to get that source code. So the key to delivering a commercial GSM system under GPLv3 is to make sure that there is at least one party to distribute the source code who

1. does not hold GSM patent licenses,
2. has up-to-date copies and
   
3. did not receive that code under GPLv3 from anyone who does hold GSM patent licenses.

Note the compound in requirement (3). A party can meet (3) by receiving the code from a party with no GSM patent licenses or by receiving the code outside of GPLv3. There are two parties in the world who have complete releases of OpenBTS who didn't get it under GPLv3: the authors (Kestrel Signal Processing, Inc.) and the Free Software Foundation, who were granted copyrights on 24 October 2008. Either party could fill the role of the distributor as long as that party does not hold GSM patent licenses.

So, here are some scenarios for equipment providers wanting to use OpenBTS:

• Kestrel distributes OpenBTS to an equipment provider under GPLv3 and then Kestrel makes the source code available to that equipment provider's customers in compliance with GPLv3. That works as long as Kestrel doesn't hold GSM patent licenses and the equipment provider does not want to add proprietary features.
• Kestrel distributes OpenBTS to an equipment provider under GPLv3 and the FSF makes the source code available to that equipment provider's customers in compliance with GPLv3. That works as long as the equipment providers' software is identical to some public release of OpenBTS. It does not work if the equipment provider wants to add proprietary features.
• Kestrel provides OpenBTS to an equipment provider under a different license that relieves the equipment provider of all GPL obligations. This would allow the equipment provider to add proprietary features to OpenBTS and operate with no ongoing reliance on Kestrel or the FSF. It is probably the option that most equipment providers would choose. This would cost money, though, since OpenBTS uses other GPL libraries that would need to be sub-licensed.

And here are some scenarios if the project founders want to get into the equipment business:

• Kestrel starts producing a complete turn-key GSM box, not just source code. Now Kestrel needs GSM patent licenses, so Kestrel no longer meets requirement (1) and can no longer distribute OpenBTS under GPLv3. But Kestrel could transfer source code to FSF let them distribute it under GPLv3 to preserve the open source project.
• Kestrels' owners set up a new, distinct company to produce and sell GSMequipment and hold the GSM patent licenses. Kestrel extends a non-GPL license to that new company. This last option preserves Kestrel's ability to release under GPLv3, but still allows the project founders to pursue the equipment business.

Is that complicated enough for everyone?

Putting SMS in the SIP World

SIP/SIMPLE defines two models for messaging.  In the "pager model" (RFC-3428), the message is simply transferred with the MESSAGE method and the receiver responds with "ok" or "accepted".  In the "session model", the two SIP endpoints establish a session with the INVITE procedure, transfer messages within the session, and then close the session with BYE.

SMS naturally follows the paged model.  Radio channels are scarce resources and cannot be held idle for long periods in any practical system.  A typical SMS transaction occurs outside of an established call and the radio channel is held just long enough to move a single message.  Store-and-forward handling is an absolute requirement, too.  When a short message is submitted to the network by a handset, the payload part (the TPDU) must be held until a delivery channel can be established to the destination . Even if the destination handset has service, this can take several seconds. And if the handset does not have service this can take hours or days.

When we first contemplated SMS for a SIP-oriented core network, we thought we could use a messaging server like Jabber to replace the SMSC.  That doesn't work.  Most messaging servers follow the session model.  And why not?  IM applications are session oriented and maintaining an open session in the IP world is very cheap.  These messaging servers are built for chat sessions and presence, not for storing and forwarding one-off messages to intermittently connected handsets.

So the OpenBTS project is off to build its own store-and-forward page-mode messaging server, probably from MySQL and some simple scripts, using Kannel as a gateway to conventional GSM SMSCs.  We'll post more as it happens and welcome any suggestions.

Free As In Freedom

I never used to have strong opinions about open source development. Then I got sued.

I started working as an independent consultant in late 2001. One reason I "went indy" was that I refused to ever again sign a standard engineering employment agreement that would give my someone else ownership of everything I do or think.  I would never again submit to servitude.

In my consulting work, I was careful to read NDAs. I knew not to sign "derivative works" agreements. I knew that most non-compete agreements were illegal in California. I was clear with my clients: "Your project is not the only thing I'm working on and you don't own those other things." I thought I was savvy enough to protect my intellectual freedom. I was wrong.

[...]

To avoid this situation in the future, Harvind and I now insist that any software work based on a publicly available specification be delivered under an open source license. And if a potential client refuses to do business that way?  We walk away because those people are not worth the trouble they will cause down the road.

A Lie Agreed Upon: Getting Hybrid Cellular into the Field

I just got back from 25C3 in Berlin. Thanks to Delta, I had a miserable time getting there, but I'm still glad I went. I met a lot of good people. I tried Club-Mate. I tried my best mangled German with the Reisegepäck staff at Flughafen Tegel. On the last night, Harald Welte treated me to a steak dinner, one of the few proper sit-down meals I had on the whole trip.

I met a lot of telecommunications professionals and we discussed the problem of carrier acceptance of the OpenBTS approach: providing a simple set of services at minimal cost and replacing the GSM "core network" with collection of peer-to-peer SIP applications.

There was a general consensus that the OpenBTS approach was technically feasible, even on large scales, and could be integrated into existing GSM core networks if needed.  There was also a general consensus that most incumbent carriers would reject the technology, even if integration into core networks is easy, largely on economic grounds.  The simple truth is that nobody in the telecommunications industry is really interested in making a modest profit by serving large numbers of very poor people.  The typical cellular executive would rather talk about extending 3G or 4G networks into rural Africa and then not do it.  Talking about bringing scorching fast networks to the poor is much more exciting than actually building a 2G system that they can afford to use.  

This consensus is not new to me.  I have had nearly the same conversation several times over the last two years with a handful of ex-employees from African cellular carriers. The big carriers will continue to concentrate on squeezing more revenue from their more affluent customers by offering more complex services. In the meantime, these big carriers will continue to sit on spectrum that is completely unavailable to the people who live under it, or, in the case of South Africa, cover the whole country with services that only a small minority can actually afford to access.  To borrow a phrase from Mark Twain, universal service is "a lie agreed upon" for the telecommunications industry.  It won't happen, even at modestly profitable levels, unless regulators force it.

This does not mean that OpenBTS will not find a commercial market.  It just means that it will not find a market with incumbent commercial carriers until regulators force them to get serious about universal service. That won't happen until early adopters, most likely small rural carriers, use OpenBTS to demonstrate that self-sustaining universal service really is possible.  So we're looking for early adopters.