The Man Burns in 341 Days

We returned from Burning Man over two weeks ago and are still unpacking and recovering. The skin on my hands still feels like old leather, the rebar cuts and blisters on my legs and feet are still healing and Jessica is still sorting through the boxes of dusty camping gear and surplus food in the garage. It was a weird burn; some of our friends and campmates had powerful experiences, good, bad, transformative and moving. Even a weird time in Black Rock City leaves you looking forward to next year, and reminds you that the event is more than just a big party. But that's not what brought you to this blog, so here are the technical highlights:

• We ran a 2-sector, 5-TRX system (3/2 configuration) from a 25 m tower. We ignored RACH bursts with TA>10, limiting our range to 5 km, deliberately excluding nearby towns from the test. Our coverage footprint was roughly 80 sq-km, solid over most of Black Rock City, with the exception of some of the outer streets past 3:00 & I. We could make very good-sounding calls from the airport, Center Camp, the gates and from 9:00 & the trash fence.
• Our second-generation radio worked like a champ.
• Speech calls were limited to three minutes.
• We powered the BTS units from a PV solar array. We had a generator, too, but that was mostly for power tools and the blender.
• We encountered roughly 40,000 unique IMSIs. Really. We were shocked, too.
• We had challenges, even for Burning Man. Our neighbors advised us that Mercury was in a retrograde phase, putting a kind of curse on all communications systems, but we and Papa Legba prevailed.
• It was Tuesday before we had a stable backhaul.
• Commnet Wireless unexpectedly came online on Thursday afternoon. Even more unexpectedly, they did so in the license block for which we had previously been cleared by Verizon. So we lost half a day double checking our license and re-coordinating spectrum with the Commnet NOC.
• We had congestion on Friday and Saturday. We were running nearly twice the capacity as in 2009, but there seemed to be at least twice as many phones in the environment, maybe more.
• We had about 4,000 autoprovisioned users, connected about 7,000 phone calls and processed about 50,000 text messages.

Heros of the Burn for 2010:

• Arturo Mayorga Cerda, for setting up the tower with nothing but hand tools.
• Jessica Burgess, for climbing the tower on the last day to connect the crane.
• Mark Petersen, for staying late to help pack up the camp, even though he was ill and feeling badly and probably should have been in bed resting.

And general thanks for 2010:

• DPW for driving and removing our anchors and for sending the boom truck to take down the tower.
• John Gilmore for supporting smqueue and loaning us a switch, Christmas lights and lots of other goodies. And for bringing some interesting people to the camp.
• Donald Kirker for on-site e-mail and SMS hacking.
• Mark Petersen again for Asterisk support and camp photography.
• Glenn Edens for staying home, manning the office and dealing with the thousands of e-mails we have been receiving as a result of recent press coverage.
• Lisa Hyde for the great 3-minute warning message and our BMIR PSA, which I would like to find a copy of. (If anyone has a copy, that would be great.)
• Jessica Burgess again for rounding up the groceries, organizing the bar and generally keeping the camp organized while we played with computers and radios.
• Tim Bowden for loaning us the solar panels.
• Ralf Muehlen for IP support and general moral support.
• All of the well-wishers who stopped by the camp.

I'll add some photos later, but Donald posted a good set on the Flickr that will do nicely for now. We are already looking forward to next year.

Burning Man 2009: Days 10 & 11

Ok, so I'm a month late with this...

Monday, 7 September, was day 10 on the Playa. Black Rock City was packing up and it was time to "leave no trace". The biggest task was to pull down out 70-foot tower. Pavel had started stripping the tower the afternoon before by removing the WD9XSP lighted sign. He started this new morning by removing the antennas.

(He's about 60 feet up there. We took this through a telephoto lens.)

While Pavel did that, most of the rest of the campers packed up the shelter and I bicycled down to Heavy Equipment Camp, part of the Department of Public Works (DPW), with a half liter of tequila. I was going to see if I could get a crane to help pull down the tower. When I got to Heavy Equipment, I walked in through the gate, set my donation bottle on a table next to the coffee pot and took a seat with a few other would-be customers in the "lobby": a bunch of old sofas drawn around a low table under a shade tarp. This is how you borrow a crane in Black Rock City. We were waiting for Chaos, the dispatcher. I was informed that the weekend burns had been very busy for DPW and that Chaos was sleeping in. No problem. On the coffee table there was an overflowing ashtray and can labeled "Tactical Bacon" with an outline of an assault rifle on the wrapper. After a while, an older gentleman came out from a shipping container with 3 or 4 VHF radios clipped on his vest. It was Chaos, the same guy who had inspected our tower installation a week earlier. He walked into the lobby area, talked to each of us and took a few notes. He told me they would be there when they got there. I went back to camp.

We kept packing. The neighbors loaned us a leaf-blower for clearing the dust out of the back of the truck. John came by and we opened a bottle of peach lambic left by a supporter a few days earlier to toast all the people who had helped us so much over the last month. Then we took down the shelter and started contemplating how we were going to remove the guy-line stake from the rock-hard ground, a task we could not safely start until the tower was down. John got impatient waiting for the crane, climbed up the tower with the gin-pole and started pulling it down himself.

John took down the top segment (#7) of the tower himself and was starting to remove segment #6 when the crane arrived. He connected the crane hook to the tower and climbed down. We detached all of the guy-lines and watched the crane lift the whole tower and lay it on its side. It took about 5 minutes.

We spent the next two hours pounding the guy-line stakes out of the ground with sledgehammers. The trick, we found, was to swing at just the right angle to hit the inside of the eye. It was more important to hit accurately than to hit hard. It just took patience.

By the time we were ready to drive out, the sun was already setting. We said our goodbyes, mounted up and went to sit in the 2-hour traffic jam on the way out the gate. We ended up spending the night in Reno due to car problems, finally unloaded most of the camp into a rented storage locker on Wednesday and spent the rest of the week blowing dust out of our equipment. And we are already scheming for next year.

Burning Man 2009: Coverage

On Monday (day 3), we had made our first coverage estimates, sending SMS to ourselves or to short codes that returned status information. We did this a few more times over the course of the week. We had hoped to use RRLP to automate the coverage estimation process, but so few phones were returning useful RRPL results that we had to do it by hand, with a bicycle, a test phone and a note pad. This is a rough estimate of our -90 dBm coverage within BRC:

"A", "B" and "C" are our three sectors, arranged to maximize coverage over the city and insure coverage at the front gate, the airport and the greeters' station. The "x" at 4:30 marks our camp and tower. The "0" at 3:00 marks Fusion Valley, the first link in our backhaul chain. These coverage estimates match those predicted by the Hata suburban model, the same propagation characteristic observed in BRC in our 2008 test.

Despite the failure of RRLP to produce data, we did get some automatic coverage information from measurements of the timing errors of RACH bursts arriving at each BTS. In a GSM system, the timing error of an arriving RACH burst relative to the signal framing yields a rough estimate of the round-trip propagation time from the BTS to the handset and back. We logged every one of the 2,018,321 validated RACH bursts we received and finally got around to sorting through some of that data late last week.

Analysis of RACH burst timing indicated that most of the arriving RACH bursts came from a distance of about 1 km (mode of 1 km and mean of 1.3 km) and nearly all of them (>99%) came from a distance of 3 km or less. That makes sense given our coverage, our tower location, the geometry of the city and the fact that BRC accounts for >90% of the area's population during the festival.

Outside BRC, though, we predicted that coverage would follow the Hata open rural model, meaning that once a handset got out of the shadow of all of the RVs, shipping containers and metal-framed domes, our range could be 15 km or more. There is some evidence of that in our logs. A small fraction of RACH (~0.5%) bursts arrived on sector A from clustered distances of 11 km and 15 km. Gerlach, NV was 15 km away in the direction of sector A, has a population of about 500 and could account for the cluster of access attempts at that distance. We still don't have a good explanation for the cluster of access attempts at 11 km.

Burning Man 2009: Days 4-6 (part 2)

Sorry for the posting delay. Back to the blog. For now.

Apart from excessive registration activity, we had a few other technical challenges, most related to backhaul and power.

Our backhaul was a point-to-point WiFi link, running from a Nanostation 5 on our tower to a 30' tower in a camp called Fusion Valley, over at the 3:00 plaza. Fusion Valley, in turn, linked us to Center Camp where we got onto a 10 MB/s microwave relay back to an ISP in the "real world". The people running this network had been very supportive of our project and their system worked most of the time, but keeping anything working reliably on the Playa is hard and they were not granted any magical protection against that. Our static IP number changed and sometimes just quit working. A few times we lost backhaul completely, like the time someone at Fusion Valley plugged some power tools into the same circuit as the network gear, tripped the breaker and then just walked away for a while. Well, it's Burning Man. Stuff happens.

We were running a mostly-local service. Occasional loss of backhaul should not have been a serious problem, but it was, because Asterisk kept trying to run DNS and reverse-DNS queries. Asterisk would just lock up when it couldn't reach the internet. We tried replacing every hostname we could find with a numeric address, we tried filling out /etc/hosts by hand, we tried setting up a local DNS cache, we really tried our best to understand and trim down our Asterisk configuration, but we just could not stop Asterisk from freezing when the backhaul was down. Since we were using the Asterisk SIP registry has our HLR for location updating and SMS address resolution, a frozen Asterisk server also shut down everything else we were doing.

Another technical problem was power. Here, we did something dumb: we assumed that new batteries would be topped off, and so we didn't bother checking the acid levels. We had enough battery capacity to run the full system for at least 10 hours but the first time we tried to leave the system on overnight we woke up to a low voltage alarm at 4:00. Since we didn't want to run a generator right next to the tent at 4:00, we shut everything down and went back to bed more than a little concerned about the batteries. The next morning we started the generator but the batteries were not responding well, not taking a charge. By the afternoon, though, we thought to check the acid levels and found they were very low. We topped off the batteries with water and had no more problems that week, except for a little boil-over from over-filling. BTW: Playa dust is excellent for neutralizing battery acid spills.

There were also a few loose ends for John to tie up in the SMS server, like saving and reloading the message queue and "bouncing" undeliverable message back to their senders. He also added some test features, like a "411" short code that would return system status information, that were really handy for coverage testing. These weren't really problems, though, just straightforward development.

By day 6, Thursday, we had fixed the registration loads, had decent power and backhaul, and a good feature set on the SMS server. We were (finally) starting to have a stable system.

Burning Man 2009: Days 4-6 (part 1)

OK, this is where the blog gets technical.

Day 4 (Tuesday) started with fried eggs, cornbread pancakes and Turkish coffee, followed by the departure of the tower crew. We were all disappointed that they could not stay for the full week. Hopefully, they will next year. After we said our goodbyes and Martin's SUV disappeared into the dust, we turned our attention back the the BTS units.

When we first turned on the system at full power, we got flooded with location updating (GSM registration) requests. We had expected this. We had seen it the year before. We had heard about it from network engineers from big carriers. I had seen it in IMSI catchers. We expected it would die down after a couple of hours, too, but there we were wrong. Now what? We had to determine the cause of the excessive load, we had to slow it down, and we had to make whatever changes we could to allow the system to serve provisioned test users even in the face of this load.

Over the next two days of experiments and analysis, we found multiple causes for the excessive registration load.

• We were using the "IMSI attach" protocol, meaning that any time a phone slipped out and service and came back it would attempt to register again. That's great for keeping up-to-date presence information in small networks, but it was a disaster in a large network with spotty coverage along the edges.
• We were trying to do RRLP queries during each registration. RRLP is the protocol used to communicate with the GPS receivers inside most US handsets. We had hoped to make maps of coverage in real time and Alon had put a lot of work into that. But without GPS assistance data the useful RRLP response rate was less than 1% and a lot of the requests were timing out, causing phones to spend more time on the radio channel and exacerbating the loading problem.
• We had the cell reselect hysteresis set too low, causing phones to change cells (and thus request registration) too frequently.
• We had the registration timer (T3212) set too low. Again, a small T3212 was good for presence tracking in small networks, but a poor choice in this environment.
• We were using a random exponential back-off for access requests but were resetting the back-off timer value (T3122) any time there was a successful channel grant. This was too permissive.
• There was another GSM network on the playa, Commnet Wireless, operating from Frog Pond, a hot spring on private property a few miles away. Any time a phone passed into Commnet Wireless' service and back out again, it would request another registration.

Most of these problems could be fixed with configuration changes and software modifications, however:

• We could not set T3212 above one hour because asterisk would not allow us to set the SIP registry timeout to more than an hour.
• Commnet was just something we would have to live with.

(Aerial photo of the Commnet site at Frog Pond.)

Fixing those problems greatly reduced the registration problem. We made two additional changes, though, to insure smooth operation:

• We reserved a few control channels for non-registration activities. This insured that provisioned users would have a chance to get service even during surges of registration activity.
• We added a control loop that would automatically adjust downlink power to limit congestion. This was a huge improvement in terms of overall system behavior, since it would automatically ramp up power at just the right rate whenever we had to restart a BTS unit.

Of course, congestion management was not the only problem we had to solve. More on that in part2.

Burning Man 2009: Day 3

On the morning of day 3 (Monday) the tower was up but not fully wired. Martin, the sponsor who had actually provided the tower, climbed up to have a look around. Our neighborhood was getting built out quickly on this first day of public access.

The first order of the morning was for Arturo to mount the third cell sector antenna, run the RF cabling, and set up our wifi link. Once that was complete, Arturo took a break to use Bill's spiffy field shower and Star, a visitor to the camp and friend of the project, volunteered to install the light-up call sign that Bill had brought.

By the late morning, we had the BTS units laid out on a trailer and wired-up under a shade tarp. We were ready to start testing.

First, we started up the BTS units without the power amplifiers and made a few test calls with handsets in the tent. Things looked normal, so we turned on the power amps. We got an immediate flood of registration attempts, which was expected. The flood continued, though, for well over two hours, which was not expected. By the third hour we turned off the power amplifiers, took a look at the logs, and started to think about why the registration load was so high and what we might to do manage it. (More on that the next post.) Meanwhile, the tower guys (Arturo, Victor and Martin) sat out on the back of the truck playing guitar and signing Mexican tunes, which was a nice touch on a windy afternoon in the high desert.

That evening, we made beef stew with the leftover meat and bones from Sunday's steaks. The tower crew got a much-deserved night out on the town in Black Rock City, starting with a tour of the city from Ranger Velveeta (Annie) and ending with a visit to Vamp Camp in the 7:30 plaza. We met back together at 3am and talked about plans for the next morning.

Burning Man 2009: Days 1 & 2

So I'm finally getting around to writing up some of our experiences at Burning Man.

Let's start with the first two days. We came in with early arrival passes on Saturday 29 August and most of us arrived in the afternoon. The first order of business was to set up the shelter before dark. John Gilmore dropped by to lend a hand and say hello. Then we set up the kitchen. The tower crew arrived that evening and we made all had a hot dinner together in the cool desert night.

The plan for the next morning was to start early and have the tower erected by noon, but no plan survives contact with the Playa. The first complication was that the ground was much harder than the tower crew had expected. They were expecting sand and loose rock, typical of their other desert installations in northern Mexico and the southwestern US. The Playa was different: 20-30 cm of brittle gypsum laying over dense, damp clay. They had made a set of screw-in anchors, but the screw pitch was too steep and anchors were impossible to drive in this hard ground. They considered borrowing some welding equipment from DPW to re-engineer the screws, but in the end we just stripped off the threads and drove the bare stakes (each about 1.7 m long) directly into the playa with a sledgehammer. Dealing with the anchors blew the whole morning and by noon it was starting to get windy.

We spent the rest of the day erecting the tower, which meant a lot of sitting around waiting for short breaks in the wind, especially as we got closer to the final height of 21 m. There was also a brief distraction when a gust of wind nearly took our shelter, ripping most of the grommets out of our tarps and breaking several of the PVC ribs. Mr. Gilmore literally saved our camp with a box of spare nylon webbing and clips left over from his own shade structure.

Meanwhile, Bill was busy rigging up our shower and greywater evaporation pond so that we could clean up properly when all this backbreaking work was complete.

By late afternoon, our main tower technician, Arturo, was hoisting the final tower section, with two antennas attached. He wrestled them into place in a 40-kt wind and a small crowd on the ground applauded. Somewhere in all that drama Cameragirl and Chaos came by to "inspect our erection" and critique our guy lines. We contemplated putting a wind turbine on the top of the tower, but decided against it. Given the wind conditions, that wind turbine could have supplied most of our power most of the time, but we were concerned that the installation would have been just too dangerous, so we would be relying on the gasoline generator all week.

That evening there were 12 for dinner in the camp. We had steak and mashed potatoes and everyone toasted Arturo for his tower work. He was definitely the man of the evening. Even though the tower itself was erected by sundown, the electronics were still not fully installed, meaning that we were at least half a day behind schedule already. But we had set up a 21 m tower in a windstorm without any injuries, so we saw the day as a success. Such is life in BRC: you frequently adjust your expectations to match reality.

We're Back

We cleaned out and returned the truck today, marking the end of our 11-day site test in Black Rock City, Nevada, site of the Burning Man festival. Like every other project in BRC, we had our ups and downs, but the trip was a success in the sense that we learned an awful lot about operating a full-range cell site under continuous load from thousands of handsets.

There were a lot of successes. We were happy with the hardware performance and packaging. We rigged a 70' tower in high winds with no injuries. Operating range matched our predictions. We discovered and fixed several bugs under conditions that would have been hard to simulate in a lab. Backhaul integration worked, both for +1 NANP and +883 iNum.

There were also problems. We had problems with Asterisk configuration and performance. We lost IP connectivity a few times every day. The presence of the Commnet Wireless GSM network complicated our operations in unexpected ways. Test users did not follow even the simplest of instructions. We will better understand what really happened as we sort through the hundreds of megabytes of logs in the flash drives of the BTS units and the CDRs and logs of Asterisk and our SMS sever.

There were delights, too. We had visits from a lot of well-wishers and supporters, many of whom volunteered their efforts and brought us gifts. We had good neighbors (the "popcorn guys") and shared dinner with them on a few nights. And we had a lot of fun, both in the hacking sessions and out on the playa. (Alon even chipped a tooth fighting in Thunderdome at the Goth camp, part of the complete Burning Man experience.)

We'll write this all up over the next few days as we sort through the data, but I'll close with some photos of our BRC neighborhood (4:30 & H) under the full moon, me and Alon in our network "ops center" (three laptops and a couple of PAP2 phones in a tent) and us standing around looking concerned while Pavel climbs the tower to strip it:

We Loaded the Truck Today

We are ready to take open source, open access cellular to Burning Man. Some photos from the last few days...

Kitting out the parts at the beginning of what turned out to be a very long week:

Harvind leaving Sam's with steaks, cheese, wine, canned fruit and ... RV batteries:

(Yes, we plan to eat well.)

Acrylic sheets that cracked overnight:

Laying it out again on aluminum:

Starting from the upper left and going clockwise, that's the power amp, the USRP, the single-board computer, the uplink cavity filter, the duplexer, the pre-amp and the power buses.

Finally, we have the three main-site BTS units and some test equipment in the truck:

(Check out the classic Tektronix 7000 mainframe on the left. Nearly as old as me and tough enough for the Playa. There's a R&S CMD-57 in the box. Not quite tough enough for the Playa, so it will live in the box until needed.)

Time for bed. We have a long drive tomorrow.

We Load the Truck Tomorrow

We are still building the hardware for the Burning Man GSM test network. We should have finished that by Monday, but:

• We used acrylic sheets for mounting radio components the first assembly. Big mistake. The sheets cracked overnight from the strain of the screws. We rebuilt them with aluminum yesterday.
• A soldering technician disappeared with our power amps and half of our USRPs. We'll deal with him when we get home. Or maybe have our lawyer deal with him next week. In the meantime, we have a system to build and are expecting replacements today.
• Some of our cavity filters got damaged in shipment and had to be returned for service. We got them back yesterday. I want to thank Anatech Microwave for their excellent customer support on that emergency repair.
• The USPS appears to have lost our wifi gear. Hopefully, a second shipment arrives tomorrow. Thanks to PCBAY.com for doing their best to clean that up.

The good news is that we've had some volunteers step forward to help, especially Alon Levy, John Gilmore and Donald Kirker, and their help is saving this project. We still plan to be on the Playa by Saturday afternoon and running the main site by Sunday afternoon. It has just meant a few long nights this week.

The Man Burns in 27 Days

Plans to run a cellular system at Burning Man are well under way. We have an FCC license and spectrum coordination with Verizon in the GSM850 band. We have most of our equipment in hand and are starting final assembly this week. We have a store-and-forward SIP/SIMPLE server, thanks to John Gilmore. We have official camp placement and early access to the site. We have a block of 10,000 iNum phone numbers in country code +883, thanks to Voxbone. We have a 70' tower and an installation crew, thanks to Martin Pelayo. Things are on schedule and a lot of people have stepped forward to help us and we thank them all.

We heard a story that Larry Harvey was very concerned when he got wind of our plans, thinking we'd turn the Playa into some kind of chatfest. I'd like to assure Mr. Harvey and anyone else that we don't have the bandwidth to provide normal calling service to 50,000 people, nor do we have roaming and settlement agreements with cellular carriers. We couldn't light up BRC with normal speech service if we wanted to, and we don't want to. Want we can do is provide speech service for about 1,000 early subscribers, which we assume will mostly be early arrivals, BM staff, Rangers, DPW, perimeter patrols, etc. After that, we will have enough bandwidth left to run about 1,500 SMS transfers per minute, allowing us to provide text service to pretty much anyone who wants it. We are hoping participants will find this service useful, as a means of locating friends, meeting new people and getting information about Playa events.

We'll post more on the details of using the service soon, after we have made a few final decisions on network configuration and policies.

The Man Burns in 130 Days

We have cleared the legal hurdles to run a test network at Burning Man 2009. This network will probably operate in the PCS1900 band, making it compatible with nearly all AT&T and T-Mobile handsets currently used in the US, as well as with any tri-band or quad-band handsets sold anywhere else in the world.

The current plan is to deploy a system largely intended for local (BRC-only) text messaging. We will also support limited speech service, connecting on-playa calls through user-provided numbers and routing inbound calls through the +883 country code. As a practical matter, the Burning Man 2009 experimental network will be important for testing hardware and software designs for use in rural villages, remote facilities and disaster relief applications.

We will release more details as they come together.