Illustrations by Mike Reddy
All track on the New York subway (and on most American rail) is broken into sections, here about 1,000 feet long. An electric current is constantly running in a loop through each section. When a train enters a section, it short-circuits the loop, which allows the system to know that the section is occupied. The signals behind it automatically turn red.
Fixed-block signaling systems, in use since the late 1800s, keep trains from getting too close to one another. Photo from the Regional Plan Association.
The neat thing about subway signals, as opposed to the ones you find on the road, is that they actually force you to stop. When a signal is red, a footlong metal T called (appropriately) a “train stop” protrudes above the track; each train car has a corresponding “trip cock” on its wheel frame connected to the emergency brakes. If you were to drive by a stop signal the train stop would hit the trip cock and you’d screech to a halt.
When New York’s signaling system was first being installed—much of the work took place at the turn of the last century, with another big wave during the FDR-fueled construction boom of the 1930s—the designers wanted to make it impossible for there to be a collision. They wanted intelligent signals: signals where even if you made a mistake, even if you wanted to, you couldn’t command two trains to simultaneously drive into the same section of track.
This becomes especially important when you have tracks that cross, and switches that route trains from one track to another. In a subway system this happens all the time. New York’s is especially hairy, since many lines have both an express and local track going in each direction. Some big stations can have as many as a dozen lines connecting.
The way you do it is by having what’s called an “interlocking.” An interlocking is just a configuration of signals, switches, and their controls that is set up in such a way that you can never have an unsafe state. In the early days, this was enforced by levers that literally interlocked. If you want to flip a switch over here, you first have to make the signals over there red. If a train entered this section over there, the switch over here would always be disabled.
To design even a single interlocking was, and still is, “terrifically complicated.” A thousand considerations—the interactions between signals, switches, and trains; the curves, grades, and other track conditions that affect train speed and braking distances—go into their design. Greenberg, also a composer, likens the problem to writing classical music. “Interlocking is a kind of counterpoint for subway tracks; each track adds about as much complexity as a line of a fugue.”
A 1968 plan for a small section of New York’s trackage and interlocking, showing the placement of signals and the part of the track whose safety each signal controls. Photo from nycsubway.org.
The switches are connected by wire or still, in some cases, by actual levers, to rooms called “towers” where operators can see which sections of track are occupied, what color the signals are, and what the state of each switch is. By pulling levers and talking to drivers over the radio, they clear paths for trains. The interlockings ensure that even if they do something wrong, they can never clear two trains to take the same path.
A view of the interlocking panel at the signal tower in Bowling Green station. Photo from Jersey Mike's Rail Adventures.
The reason there are no realtime countdown clocks on the F line is that even the tower operators don’t know which train is where. All they can see is that a certain section is occupied by a certain anonymous hunk of steel. It’s anonymous because no one has a view of the whole system. A hunk comes into one section of track from somewhere else; the tower’s job is to get it through their section efficiently. The next tower they pass it to will likewise not know whether it’s an F, say, or a G. When there are incidents, trains are located by deduction.
The equipment is old and breaks all the time. In fact it’s so old that the MTA can no longer buy replacement parts from the manufacturer; it has to refurbish them itself. Some of the controls for the interlockings are originals from the 30s. Much of the wiring is still insulated with cloth, instead of rubber; ten years ago the entire Chambers Street interlocking caught fire. Salt water from Hurricane Sandy did damage to trackside switches and signals that is still being repaired.
Inside the Signal School there is equipment from every major era, since it’s all still active in various parts of the system. As a demo, Habersham at one point flips an old-style switch on the big replica track. It lets out a giant pneumatic wheeze, as though the tired station itself were sighing. Even the little toy train that he used to demonstrate signaling basics is falling apart; there was so much rust and dust on the tracks that at several points another MTA employee had to help it along with his hand.
In all there are more than 12,000 signals and 300,000 relays that comprise the system’s interlockings. Signal failure is the largest source of delays in the subway. There is an incident on average every 11 hours. Whenever a signal fails, the ones behind it go red, since they can’t be sure there isn’t a train in that section. Often, a can or a piece of aluminum foil is enough to fool a track circuit. “Something as simple as that can create a Conga line of trains,” Habersham says.
Habersham and the whole of the MTA are unexpectedly forthcoming about the sorry state of the signaling system. They even put out a video that gleefully shows off the worst stuff. Look at how broken it is! Look at how old!
Their point is not to flagellate themselves. It’s to drum up capital support for a systemwide upgrade and in particular for a program called CBTC. The MTA, too, thinks it’s ridiculous that all a tower operator knows when they look at their board is that some hunk of steel—which one, they can’t be sure—is sitting on some section of track. You want fewer delays? You want realtime countdown clocks?
CBTC is the answer.
* * *
For decades, New York had little interest in modern signaling technology. It was happy just to keep the trains running. The MTA was biased in favor of brick-and-mortar work. The attitude was that if you weren’t building a new station or track, it probably didn’t need to be done.
Then they got a wake-up call. Just after midnight on August 28, 1991, a 4 train running downtown with more than 200 passengers derailed at a switch just short of Union Square. Five people died. So many support columns were destroyed that the street above the station sunk by half an inch. The driver had been drinking heavily the day before the accident and was asleep as the train approached the station. They were traveling at 50 miles per hour, five times the normal speed.
A train stop on the tracks triggered the emergency brakes but it turned out to be too little too late. The signals had been spaced too closely; there wasn’t enough track to slow the train. It was a sobering moment. The foundations of the system—track circuits, signals, train stops—were cast into doubt.
CBTC does away with the “fixed-block” signaling system, where track is broken into sections that report whether they’re occupied. Instead, each train is equipped with a radio and onboard computer that identifies its precise location, and coordinates that information in real time with a central control center and other trains to decide exactly how fast it can safely go. Trains therefore run with a moving window around them, which constantly shifts depending on their own speed, size, track conditions, and traffic.
The big benefit is that this allows you to run trains closer together. And you can use the track more flexibly. For example, in many fixed-block signaling systems, including New York’s, the signals along certain sections are only set up for trains to go in one direction. But CBTC just sees a bunch of track; it can automatically figure out which parts of it are available, say, for turning trains around.
A diagram of a moving-block signaling system powered by CBTC. Photo from the Regional Plan Association.
In fact, with CBTC, you no longer even need signals or interlockings. If you were willing to go all-in, you could completely remove all your old levers, relays, track circuits, and signal towers. Safety would instead be guaranteed by the fact that the system as a whole knows about every train and can control its speed. If there were ever a communications failure, failsafe equipment on each train could instantly trigger the emergency brakes.
Maintenance costs would plummet. The bulk of the system’s complexity is moved into software: software that interfaces with the physical train controls; software that understands the entire system’s track layout, including all switches, grades, and stations; and software with a huge number of rules about what kind of movements are allowed when.
Since CBTC requires realtime location data to operate, relatively little additional work is required to give that data to customers, say in the form of a smartphone app that reports arrival times or in countdown clocks hung at stations.
The Canarsie Line running from 8th Avenue in Manhattan to the Canarsie neighborhood in southeast Brooklyn was chosen to pilot the technology. It was one of the oldest lines in New York, starting life as a steam railroad in the early 1900s, but seemed ideal as a test case: It was completely isolated from the rest of the system, with just one line, the L, running along it; it was shorter than most other lines; and it had relatively low rider volume.
Work began in 1999. It wouldn’t be fully operational until 2011. At the current pace of installation, the subway system as a whole won’t be converted to CBTC for another 175 years. It will cost $20 billion.
* * *
The Regional Plan Association has its offices at 4 Irving Place, in the Con Edison building. I went there one recent morning to talk to Richard Barone, the director of the RPA’s Transportation Programs and the author of their 2014 report on CBTC, “Moving Forward: Accelerating the Transition to Communications-Based Train Control for New York City’s Subways.” The RPA is a kind of urban-policy think-tank that focuses on the tri-state area. Its main outputs are tomes it calls Regional Plans, the first three of which were released in 1929, 1968, and 1996. The fourth is due to come out soon.
Barone and I walked through a room full of maps to a nondescript corner office on the 7th floor. On the bookshelf was a copy of The Power Broker. In graduate school, Barone told me, he studied labor relations.
I was there to find out why CBTC was taking so long and costing so much.
“You try to benchmark New York to other places and you can’t,” he said. Everything is harder here. Everything takes longer here.
He explained that the Canarsie pilot suffered from problems that weren’t unusual for big transit projects in New York.
The first was outmoded work rules. CBTC is designed so that trains can run themselves. But the L still has two-person crews on board every train. They’re not very busy: An April 2007 article titled “Look, Ma—no hands!” in the trade magazine Railway Age featured a delighted train supervisor named Lance Parrish riding in a CBTC-equipped train on the Canarsie Line. “All Parrish has to do is scan the onboard displays and acknowledge a flashing/beeping alerter every 20 seconds.”
Barone says New York just wasn’t willing to rip the band-aid off. Cities like London deal with major transit upgrades by packing maintenance and line closures into as short a window as possible, however painful that might seem at the time. New York, by contrast, draws out its track maintenance. When I spoke to the president of Thales Transport & Security, one of two major CBTC suppliers to New York, he said that “getting time on the track is by far the biggest schedule driver.” Crucial test-runs get queued behind miscellaneous track maintenance, so that it takes months to validate even small changes. “In the New York mindset,” he said, “there just isn’t the concept of the trains ever stopping.”
All that waiting isn’t free. These are huge projects for a company like Thales; they’ll spin up a whole office, a whole mini workforce, just to work on it. And when they’re waiting for track time, that workforce doesn’t just spin down—it continues to get paid. Anticipating delays, contractors inflate their bids.
The project took 11 years to become fully operational and cost $340 million. The next steps for CBTC are the Flushing line, which is already underway, and the Queens Boulevard line, where the contract was just awarded. It is shaping up to be a more capital-intensive and longer-running project than the first leg of the Second Avenue subway, and almost as big as Boston’s Big Dig.
But compared to those projects it is awfully nebulous. It’s hard to explain an underground-systems upgrade to customers; and it’s hard for customers to notice, especially if the city keeps maintaining the old signal systems and neutralizing its own cost savings.
That’s why the MTA has tried to associate CBTC with countdown clocks. New York riders crave realtime information about trains. They don’t care how they get it. So when Transit wants to drum up support for an obscure, costly, many-decades-long capital project to upgrade to CBTC, they always point to the clocks. (“Sustained Investment Makes Real-Time Information Possible,” declares one 2012 press release.) Reporters, struggling to make sense of a half-dozen interrelated projects, follow the MTA’s lead and assume that realtime train-location information depends on signal upgrades.
It’s a misleading narrative. You can get countdown clocks without touching the signals. The MTA knows this. They learned the hard way that tying countdown clocks to a massive CBTC-esque signal upgrade is a mistake. That’s what happened with the 1-6 lines: The project, called ATS, cost $220 million and took 5 years longer than it was supposed to. To deliver countdown clocks to the rest of the lines by 2017, they intentionally developed a simpler plan. The whole point of it was to avoid trackside signal upgrades.
If they’re being realistic, it’s that project, not CBTC, that the MTA should be talking about when it talks about clocks. The thing is, it’s kind of awkward to talk about. That project is to ATS as ATS was to CBTC: a lesser, overlapping, incompatible and possibly unnecessary stopgap. One of three redundant initiatives started at different times for different reasons, by an agency clumsily trying to rewire itself.
The story of how it could take 28 years to install clocks that tell you when the damn trains are coming turns out not to be about some dinosaur fixed-block signaling system and the gleaming new technology here to replace it. It’s simpler than that: It’s the story of a large organization’s first encounter with a large software project.
* * *
New York’s subways were initially built by two different companies, the IRT and the BMT, with different-sized cars and tunnels. Today there are still two systems that run more or less independently, in the sense that trains from one can’t run on tracks from another. These form the A division (the numbered lines) and B division (the lettered lines) respectively.
In the mid-90s, the MTA decided to make major upgrades to the older of these two systems, the A division, devising a $200 million plan to repair old signals, convert mechanical interlockings to use electronic relays, digitize the interlocking data, pipe that data into a central control center, use it to activate switches remotely and reroute trains, and distribute arrival information in real time to countdown clocks installed in every station on the 1-6. It would be a massive consolidation of what had been disparate systems. It was going to require serious technical expertise: not just to install the new gear, but to retrofit and interface with the existing interlockings. The project was called “ATS,” for Automatic Train Supervision.
If it sounds similar to CBTC, that’s because in all respects the goals are the same, except that ATS stops short of allowing remote control of trains. Both approaches consolidate location data that had been distributed and use it to build a picture of the system as a whole, allowing operators to put a name to each train and therefore to make better decisions about how to route them. But CBTC, when implemented as designed, is by far the simpler way to do this, since it obviates signal towers and interlockings altogether.
Given that CBTC would—and will, someday—accomplish the same thing as ATS; and given that it could be done much more efficiently; it makes you wonder how ATS got started in the first place. The answer is that planning for ATS began in 1992, two years before the MTA started seriously considering CBTC as an option. Had ATS been started two years later, it might have never been started at all.
A post-mortem by the Federal Highway Administration details how from the start, an agency which had had little experience with large “systems” projects tried to wing it. For instance, the consulting firm tasked with developing the project plan never made a list of requirements, didn’t talk to the workers who would be maintaining the system until after it was designed, and left vague instructions for large chunks of work—specifying, for instance, “similar functionality to what is currently available”—that later became the focus of drawn-out contract disputes.
The MTA thought that they could buy a software solution more or less off the shelf, when in fact the city’s vast signaling system demanded careful dissection and reams of custom code. But the two sides didn’t work together. The MTA thought the contractor should have the technical expertise to figure it out on their own. They didn’t. The contractor’s signal engineer gave their software developers a one-size-fits-all description of New York’s interlockings, and the software they wrote on the basis of that description—lacking, as it did, essential details about each interlocking—didn’t work.
The list goes on: Software prototypes were reviewed exclusively in PowerPoint, leading to interfaces that were hard to use. Instead of bringing on outside experts to oversee construction, the MTA tried to use its own people, who didn’t know how to work with the new equipment. Testing schedules kept falling apart, causing delays. The training documentation provided by the contractor was so vague as to be unusable.
You get the impression that the two groups simply didn’t respect each other. Instead of collaborating, they lobbed work over a wall. The hope on each side, one gathers, was that the other side would figure it out.
Miraculously, they finished. It took nearly twice as long as they thought it would. After everything, after more than a decade, they were left with a system that isn’t even compatible with CBTC. And they were still less than halfway done.
* * *
On the B division, all train activity is still written down on paper and sent in batches to a central location for processing.
But once the public had countdown clocks on some lines, they couldn’t understand why they didn’t have them on the others. “An interesting dynamic developed in response to the ATS-A system’s ability to make predictive train arrival announcements,” writes the MTA’s Thomas Calandrella in a report. “A customer-driven campaign, filtered through city and state governmental representatives, demanded that the train-arrival signs on the 1 through 6 lines be made available to all NYCT customers. Based upon this pressure, the agency committed to providing predictive train-arrival information to the customers on the platform and on the web by 2017.”
They hope to accomplish this with a new project called ISIM-B. It stands for “Integrated Service Information and Management.” It is meant to be like the better parts of ATS reborn, this time leaner, simpler—not so much an omnibus system as a collection of smaller ones.
Recall that ATS was just for the A division. (That’s why it’s sometimes called “ATS-A.”) Separately, CBTC—which is like ATS+, since it includes the ability to remotely control trains—was started on a few individual lines. ISIM-B was conceived to finish where ATS had left off.
Instead of rebuilding trackside signals, most of the upgrades on ISIM-B will take place in signal towers, meaning trains won’t have to be taken out of service. All that has to happen is for the so-called “indications” from the track circuits—the data about which hunk of steel is on which section of track—to be digitized and centralized. Only when the indications from all the towers are combined can you be sure that that hunk of steel is that F train. (Once you have a view of the whole system, instead of just a part of it, you can trace a single hunk as it moves from section to section—giving it a persistent name.)
The focus is on delivering incremental value, rather than doing any kind of total overhaul. On one view it represents a retrenchment after the drawn-out battle that was ATS. On another, it’s what ATS was meant to be in the first place. Either way it sounds like an improvement, except that according to the last two MTA capital plans, the cost for ISIM-B is projected to be in the hundreds of millions of dollars and counting. Kicked off in 2011, it was expected to be finished by 2017, but already the date is slipping: The Wall Street Journal reported a few weeks ago that funding for the project had been cut from the latest capital plan, pushing the clocks’ arrival to at least 2020.
It is growing to look more and more like ATS. A request for proposal as recent as six months ago—back when funding looked more secure—called for a 77-month software contract to build out a sophisticated Rail Traffic Management System as part of ISIM-B. That piece of the project is envisioned as a complex centralized “expert system” that would allow operators to quickly diagnose service problems and would intelligently suggest ways to work around the disruption. It is, in a word, ambitious. And ambition is the death knell for big software projects. It’s what made ATS such a quagmire in the first place. It is, one suspects, why funding for countdown clocks has been cut from the latest capital plan: The rest of ISIM-B costs too much. It costs too much because it is trying to do too much. The consequence being that for five or six years, customers will hardly see anything get done at all.
The strange thing is that the MTA has it in them to do things differently. Really differently.
* * *
In the late ’90s, just as ATS and CBTC were promising to rebuild the informational guts of the subway, a project was cooking aboveground to bring arrival-time information to buses.
The earliest attempts were in the usual style: A single big contractor was given a fixed-fee award to do the whole thing. In 1996 it was Orbital Sciences Corp. who got the contract; they were fired four years later after making little progress. In 2005 Siemens won the bid for a $13 million pilot program to put clocks in the stations on six bus routes; that project was canceled a year later. (Siemens was the same company who worked on ATS. A 2009 report by the MTA's oversight organization shows that despite the contractor’s poor performance on that project, ATS program managers had been told by their supervisor not to give Siemens an overall “Unsatisfactory” rating, for fear that this would preclude it from participating in future projects.)
In the spring of 2010, under the leadership of chairman Jay Walder, who was known both for straining labor relations and for having an entrepreneurial streak, the MTA went in a different direction. They hired a small team of software-savvy MIT grads to come in-house and manage the bus project. Instead of procuring a single contractor, they defined the specifications for the project themselves, broke it into pieces, and brought contractors on to build each one.
Their big idea was actually to do less. Walder felt that the MTA’s credibility was at stake. He knew riders wanted realtime arrival information. He worried that failing to deliver something tangible in the first half the new administration might threaten the next capital plan. So he decided that BusTime, as the project became known, would deliver the minimum viable product. Where previous efforts had tried to install countdown clocks at every stop, this time around they would make a website and app that delivered location data directly to riders. It wouldn’t be ideal—lots of riders, particularly lower-income New Yorkers and the elderly, didn’t have ready access to the web—but it eliminated a huge part of the project’s scope.
Having full-time software experts running the show turned out to be crucial. Previous incarnations of the project didn’t have a technical leader at the MTA—just old-school senior managers who would try to wrangle the contractors by force of will. The new in-house team, by contrast, was qualified to define exactly what they wanted from software providers in terms those providers could understand. They were qualified to evaluate progress. They could sniff out problems early.
A working version was available on a route in Brooklyn by February 2011, just months after the project began. It came to Staten Island by January 2012, and expanded to all of Manhattan by late 2013. All-in it cost on the order of $10,000 per bus.
Walder now works in the private sector. BusTime turned out not to be the new normal for the MTA—just the exception that proves the rule. In August of this year, for instance, the MTA began the next phase of its CBTC rollout with a $156 million contract. It was awarded to Siemens.
* * *
Barone—the author of that CBTC report—and I are sitting at a conference table out in the RPA common area. (We’d been squatting in the corner office.) We’ve been talking for about an hour. All the “ATSes” and “CBTCs” and “ISIM-Bs” are starting to take their toll.
“One of the things that was most frustrating when doing this work,” Barone says to me, referring to preparing the report, “was the murkiness. And the lack of uniformity in how each of these systems is being done.”
“It seems to me that there are concurrent projects going on that—” He trails off, thinking for a minute. “It’s like, you’re building ISIM to find out where the trains are located—but CBTC does that. You’re spending money to get your interlockings to be centrally automated, yet CBTC can do that too… ATS initially came out before they really thought about moving to CBTC, and therefore the first ATS is not even compatible with it. It can’t plug in. There’s a whole plan now to do a new version of it…”
He seemed weary. I certainly was. I told him I honestly just wanted to know why the F train didn’t have clocks. I never expected it would be so complicated.
The MTA has a thankless and extremely difficult job: They have to keep the trains running. They have to do it with equipment from the 1930s, in a hostile funding environment, as administrations come and go, as public interest comes and goes, in the face of storms and accidents and pieces of aluminum foil. This they manage to do. 1.6 billion people every year take the New York subway. The system carries more than 60 percent of all people coming into Manhattan every day. It is, for the most part, safe, affordable, and there.
But still, a reasonable person looking at three projects that aim to do roughly the same thing, projects that have different teams and different agendas and seem to take, always, five years longer than planned and seem to cost, always, hundreds of millions of dollars—well, this person has to wonder whether there’s some law of the universe that makes large government software projects an expensive drag or whether in fact there’s a better way.
I keep thinking of Healthcare.gov. Everyone knows that the initial project was a costly disaster, but less well known is that a small team came along and saved it. The story includes this remarkable fact: The old system cost $250 million to build and $70 million a year to maintain. The new system—which actually worked—cost about $4 million to build; its yearly maintenance was about $1 million.
If the initial system hadn’t been such a public failure, we might not have learned that it was two orders of magnitude more expensive than it needed to be. How many more projects are being run at 60 times their actual cost?
Should we stand around waiting to find out?