Why a predictable cold snap crippled the Texas power grid

As Texans cranked up their heaters early Monday to combat plunging temperatures, a record surge of electricity demand set off a disastrous chain reaction in the state's power grid.

Wind turbines in the state's northern Panhandle locked up. Natural gas plants shut down when frozen pipes and components shut off fuel flow. A South Texas nuclear reactor went dark after a five-foot section of uninsulated pipe seized up. Power outages quickly spread statewide - leaving millions shivering in their homes for days, with deadly consequences.

It could have been far worse: Before dawn on Monday, the state's grid operator was "seconds and minutes" away from an uncontrolled blackout for its 26 million customers, its CEO has said. Such a collapse occurs when operators lose the ability to manage the crisis through rolling blackouts; in such cases, it can take weeks or months to fully restore power to customers.

Monday was one of the state's coldest days in more than a century - but the unprecedented power crisis was hardly unpredictable after Texas had experienced a similar, though less severe, disruption during a 2011 cold snap. Still, Texas power producers failed to adequately winter-proof their systems. And the state's grid operator underestimated its need for reserve power capacity before the crisis, then moved too slowly to tell utilities to institute rolling blackouts to protect against a grid meltdown, energy analysts, traders and economists said.

Early signs of trouble came long before the forced outages. Two days earlier, for example, the grid suddenly lost 539 megawatts (MW) of power, or enough electricity for nearly 108,000 homes, according to operational messages disclosed by the state's primary grid operator, the Electric Reliability Council of Texas (ERCOT).

The crisis stemmed from a unique confluence of weaknesses in the state's power system.

Texas is the only state in the continental United States with an independent and isolated grid. That allows the state to avoid federal regulation - but also severely limits its ability to draw emergency power from other grids. ERCOT also operates the only major US grid that does not have a capacity market - a system that provides payments to operators to be on standby to supply power during severe weather events.

After more than 3 million ERCOT customers lost power in a February 2011 freeze, federal regulators recommended that ERCOT prepare for winter with the same urgency as it does the peak summer season. They also said that, while ERCOT's reserve power capacity looked good on paper, it did not take into account that many generation units could get knocked offline by freezing weather.

"There were prior severe cold weather events in the Southwest in 1983, 1989, 2003, 2006, 2008, and 2010," Federal Energy Regulatory Commission and North American Electric Reliability Corp staff summarized after investigating the state's 2011 rolling blackouts. "Extensive generator failures overwhelmed ERCOT's reserves, which eventually dropped below the level of safe operation."

ERCOT spokeswoman Leslie Sopko did not comment in detail about the causes of the power crisis but said the grid's leadership plans to re-evaluate the assumptions that go into its forecasts.

The freeze was easy to see coming, said Jay Apt, co-director of the Carnegie Mellon Electricity Industry Center.

"When I read that this was a black-swan event, I just have to wonder whether the folks who are saying that have been in this business long enough that they forgot everything, or just came into it," Apt said. "People need to recognize that this sort of weather is pretty common."

This week's cold snap left 4.5 million ERCOT customers without power. More than 14.5 million Texans endured a related water-supply crisis as pipes froze and burst. About 65,000 customers remained without power as of Saturday afternoon, even as temperatures started to rise, according to website PowerOutage.US.

State health officials have linked more than two dozen deaths to the power crisis. Some died from hypothermia or possible carbon monoxide poisoning caused by portable generators running in basements and garages without enough ventilation. Officials say they suspect the death count will rise as more bodies are discovered.

Thin Power Reserve

In the central Texas city of Austin, the state capital, the minimum February temperature usually falls between 42 and 48 degrees Fahrenheit (5 to 9 degrees Celsius). This past week, temperatures fell as low as 6 degrees Fahrenheit (-14 degrees Celsius).

In November, ERCOT assured that the grid was prepared to handle such a dire scenario.

"We studied a range of potential risks under both normal and extreme conditions, and believe there is sufficient generation to adequately serve our customers," said ERCOT's manager of resource adequacy, Pete Warnken, in a report that month.

Warnken could not be reached for comment on Saturday.

Under normal winter conditions, ERCOT forecast it would have about 16,200 MW of power reserves. But under extreme conditions, it predicted a reserve cushion of only about 1,350 MW. That assumed only 23,500 MW of generation outages. During the peak of this week's crisis, more than 30,000 MW was forced off the grid.

Other US grid operators maintain a capacity market to supply extra power in extreme conditions - paying operators on an ongoing basis, whether they produce power or not. Capacity market auctions determine, three years in advance, the price that power generators receive in exchange for being on emergency standby.

Instead, ERCOT relies on a wholesale electricity market, where free market pricing provides incentives for generators to provide daily power and to make investments to ensure reliability in peak periods, according to economists. The system relied on the theory that power plants should make high profits when energy demand and prices soar - providing them ample money to make investments in, for example, winterization. The Texas legislature restructured the state's electric market in 1999.

Looming Crisis

Since 2010, ERCOT's reserve margin - the buffer between generation capacity versus forecasted demand - has dropped to about 10% from about 20%. This has put pressure on generators during demand spikes, making the grid less flexible, according to North American Electric Reliability Corporation (NERC), a nonprofit regulator.

That thin margin for error set off alarms early Monday morning among energy traders and analysts as they watched a sudden drop in the electrical frequency of the Texas grid. One analyst compared it to watching the pulse of a hospital patient drop to life-threatening levels.

Too much of a drop is catastrophic because it would trigger automatic relay switches to disconnect power sources from the grid, setting off uncontrolled blackouts statewide. Dan Jones, an energy analyst at Monterey LLC, watched from his home office in Delaware as the grid's frequency dropped quickly toward the point that would trigger the automatic shutdowns.

"If you're not in control, and you are letting the equipment do it, that's just chaos," Jones said.

By Sunday afternoon about 3:15 pm (CST), ERCOT's control room signaled it had run out of options to boost electric generation to match the soaring demand. Operators issued a warning that there was "no market solution" for the projected shortage, according to control room messages published by ERCOT on its website.

Adam Sinn, president of Houston-based energy trading firm Aspire Commodities, said ERCOT waited far too long to start telling utilities to cut customers' power to guard against a grid meltdown. The problems, he said, were readily apparent several days before Monday.

"ERCOT was letting the system get weaker and weaker and weaker," Sinn said in an interview. "I was thinking: Holy shit, what is this grid operator doing? He has to cut load."

Sinn said he started texting his friends on Sunday night, warning them to expect widespread outages.

'Seconds And Minutes'

Early Monday morning, one of the largest sources of electricity in the state - the unit 1 reactor at the South Texas Nuclear Generating Station - stopped producing power after the small section of pipe froze in temperatures that averaged 17 degrees Fahrenheit (9 degrees Celsius). The grid lost access to 1,350 MW of nuclear power - enough to power about 270,000 homes - after automatic sensors detected the frozen pipe and protectively shut down the reactor, said Victor Dricks, a spokesman for the US Nuclear Regulatory Commission.

About 2:30 am (CST), the South Plains Electric Cooperative in Lubbock said it received a phone call from ERCOT to cut power to its customers. Inside the ERCOT control room, staff members scrambled to call utilities and cooperatives statewide to tell them to do the same, according to operational messages disclosed by the grid operator.

Three days later, ERCOT Chief Executive Bill Magness acknowledged that the grid operator had only narrowly avoided the calamity of uncontrolled blackouts.

"If we hadn't taken action," he said on Thursday, "it was seconds and minutes (away), given the amount of generation that was coming off the system at the same time that the demand was still going up."