F. Russell Robertson, W. Terry Boston
The 14 august 2003 blackout in the United States and Canada was big, but it wasn’t the first big one. On 9 November 1965, at 5:16 p.m., power failed throughout New York state, spreading to seven nearby states and parts of eastern Canada. Up until the blackout of August 2003, the November 1965 blackout was the biggest power failure in U.S. history, leaving 30 million people and virtually all of New York, Connecticut, Massachusetts, and Rhode Island without power for up to 13 h. The 1965 blackout led to many improvements in the resiliency and reliability of the grid, including the formation of the North American Electric Reliability Corporation (NERC) in 1968, the creation of the Electric Power Research Institute in 1972, and perhaps most significantly, the operational robustness gained through the planning and buildout of the 500-kV interconnected electric grid. The 1965 blackout was also the motivation for enhancing the existing grid control systems to include underfrequency relays, automated generation control, as well as supplemental applications to perform network analysis, such as state estimation and training simulators for operators.
Two major western interconnection outages in the summer of 1996 reinforced the findings from the 1965 blackout with misoperations of relays on adjacent lines being the root cause of both 1996 events. The Bonneville Power Administration (BPA) had three demonstration phasor measurement units (PMUs) in service at the time. Data collected by those units provided engineers with a few key insights about the nature of the events, so much so that these results motivated the BPA to expand its synchrophasor measurement network.
Complicating the coordinated operation of the grid was the separation of the electric utility industry by function, which began with Federal Energy Regulatory Commission Order no. 888 in 1996, setting the stage for nonutility-owned generation to enter electricity markets. What had been a monolithic, vertically integrated industry now included thousands of electricity market participants. This restructuring of the power industry led the NERC to develop the reliability functional model that defines the functional entities and the monitorable seams among these entities so that operational requirements could be effectively developed for the multiple entities. By 2000, regional transmission organizations (RTOs) and independent system operators (ISOs) had emerged to take on responsibility for operating, balancing, and interchange reliability, including the responsibility (among others) to monitor the grid in real time and act if necessary to direct changes in generation, transmission, or load (including the interruption of firm load) to move the grid to a more reliable state to avoid blackouts.
The California energy crisis of 2000–2001 showcased the complexity of a deregulated electric industry and the need for polishing the federal and state policies that deregulated it. Electricity market manipulation during the California crisis resulted in large, adverse societal impacts.
The operational response on 14 August (or lack thereof) by First Energy Corporation, the transmission owner/operator, and by Midcontinent Independent System Operator, the reliability coordinator, clearly demonstrated the need for more information on the real-time state of the grid as well as on the status of the computer systems used to monitor and control it.
Following the blackout, time-synchronized forensics data were hard to find and difficult to wrangle. It took an amazingly long three months to reconstruct the sequence of events that caused the 14 August blackout. Later analysis showed that time synchronization of data was off by several minutes from one control area to another.
In April 2004, the United States–Canada Power System Outage Task Force issued a report on the August 2003 blackout that was followed in July 2004 by an NERC report analyzing “What happened, why, and what did we learn?”. The NERC report contained 14 recommendations that have been summarized as the “Trees, Tools, and Training” recommendations. Many of these recommendations related to the strengthening of NERC’s compliance programs. Five of these recommendations were more directly related to operations:
Following the release of these reports, the electric industry quickly responded, and from a tools perspective the priority changed from state estimation and real-time contingency analysis (RTCA) being an optional backstop with availability tracked by local metrics to one of these systems being a critically important component to maintaining situational awareness to enable operators to successfully respond to the unexpected.
Like many utilities and grid operators, the Tennessee Valley Authority (TVA) acted in response to the 14 August 2003 blackout before the recommendations for improvement were released by the U.S. Department of Energy (DOE) and the Federal Energy Regulatory Commission, as well as NERC investigators. Within a couple of weeks, the TVA launched a major initiative to improve the computer and control systems supporting grid operators, not only to the RTA tool suite (SCADA/EMS, state estimator, RTCA, and power flow study systems), but for all computer systems that support real-time operations. The TVA represents a good case study for utility response to the August 2003 blackout since TVA’s responsibilities include those typically associated with an RTO as well as being a transmission owner/operator, along with other NERC functional model responsibilities. The TVA sent subject matter experts to both the New York ISO and NERC to assist in the lessons learned from the 2003 event.
The TVA’s response showcases the abrupt change in priorities that resulted from the series of events on 14 August 2003. Changes and improvements were made in weeks, which might have taken months or years without the impetus provided by the hard lesson learned in the Midwest and Northeast blackouts.
The operations organization at the TVA made an open request for ideas to improve the quality and availability of situational awareness tools for operations staff. It resulted in a list of 268 ideas to improve the TVA’s operational technology tools. There were 190 quick hitters on this list, and by 1 October 2003, all 190 of these suggested items had been implemented. Many of these quick hitters made it relatively easy to implement human factor-based adjustments to SCADA/EMS displays. For example, changes were made for new colorization or highlighting or for reorganization of information on SCADA/EMS displays to improve situational observability and operator awareness so that key information would not be overlooked.
Additional operational technology action items implemented at the TVA included:
Phasor measurement was a research activity initiated by only a few individual utilities in the eastern interconnection (EI) prior to the blackout of 2003, notably by the TVA (see “Phasor Measurement and the TVA”), New York Power Authority, Ameren, and Ontario Hydro. The recommendations of the blackout reports for a layered collection of better “tools” quickly raised the interest in synchrophasors.
Phasor Measurement and the TVA
A “phasor” is a mathematical representation of a sinusoidal waveform that has a magnitude and an angle. The magnitude is the root-mean-square value of the sinusoid waveform. The angle is in degrees or radians, representing the shift of the waveform with respect to a given reference waveform. Electrical synchrophasors are measurements of power system sinusoidal quantities, synchronized in time and referenced to the nominal power system frequency, and expressed as phasor values, i.e., a magnitude and an angle.
Synchrophasor measurements were first introduced in an article published in 1983 by Arun Phadke, Jim Thorp, and Mark Adamiak. Using lessons learned from pioneering vendors of PMUs, the BPA created the first real-time phasor data concentrator (PDC) in 1996. Based on the differences in communications times from individual PMUs that may be scattered over a wide area, a PDC is needed to align, or concentrate, the input phasor values into a set of measurements that were all made at the same time so that they can be used collectively.
The big boost in the usefulness of synchrophasor measurements came from President Clinton’s order in 2000 May to not intentionally precision-limit GPS time signals. This order enabled public users of GPS clocks to access time to ±3-ns precision. With this additional precision, electrical phasor measurements could be precisely time-synchronized, making these measurements immensely more valuable. Following the August 2003 blackout, the DOE launched the EI Phasor Project (EIPP) to promote the use of synchrophasor measurements and breakdown barriers to their deployment. As a major participant in the EIPP, in 2004 the TVA announced that it had developed both the PMU connection tester and the TVA-phasor data concentrator.
The PMU connection tester shown in Figure S1 significantly simplified the process of testing and turning up new PMUs in the field. It supported all the major streaming protocols for PMU data and could be used to validate the quantities received from a PMU to assure that it had been properly configured. The TVA PMU connection tester was used as the “gold standard” for proper implementation of the newly released IEEE C37.118 method for sending synchrophasor measurements back to the control center. Its availability accelerated the successful implementation of the communication protocols defined in the IEEE C37.118 Standard for substation devices. This connection tester is now an open-source project, and it remains an actively used tool internationally.
The TVA–PDC was the first software-based solution for time-aligning and saving PMU data sent to the control center in a variety of formats. Its flexibly managed metadata about the synchrophasor measurements, created multiple real-time output streams at low latency so that aggregate or filtered PMU data could be routed to tools for real time analysis or display, and enabled the development of “adapters” that could be plugged directly into the PDC platform so that the results of analytics (i.e., calculated values) could be injected into the real-time stream at PMU data sampling rates. Soon thereafter, the TVA–PDC became generally known as the SuperPDC since it had the architecture and performance necessary to aggregate synchrophasor data in real time from multiple wide-area PDCs.
Using the SuperPDC, the TVA volunteered to host the warehouse for synchrophasor data in the EI. The need for this centralized data warehouse was important at the time to improve the quality of synchrophasor measurement systems and to enable development of new operator and engineering tools from it. Having this warehouse in a single location overcame the two major barriers to the advancement of synchrophasors—those of individual utility IT infrastructure and cost of the IT infrastructure—at the time many utility operating organizations were SCADA-centric and did not have the capability to easily host other information systems. As for cost, each PMU with a typical 16 channels of data produces about 400 GB per year of data; in 2004 the first cost for nonenterprise, site-level storge was US${\$}$20–US${\$}$30 per GB. The TVA-hosted system grew quickly, and as shown in Figure S2, by early 2006 there were 30 PMUs streaming data to the SuperPDC.
The synchrophasor communication standard, IEEE C37.118, was completed in 2005. It introduced the total vector error (TVE) concept for evaluating measurements and established that PMUs must produce measurements with not more than 1% TVE (or about 0.57°); it specified steady-state performance requirements for PMUs and extended the data communication profile for synchrophasors using concepts from IEEE 1344 and BPA’s PDCstream protocols, such as the ability to combine data received from multiple devices into a single larger frame of data. The IEEE C37.118 protocol filled what at the time was a major gap in standard utility deployment of synchrophasor data systems: the need for an efficient protocol to support the reliable communication of high-volume (relative to other substation data flows) synchrophasor data from the measurement device to the control center.
In 2008, the NERC stepped in with funding to support the growth and operation of the TVA SuperPDC for the EI, as shown in Figure S3. With this funding, the TVA installed a 76-TB Hadoop cluster—a full rack with 228 TB of physical storage—on which to archive and enable high-performance filtering of PMU data.
In 2009, the IEEE and the International Electrotechnical Commission (IEC) formed a joint task force that worked on methodologies and agreements that led to the 2011 release of the new two-part IEEE C37.118 Standard and the creation of IEC TR 61850-90-5. Also in 2009, in concert with the announcement that smart grid investment grants were a part of the American Reinvestment and Recovery Act, the TVA open-sourced its SuperPDC software to enable the software to become part of new or expanded synchrophasor data systems implemented by the RTOs and transmission owner/operators that were awarded grants. Now called the openPDC, this software continues to be actively maintained and supported and is in production use at many utilities.
The smart grid investment grants enabled the number of PMUs in North America to grow from a few hundred in 2009 to more than 1,700 by 2015 as the EIPP evolved to become the North American SynchroPhasor Initiative (NASPI). As part of the smart grid investment grants, the New York ISO was the first reliability coordinator to move real-time synchrophaser data from an engineering tool to the control room to improve real-time situational awareness (i.e., for blackout prevention, not just for blackout investigation). By 2015, Midcontinent Independent System Operator estimated that synchrophasor applications would reduce August 2003-level blackouts from one in 20 years to one in 30 years.
Today, there are well over 3,000 PMUs in service in the United States providing real-time data to operators and supporting the improvement of network modeling and state-estimation systems. This network of PMUs has enabled the development of new technology. For example, the availability of precisely synchronized power system measurements provided by PMUs has made it possible to more easily deploy remedial action schemes, such as dropping load or generation or adjusting value-added reseller compensation based on an assessment of these measurements over a wide area. This wide area often spans the responsibility of multiple local grid operators, providing protection for grid instability that requires the big-picture view to take proper, prompt actions to prevent cascading failures.
The Value of Synchropahsors
When introduced, it was said that the transition from traditional SCADA to synchrophasors would be as significant for electric grid diagnostics as the introduction of magnetic resonance imaging was over X-rays for making medical diagnoses. The availability of synchrophasor data improves grid reliability by reducing the number and duration of outages, as well as the number of customers affected by outages. Following an outage, restoration of service is quicker. Outages are also reduced by using synchrophasor data to identify potential equipment failures, fixing them before the failure occurs.
More specifically, the real-time benefits of synchrophasors are:
• Wide-area measurement and situational awareness: High-speed, real-time synchrophasor data enable better monitoring, data-trending, and visualization, which in turn allow grid operators to identify problematic situations and quickly develop good responses to them.
• Oscillation detection: An unstable oscillatory mode on a transmission network can lead to large amplitude oscillations and blackouts. Analytical tools using synchrophasor data enable operators to identify active oscillatory modes and predict whether these oscillations can be damped safely.
• Phase angle and voltage stability monitoring: Phase angle monitoring can be used to monitor and improve the speed and accuracy of line reclosing and generator synchronization. Voltage stability monitoring provides power system operators with valuable online information to assess the power margin, the amount of additional active power that can be tolerated on a transmission line without jeopardizing voltage stability.
• Event detection: PMUs can detect and record events that a conventional SCADA/EMS system would have missed, thereby enabling better visibility into grid conditions and the performance of specific assets.
• Islanding detection and system restoration: During Hurricane Gustav, which made landfall on Labor Day in 2008, the local utility Entergy discovered through analysis of synchrophasor data that there was an electrical island operating in southern Louisiana, separate from the rest of the EI.
For off-line studies, the following benefits of synchrophasors are realized:
• Model validation and improvement: Synchrophasor data can be used for model validation, producing notably better models of generators and grid assets at lower analytical cost with rapid update capability. Equipment failure prediction: Synchrophasors can improve grid reliability by enabling diagnosis of many impending equipment failures before an actual failure.
• Forensic analysis: Following a grid disturbance, synchrophasor data can be used for forensic event analysis to find the cause of the disturbance, simulate the event, and determine potential future remedial actions. Evidence of this value was showcased following the 2011 Southwest blackout. Investigators were able to finish analyzing operational data within five hours. Within eight hours, the investigation team had reconstructed the sequence of events, a big improvement over the three months required for the August 2003 blackout.
At the TVA, the position changed from being an advocate for synchrophasor data research to one of assuming a leadership role in the EI to accelerate the installation of PMUs and the adoption of synchrophasor technology.
For example, at the time of the August 2003 blackout, the TVA was the frequency monitor for the EI. This monitoring was improved in 2004 with a system for operators that was based on PMU frequency measurements at multiple points on the TVA system. As shown in Figure 2, the system provided a trend line of frequency history with greater resolution than the SCADA system, and it provided the rate of frequency increase or decrease since the last change in sign in the frequency slope, something that the SCADA system did not do. This monitor refreshed four times per second and alarmed when there was a 15-min deviation in frequency of greater than .05 Hz from nominal.
For example, in early 2006. the TVA used PMU data to implement an oscillation alarm in its SCADA system for one of its large fossil plants, as shown in Figure 3. A service reported the mean amplitude of oscillations as based on synchrophasor measurements to the SCADA system over each 2- to -4-s SCADA interval. An alarm based on this value enabled operators to study and take corrective action to reduce the oscillations prior to protective systems tripping the plant.
The National Academy of Engineering declared that the grid and electrification was the greatest engineering achievement of the 20th century, noting that nothing has improved productivity and the standard of living more than electricity. The North American electric grid is the largest synchronous machine in the world.
Changes to this “big machine” will be necessary based on new regulatory laws and policies. Recent federal and state legislation aspires to rapidly transform the nature of generation and load. At the federal level, the Infrastructure Bill signed into law in November 2021 included more than US$65 billion for grid infrastructure, and the Inflation Reduction Act signed in August 2022 is designed to accelerate the manufacture and sale of electric vehicles and batteries, among other clean energy initiatives, including incentives for new dispersed wind and solar generation. States, most notably California, are also planning policies that intend to radically accelerate the adoption of electric vehicle technology and renewables.
Changes to the “big machine” will also be necessary based on new regulations intended to prevent or mitigate major power system events, e.g., fires and hurricanes. The societal impact of blackouts was most recently evidenced in February 2021 during the power crisis in Texas, where more than 2 million households were without power for days, leading to at least 210 deaths and, by some estimates, economic losses as high as US$130 billion. The Texas blackout also highlighted the interdependencies of infrastructures where power outages created shortages of both potable water and food.
Twenty years after the “big one” in August 2003, significant work remains to research, demonstrate, and implement new methods and technologies that can allow grid operators to continue to keep the lights on as the nature of electric loads change and as old power plants are shut down and are replaced by variable, inverter-based generation.
Some possible examples are:
Clearly, our work on the power grid and blackout prevention is not completed. Life as we know it is dependent on a robust, reliable, and resilient grid. Power engineering is not rocket science: it is much more important than that.
“Advancement of synchrophasor technology in projects funded by the American recovery and reinvestment act of 2009,” U.S. Department of Energy, Washington, DC, USA, Mar. 2016. [Online] . Available: https://www.energy.gov/oe/articles/advancement-synchrophasor-technology-projects-funded-american-recovery-and-reinvestment
“The value proposition for synchrophasor technology,” North American Synchrophasor Initiative, 2015. [Online] . Available: https://www.naspi.org/
K. Martin and J. R. Carroll, “Phasing in the technology,” IEEE Power Energy Mag., vol. 6, no. 5, pp. 24–33, Sep./Oct. 2008, doi: 10.1109/MPE.2008.927474.
J. R. Carroll and F. R. Robertson, “A comparison of phasor communication protocols,” U.S. Department of Energy, Washington, DC, USA, PNNL-28499, Feb. 2019.
Grid Protection Alliance. “Open phasor data concentrator (openPDC) overview.” Github. [Online] . Available: https://github.com/GridProtectionAlliance/openPDC
F. Russell Robertson is with the Grid Protection Alliance, Chattanooga, TN 37402 USA.
W. Terry Boston is the former CEO of PJM and resides in Signal Mountain, TN 37377 USA.
Digital Object Identifier 10.1109/MPE.2023.3247055
Date of current version: 19 April 2023
1540-7977/23©2023IEEE