-
Alt Text for Image
A consulting firm has identified a number of gaps and opportunities for Pedernales Electric Cooperative (PEC) to improve its response and resiliency in the wake of February’s Winter Storm Uri, which knocked out power to more than 109,000 customers during sub-freezing temperatures.
PEC’s Board of Directors engaged the services of Utilicast, a Kirkland, Washington-based consulting firm with an office in Austin, to perform an after-action review. Utilicast was charged with identifying potential gaps and opportunities in PEC’s operations during the winter event. The firm recently shared its findings with PEC board members and executives.
“The board felt it was necessary to have an objective group come in and take a hard look at what worked during the winter storm event as well as areas for improvement,” said Board President Emily Pataki, the District 2 director. “Utilicast spent several weeks inter viewing staff who were actively involved in responding to the storm. We know staff responded extremely well under the circumstances but felt it necessary to get another set of eyes on PEC’s response.”
Three severe winter storms swept across the United States between Feb. 10-20, which caused blackots for more than 9.9 million people in the United States and Mexico, with the biggest impact coming from Winter Storm Uri in Texas, where “massive electricity generation failures occurred and emergency operations mandated load curtailment across the Electric Reliability Council of Texas (ERCOT) Region,” Utilicast reported.
This led to only the fourth time in its history that ERCOT would initate system-wide rotating outages, following incidents on Dec. 22, 1989; April 17, 2006; and Feb. 2, 2011.
“While some outages on PEC’s system were due to a state-wide mandate by ERCOT to manage power load, others were a result of extreme temperatures that led to infrastructure damage across PEC’s 8,100 square mile service territory,” the report stated. “Hazardous roads made it difficult for PEC crews to restore power loss caused by infrastructure damage.”
Utilicast concluded the storm event allowed for opportunities to strengthen the cooperative’s ability to maintain resiliency, which occurs when systems, standards, and procedures have been built and tested to withstand extreme conditions. Among the main findings by Utilicast were:
• Communication to members and employees was reactive in nature, driven by a lack of preparation for the subevents that when culminated together, made the Winter Storm Event an unprecedented event. Recommendations related to people include strategies to bolster staff, improve emergency communications, and proactively connect with members.
• Reliance on manual processes and operations experience stretched personnel to their limits and exposed gaps in documented procedures and communication protocols. Process recommendations address methods to automate manual operations or formalize manual procedures into standard and emergency operations procedures.
• During emergency operations, the flow of information was compromised when several system issues deterred personnel from normal communication and collaboration. Technology recommendations require assessing, upgrading, re-configuring, testing, tuning and potentially replacing core modules of operational systems.
Utilicast praised the work of PEC personnel in responding to the storm and putting their own personal needs aside to help cooperative members.
“Despite key issues discussed in the detailed report, PEC personnel and leadership continue to serve the PEC membership with a focus on constant improvement,” the report said. “During the Winter Storm Event, support staff and restoration crews worked tirelessly to meet the demands of the membership and to restore the PEC service territory to a steady state of operations, many times at the expense of their own personal comforts.”
Utilicast reported a total of 22 recommendations, of which 11 directly modify PEC Operations across the field and control center; nine address technology improvements to assess and establish a baseline of technology architecture and improve the performance of operational technology.; while two extend communications procedures.
“If all the recommendations were implemented in sequence, PEC would need approximately 165- months to complete. Thus, many recommendations must be implemented in parallel to achieve benefits in a timely fashion,” the reported noted.
Operations
Utilicast said during the storm, Operations was responsible for two critical tasks: meeting load shed obligations and restoring system power.
“These two core responsibilities, by definition, have competing interests. The Incident Analysis uncovered system and process challenges at PEC that hamstrung power restoration efforts and complicated the load management process,” the report stated, recommending streamlining and improving communication and coordination between Field Operations and Control Center Operations.
The report found continuous load shedding masked real outages and impaired PEC’s system-restoration efforts, resulting in power outages to a large amount of the membership.
Recommendations include configuring the Outage Management System (OMS) to improve operations ability to identify customer outages during a load shed event; designating a control center as Command Central; rotating field engineers with dispatchers to improve outage dispatch response times; improving situational awareness tools; and extending dispatch coordination to districts during an emergency that lasts longer than 24 hours.
Load management applications and processes typically designed for peak shaving under short-term daily peaks, or used intermittently over multiple days of storm recovery, were receiving continuously changing, and substantially large, load-shed obligations for 70.5 hours. This caused allocation of rolling outages to only feeders without critical load, the report noted.
Suggestions for improvement include re-prioritizing the critical load registry, validating the load shedding program configuration, working to customize the AMI load control application and tuning outage predictions and the confirmation process.
Technology
When it comes to technology, the infrastructure, resources and applications which support Operations Technology compete with corporate and enterprise solutions, which masks weaknesses that exist within the systems. Operational Technology is “hardware and software that detects or causes a change, through the direct monitoring and/or control of industrial equipment, assets, processes, and events.”
Utilicast recommended completing an assessment of the system supporting outage map; augmenting IT personnel and service agreements and reviewing enterprise architecture.
Communications
The report found that directors and staff from top down “all expressed varying level of frustrations of being reactive to the circumstances reported to them, or unable to represent the restoration effort or magnitude of the outages with confidence. The common thread that mends these frustrations together is improving internal communications.”
The incident analysis recommended establishing distributed communication hubs and bolstering market relations training among PEC staff to help improve resiliency.
