Between the Poles

Cybersecurity vulnerability of the U.S. power grid underscored in new report

A report on grid cybersecurity released last week by US Representatives Ed Markey and Henry Waxman makes for fascinating reading.

Background

The U.S. bulk power system is relied on by 300 million people and is comprised of 200,000 miles of transmission lines and about a thousand gigawatts (GW) of generating capacity.  It is valued at over $1 trillion.  Most of the bulk power grid is owned and operated by private companies, municipally- and coop-owned utiltiies.

The components of the grid are highly interdependent.   An outage in one area can lead to cascading outages in other areas.  The classic example occurred in 2003 when four high voltage power lines in northern Ohio brushed trees and shut down. A computer system error caused a cascade of failures that left 50 million people without power for two days across the United States and Canada.  The largest blackout in North American history cost the economy an estimated $6 billion.

Vulnerabilities

This report makes the case that grid vulnerabilities pose substantial risks to U.S. national security.  It cites a 2008 report by theTask Force on Department of Defense (DOD) Energy Strategy that said that  “critical missions . . . are almost entirely dependent on the national transmission grid."  About 85% of the energy infrastructure upon which DOD depends is commercially owned, and 99% of the electricity DOD consumes originates outside of DOD. In most cases, neither the grid nor on-base backup power provides sufficient reliability to ensure continuity of critical national priority functions and oversight of strategic missions in the face of a long term (several months) outage. An October 2009 report by the Government Accountability Office said that 31 of DOD’s 34 most critical global assets rely on commercially operated electricity grids for their primary source of electricity.

I remember a startling statistic in an Energy Information Adminstration (EIA) publication that the failure of 4% of U.S. substations would result in 60% of the U.S. losing power. The Markey and Waxman report also cites a declassified National Academy of Sciences report declassified that found that physical damage to large transformers could disrupt power to large regions of the country and take months to repair.

Very recently, the Department of Homeland Security testified that it had processed 68% more cyber-incidents in 2012, involving Federal agencies, critical infrastructure, and other select industrial entities, than in 2011.

Cybersecurity regulation

Measures to protect the U.S. electric grid from cyber-attack include mandatory reliability standards developed by the North American Electric Reliability Corporation (NERC) plus voluntary actions recommended by NERC.

In 2010, bipartisan cyber-security legislation called the GRID Act passed the House of Representatives.  This legislation would have provided the Federal Energy Regulatory Commission (FERC) with the authority to require necessary actions to protect the grid. However, this legislation did not pass the Senate.

Cybersecurity survey of U.S. utiltiies

In January of this year, Representatives Markey and Waxman requested information from more than 150 investor-owned utilities (IOUs), municipally-owned utilities, rural electric cooperatives, and federal entities.  More than 60% of the entities have responded including

• 54 investor-owned utilities
• 47 municipally-owned utilities and rural electric cooperatives
• 12 federal entities

The Markey and Waxman report is based upon those responses.

Findings

The electric grid is the target of numerous and daily cyber-attacks.

More than a dozen utilities reported “daily,” “constant,” or “frequent” attempted cyber-attacks ranging from phishing to malware infection to unfriendly probes.

One utility reported that it was the target of approximately 10,000 attempted cyber-attacks each month. More than one public power provider reported being under a “constant state of ‘attack’ from malware and entities seeking to gain access to internal systems." A Northeastern power provider said that it was “under constant cyber attack from cyber criminals including malware and the general threat from the Internet...”  A Midwestern power provider said that it was “subject to ongoing malicious cyber and physical activity. For example, we see probes on our network to look for vulnerabilities in our systems and applications on a daily basis. Much of  this activity is automated and dynamic in nature – able to adapt to what is discovered during its probing process.”

Most utilities only comply with mandatory cyber-security standards, and have not implemented voluntary NERC recommendations. 

Almost all utilities cited compliance with mandatory NERC standards. Of those that responded to a question of how many voluntary cyber-security measures recommended by NERC had been implemented, most indicated that they had not implemented any of these measures.

For example, NERC has established both mandatory standards and voluntary measures to protect against the computer worm known as Stuxnet. Of those that responded, 91% of IOUs, 83% of municipally- or cooperatively-owned utilities, and 80% of federal entities that own major pieces of the bulk power system reported compliance with the Stuxnet mandatory standards. By contrast, of those that responded to a separate question regarding compliance with voluntary Stuxnet measures, only 21% of IOUs, 44% of municipally- or cooperatively-owned utilities, and 62.5% of federal entities reported compliance.

Most utilities have not taken concrete steps to reduce the vulnerability of the grid to geomagnetic storms and it is unclear whether the number of available spare transformers is adequate.

Only 12 of 36 (33%) responding IOUs, 5 of 25 (20%) responding municipally- or cooperatively-owned utilities, and 2 of 8 (25%) responding federal entities stated that they have taken specific mitigation measures to protect against or respond to geomagnetic storms.

Most utilities do not own spare transformers. Only twenty IOUs, six municipally- or coop-owned utilities, and eight federal entities reported owning spare transformers. While other utilities reported participation in various mutual assistance agreements or industry equipment sharing programs, none knew how many other utilities would claim contractual access to the same equipment in the event of a large-scale outage.

IEEE PES Smart Grid Technologies: The next challenges in grid modernization

At the IEEE PES Conference on Innovative Smart Grid Technologies in Washington DC,  Patricia Hoffman, Asst Secretary Department of Energy, Office of Electricity Delivery and Energy Reliability, gave the  keynote address. 

The whole is greater than the sum of its parts

In the past the focus for the Department of Energy and for utiltiies has tended to be on individual technologies such as smart meters and AMI, self-healng networks, synchrophasars, substation automation, and so on.  Ms Hoffman's message is that we need to start focussing on the smart grid as a whole and in particular we need to focus on grid resiliency.  Things will happen, floods, tornadoes, hurricanes, derechos and other natural and man-made problems.  We need to ensure that the grid is resilient so that we can recover from disasters quickly.  The example she gacve was one that I have blogged about,  Southern Co.'s use of  smart meters to track outages caused by tornadoes in April, 2011 and then help with restoration. 

Ms Hoffman then discussed specific areas where there are opportunities to make the grid more resilient.

Transmission operations

There is a lot of data that is becoming available, primarily because about 1000 synchrophasars have been deployed over the national transmission grids.  What are needed are improved interoperability, analytic, and visualization to better manage the transmission grid.  For example, it took almost a year to analyze the data and determine the cause of the 2003 Northeast blackout.  The 2011 San Diego blackout  required about 3-4 months, so fault analysis is getting better, but we need to be able to use the available data to predict the state of the transmission grid in real-time.

Distribution operations

In the area of electricity distribution, Ms Hoffman singled out peak load reduction as the area where there are the greatest econimic and environmental benefits.  Reducing peak load means that the construction of new power plants (peakers) that may only be used a hundred hours a year can be avoided.  The other areas she identified include distribution automation, especially automated self-healing networks using devices such as intelliRupters,  improved outage management, using predictive analytics to replace equipment before it fails, and microgrids.

Customer empowerment

Areas that directly impact the customer are residential diagnostics to help the customer understand how power is being used in the home and ways that it can be used more effectively, at lower cost, and as much as possible moved to off-peak.  Photovoltaic cells and plug-in electric vehicles are becoming much more prevalent and utlities need to be prepared for these.  Improving reliability and resiliency directly impacts customers by reducing the number and duration of outages.  An area that is getting a lot attention is rate structures, for example, pricing electric power to motivate shifting load from peak to off peak.  A new technology that is being investigated by DoE is transactive control signals.  This is a  smart grid signaling approach that communicates prices and other information, such as expected future prices, to encourage the customer to change his/her electric power usage.

Energy infrastructure cyber protection

Ms Hoffman's message on cyber protection is that it needs to be dynamic and multi-dimensional.  Infrastructure is complex and changing and new vulnerabilities are detected every day.  In addition multiple organizations are responsible for the grid.  Effective cyber security management has to deal with a dynamic landscape of control and information systems.  There are a broad range of threats and motivations including unintentional, natural phenomena, and those with malicious intent including theft, revenge, and national security.  It is also important to recognize the interdependencies among critical infrastructure.  A gas explosion can affect the telecommunications and electric power networks.  Loss of electric power can affect many other networks including water and telecommunications.  Disruption of communications, for example, severing a fiber optic cable can directly affect the electric power grid.

To address these challenges cybersecurity solutions need to be mutli-dimensional.including cyber-physical, wide area monitoring and protection, cryptography, anti-tamper devices and advanced architectures. 

One of the most serious challenges that Ms Hoffman singled out specifically is developing the cybersecurity workforce that is going to be responsible for implementing these measures.

Where are we headed ?

The next major challenge is turning the smart grid into a driver of economic development.  The example Ms. Hoffman used was Chatanooga.  The Electric Power Board (EPB) saw the smart grid as not only benefiting the utility and improving the quality of life of its cusomers, but also as a driver of economic development.  The city’s decision to deploy one of the country’s highest capacity fiber-optic networks not only improved power quality, reliability, customer service and energy efficiency but also made Chananooga more attractive to business.  For instance, because bandwidth is virtually unlimited, EPB is able to offer its customers Internet upload and download speeds of up to one gigabit/sec, making Chattanooga very attractive for businesses that rely on the internet for critical aspects of their operations.

NRECA TechAdvantage: MultiSpeak, where it is and where it is going

At the NRECA TechAdvantage conference in New Orleans Gary McNaughton, MultiSpeak Techical Coordinator gave a comprehensive overview of where the MultiSpeak interoperability standard is and where it is headed.  If you're not familiar with MultiSpeak, it has been developed under the auspices of NRECA and is intended as an industry-wide standard for the exchange of information for electricity distribution utilities and all vertically-integrated segments except power marketing and generation.  It is used by more than 725 electric coops, investor-owned, and municipal utilities in 19 countries and is widely recognized and supported by North American software vendors.

Cybersecurity

In light of President Obama's recent Executive Order on cybersecurity for critical infrastructure, security has become even more critical and Gary's message is that utilities need to get very serious about it.  To support this a new security standard for MultiSpeak was released in January 2013.  It goes beyond secure sockets (SSL) and transport layer security (TLS) and implements message-level security.  The standard requires that vendors support four security options, which they have agreed to do.

1. None (for debugging)
2. SSL or TLS
3. Message-level - by message
4. Message-level - by session

It is up to the utility to decide which of these it needs to implement.  There is a Cooperative Research Network (CRN) security policy template that is designed to help utilities to choose the appropriate option.  The standard has been submitted to Smart Grid Interoprability Panel (SGIP) for inclusion in the Catalog of Standards for the smart grid.

Use cases

The MultiSpeak team has developed over 300 use cases for specific business processes such as connecting a customer.  They cover business processes for demand response, meter data management, meter prepayment, outage management, and asset management.  So far they do not include GIS-related use cases (MultiSpeak has supported geospatial data types for years.)

Harmonization with CIM

MultiSpeak and the Common Information Model (CIM) have been converging in many ways over the years.  But according to Gary, at this point there is no plan to harmonize the two models in Version 5.  And it is not clear if they will ever be harmonized.

MultiSpeak 3

The big news is that there will be no further bug fixes and enhancements for Version 3, which is the most widely used version of MultiSpeak.  Gary said there was no reason for utilities to stop using Version 3, which is 10 years old now, unless the additional functionality in later versions is required.

MultiSpeak 4

Versions 4.1.5 and 4.1.6 have been released, with enhancements specifically to support the MultiSpeak Regional Demonstration project. It is expected that vendors will begin implementing support for this Version 4.1 in the next few months.  The big advantages of 4.1 are a more consistent data model and support for AMI, DR, MDM, AVL, and work management.  A nice thing about. MultiSpeak's web services implementation is that it means that a utility can run several versions of MultiSpeak concurrently.

MultiSpeak 5

Work has begun on a new version which will include work management enhancements and, most importantly, modernized web services support.  A prerelease will be available in May 2013.

Enterprise Service Bus

Gary made a persuasive case for using Multispeak in all but the simplest implementations with an enterprise service bus because it eliminates some of the problems associated with point to point messaging over TCP/IP.  For example, TCP/IP does not guarantee that a message is received or that several messages will be received in the expected chronological order.

President Obama signs Executive Order on cybersecurity for critical infrastructure

IPrior to the President's State of the Union Address, President Obama signed an Executive Order on cybersecurity. This has  important implications for the electric power industry and is a wakeup call for utilties that have not yet developed or updated their cybersecurity policy.

Michael Daniel, Special Assistant to the President and Cybersecurity Coordinator, on the White House Blog, hs given an overview of the Executive Order including the motivation for it and its key provisions.

According to Mr Daniel, the government’s senior-most civilian, military, and intelligence professionals all agree that inadequate cybersecurity within the nation's 18 critical infrastructure areas poses a threat to the security of the United States.  Because of the seriousness of the threats, the President issued an Executive Order directing federal departments and agencies to use their existing authorities to provide better cybersecurity for the Nation.  The Administration received input from a broad range of stakeholders in industry, the public sector, the legislative branch, and the advocacy community including over 30 organizations representing all 18 critical infrastructure sectors.

The Executive Order focusses on the three areas, information sharing, a framework of core security practices based on existing standards, and privacy protections.

Information sharing

The Order makes it is a national priority to increase the cyber threat information shared with authorized individuals and companies.  In particular, it aims to improve information sharing between the private sector and all levels of government.  It expands the Department of Homeland Security’s (DHS) Enhanced Cybersecurity Services program to provide near real-time sharing of information on cyber threats with critical infrastructure companies and state and local governments.

Cybersecurity framework

The Executive Order directs the National Institute of Standards and Technology (NIST) to lead the development of a framework to reduce cyber risks to critical infrastructure. NIST is directed tol work with industry to identify existing voluntary consensus standards and industry best practices to incorporate into the framework.  The Order puts private-sector cyber leaders in critical infrastructure sectors at the core of  the development of voluntary best practices for the framework.  The DHS Secretary is directed tol establish a voluntary program to support the adoption of the Cybersecurity Framework by owners and operators of critical infrastructure.  This has direct and immediate implications for utilities.

Privacy

The Executive Order directs departments and agencies to incorporate privacy and civil liberties protections into cybersecurity activities based upon widely-accepted Fair Information Practice Principles as well as other applicable privacy polices.

The state of malware on the internet or why utilities need to prioritize cyber security

In its essence smart grid is about turning the power grid into an internet of intelligent devices.  That means that the next generation  power grid will be susceptible to all the cybersecurity issues that affect the internet.   This makes cybersecurity a priority for every utility that is implementing some form of a smart grid.

At the EDIST 2013 Conference in Toronto, the keynote speaker was Mikko Hypponen of F-Secure, who has been chasing cybersecurity problems since the 1990's and has been responsoble for taking down a number of internet malware exploits.  He gave a very current assessment of today's cybercrime. 

Cybercrime on the internet today is very different from a decade ago  when it involved "kids" with no real motive, who just did it because they could. 

Now there are three categories of people who indulge in cyber expolits and they all have motives.

1. Criminals - organized crime groups (in MiKko's experience primarily in Russia, Ukraine, Brazil, China, but all over the world) whose motive is money
2. Hacktivists - groups with a political agenda and whose motive is protest
3. Governments - both totalitarian and democratic governments are involved in offensive malware with a variety of motives, espionage and military are obvious ones, but governments also target malware on their own citizens

Criminals

Mikko gave an example of a criminal cyber exploit involving credit card theft but with a wrinkle.    Typically when this activity is tracked to a particular account, the ISP is notified and will shutdown the account.  But in this case the ISP was owned by a criminal.  When notified he would promise to would look into the problem, but wouldn't take any action except charging the account owner more for hosting.  Hosting would get to be very expensive for an account holder if it got a lot of attention from the credit card companies.  The ISP owner bought a lot of real estate with the money he made - 155 properties have been identified.  He is currently awaiting trial.

Another example of a criminal exploit is banking trojans.  The best known is called "Zero-access" and has infected 9 million computers.  The way it generates revenue is by interrupting your on-line banking session and displaying what appears to be a message from your national police force ( it is a different "police force" depending on the country where you are using your computer) telling you that illegal financial acitivities have been traced to your account and your access the the account has been disabled.  This all sounds legit, until it goes on to say that if you pay a $100 fine, access will be restored.  And it provides a legitimate way for you to transfer the money.

One of the surprising changes in cyber crime is that until recently it has involved exploits targetting computers running Windows.  Millions of Windows machines are infected every year.  But the 75 % of the world's servers that run Linux have not been affected. (Last quarter for the first time Linux passed Windows in number of new installations.)

But now Linux, in the form of Android on smart phones, has a malware problem.  The number of devices affected by attacks currently number in the thousands, but this is increasing rapidly and involves mostly criminals, not hacktivists or governments.

Hacktivists

One of the best known hacking by hacktivists involves a programmer who wanted to customize his Playstation software.  Unwisely Sony sent its lawyers after him, which immediately became known and had the effect that Sony became a favourite target of the hacker community.

In an interesting contrast, a programmer who wanted to customize his IPhone, was hired by Apple.

Governments

The best known is the stuxnet worm that targeted Iran's uranium centrifuge equipment running Siemens Step7 software.  The source of this exploit is suspected to be a Western government.  Mikko presented a fascinating story about how this exploit was traced and identified from public TV broadcasts.

ARAMCO, the largest company in the world, was attacked by an expoit that shut down 75% of its computers.  A Middle Eastern government is the prime suspect.

The "red october" exploit has just been identified by Kapersky and the source is suspected to be the government in the Far East.  This exploit involves sending what looks like innocuous official and academic articles to government officials which then infects their computer with a trojan.

But governments also use malware against their own citizens.  Mikko mentioned Syria, the previous regime in Egypt, but also the US, Germany and the UK.

In the latter cases, this typically involves trojans that are used during criminal investigation to track the communications of suspects and is similar to telephone tapping.  Mikko described a fascinating example where a person suspected of criminal activity was separated from his luggage and questioned by Customs and Immigration at an airport.  The questioning was bogus, but while he was being questioned, officials went through his luggage, found his computer and installed a trojan on it.  He was subsequently found to be innocent. The officials then had to stage the same airport exercise to get the trojan off of his computer.

This was a fascinating presentation which I suspect created in the utilities folks in the audience a significantly heightened awareness of how seriously the cybersecurity challenge to the electirc power grid must be treated.