Lessons from cybersecurity programs in power plants

Dave McMullan of Xcel Energy and Jeff Reams from Grant County PUD presented two case studies on ‘Security in Real Life’ at the recent Power Gen International conference.

The industry’s challenges discussed at the meet were equipment challenges like aging infrastructure, mechanical interruption, plant performance, and instrumentation. The human factors included issues such as more technology, retiring workforce, fewer specialists, fewer incoming workers, and inexperienced workers.

The presentation touched upon control systems (disparate systems and equipment failures) as well as compliance and security issues like EPA, NERC/CIP, natural disasters, unintentional insider threats and targeted cyber attacks.

The case study focused on the following questions:

Does compliance ensure systems are secure? Does a strong security program ensure compliance? Is there a difference? What can we learn?

Below are excerpts from the presentation:

Case 1 – Xcel Energy - Pawnee

• No compliance obligations under NERC CIP V3

• Practical approach

• Secure systems to prevent hackers and attacks

Case 2 – Grant County PUD

• Critical assets under NERC CIP V3

• Must be auditably compliant

• Compliance first, security best practices second

Xcel Energy – Dave McMullan – Technical Resources and Compliance

 Pawnee Station – 540MW fossil

 Ovation 3.5.0 with OSC 3.0 (includes Intrusion Detection)

Security Approach at Xcel Pawnee

·         No compliance obligations under NERC CIP V3

·         Drivers for security

1. Secure systems – Do the right thing

2. Don’t get hacked

3. Stay out of the media

·         Security program strategy

– Documented program and policies avoid NERC specific language - focuses on security best practices

– Practical approach without going overboard

– Separation between IT and OT – plants control DCS networks

– Tight control of DCS network

– Deployed Ovation Security Center (OSC)

Technologies deployed and processes implemented

Technologies include Ovation Security Center (antivirus, patch management, whitelisting, SIEM & IDS, no external connection) and user accounts (shared by role), back to back firewalls, monthly backups, and trusted USB drives.

Processes & Procedures include physical security controls, identified interconnection rules and rules for control system connections, documentation requirements, training, and evergreen program every three years.

Challenges & lessons learned

Challenges – Training

• Teaching individuals “the right thing to do” when it comes to security

• Understanding technology, OSC learning curve

– Manpower to manage systems

– IT/OT mutual distrust

Lessons Learned

– Keys to a successful security program

• Communication, Communication, Communication

• Understanding, training, familiarity

• Demonstrate benefits, relate to home use

– OSC reduces burden

• Patch deployment reduced from 4 days to 4 hours

What’s Next for Xcel Energy

 Compliance obligations under NERC CIP V5

 Review existing processes and procedures annually

 Re-evaluate and modify policies to follow best practices

 Drive to meet NERC CIP V5 low risk requirements by end of 2014

 Implement OSCs at Xcel sites in Minnesota

Systems Support Engineer Jeff Reams’s presentation highlighted two hydroelectric powerhouses on Columbia River and showcased WECC Audits in June 2011 and June 2014. The two powerhouses were Priest Rapids Dam (10 Units – 950 MW) and Wanapum Dam (10 Units – 1,100MW; 550MW reduced head)

Security approach at Grant County PUD

 Control rooms listed as critical assets under CIP-002

 Drivers for security

– Compliance obligations

– Eliminate self reports

– Secure the Ovation DCS system

 Compliance program strategy

– Documented compliance program and procedures

– Separation between IT and OT – plants control DCS networks but supported from Telecom/Cyber Security engineers

– Only assets on the DCS network are identified as CCA

– Peripheral devices moved to business network or DMZ

– Deployed Ovation Security Center (OSC)CIP Process Owners

Senior Management “volunteered” staff to be Process Owners responsible for each of the CIP Standards

Technologies deployed and processes implemented

The technologies deployed were Ovation Security Center (antivirus, patch management, whitelisting, SIEM (cyber asset), physical security controls, Document Management System, and daily backups via IT.

Processes & Procedures

 Comprehensive processes and procedures cover both EMS and GMS

 Protected Information

Practices

 User Accounts

– Shared for operators

– Unique accounts for engineers

 Planned system upgrades program every 5 years

Challenges

Manpower – security is a one man show

– Finding time to keep up with compliance

OSC Learning Curve – new skill set

– OSC appliances not engineer friendly

Evidence collection – finding good apps

– Grabbing ports, services, account data

– Change management difficult

Consistency between systems

– GMS and EMS use different tools

– Different forms of evidence

Minimizing CIP impact when upgrading equipment

Lessons Learned

Dedicate resources for cyber security

– CIP changes your org chart

– One person per system is not nearly enough

Take time to learn the standards and find technology than can help with tasks

Require others to follow procedures

Procrastinate and you will pay later

Trivial information now could be evidence later

– Email notifications

– Team meeting notes

What’s Next for Grant County PUD

 Already starting to review existing compliance program against NERC CIP V5 requirements

 Moving to SharePoint for document management

 Process Owners assigned to CIP-010 and CIP-011

 Modify current procedures for V5 changes

 Train next generation of cyber security enthusiasts