FEP Diversity Project

RTS System Admin Group

RTUs, Communications
Agenda

1. Substation – EMS Communications Overview

2. Traditional 4-wire communications
3. FEP Diversity Project
a. Motivators
b. Pilot Project - POC
c. Technical Solutions
i. Communications Architecture
ii. AREVA Database Modifications/Changes
d. Additional Benefits

4. Future Direction – Expansion to other Substation

needs

Real Time Systems

2
Substation – EMS Communications Overview

Real Time Systems

3
Substation to EMS data flow

Energy Management System Applications

FEP SCADA Analogs & Statuses Network

SMP – SCADA Management
Subsystem Estimated Values Subsystem
Platform
Frequency, time,
plant & tie-line External unit MW schedules,
measurements, pump unit limits,
Analogs unit controls, incremental cost curves
Accumulators ACE, system load
Statuses Generation Unit security limits,
Controls
Unit Controls Subsystem loss sensitivities,
AGC participation
factors

RTUs Real-Time Control System Environment

HABITAT NETIO Web-FG
Utilities: Configuration Manager MRS Alarm

Substation Operating System

Unix/ Windows 2000

Real Time Systems

4
SCADA protocols

 Protocol is a way of organizing and presenting data in a

consistent, repeatable format
 Bit and byte oriented protocols
– Byte - Harris, DNP
– Bit - Conitel, CDC2, TRW S9000, TRW S9550
– Special interfaces - Intrac and MOSCAD radio systems

 All master station SCADA protocols do the same basic

functions
– analog data conversion
– status point monitoring
– support “select before operate” function for controls
– provide analog output capability
– provide different types of scans to capture transient events between scans
and minimize bandwidth usage
– provide some form of “cyclic redundancy check” CRC so that if the
received message has been corrupted during transport it will be rejected.

Real Time Systems

5
FE RTU numbers and types

 Approximately 2000 RTUs on FE system

 650 in west, 1350 in the east and growing
 Diverse hardware platforms
 Diverse protocols – including:
– DNP 3.0
– Harris 5000
– CDC II
– TRW S9000 / 9550
– Conitel 300/3000
– Modbus
– Intrac/Moscad

Real Time Systems

6
RTU communications

 Primarily four wire leased circuit

– 1200 baud modem

 Radio used extensively in east ~ 650 radio equipped RTUs

– Primarily found at line switch controllers

 Dialup - Approximately 100 dialup RTUs in west

– Primarily found at line switch controllers

Real Time Systems

7
Substation to EMS data flow

Energy Management System Applications

FEP SCADA Analogs & Statuses Network

SMP – SCADA Management
Subsystem Estimated Values Subsystem
Platform
Frequency, time,
plant & tie-line External unit MW schedules,
measurements, pump unit limits,
Analogs unit controls, incremental cost curves
Accumulators ACE, system load
Statuses Generation Unit security limits,
Controls
Unit Controls Subsystem loss sensitivities,
AGC participation
factors

RTUs Real-Time Control System Environment

HABITAT NETIO Web-FG
Utilities: Configuration Manager MRS Alarm

Substation Operating System

Unix/ Windows 2000

Real Time Systems

8
What is a FEP?

 Windows 2k3 server OS

 Standard Infrastructure server Hardware
– One exception, the FEP has additional Digi PCI cards so that
server has 128 COM ports
 Runs the AREVA e_terraControl suite of applications
– Process Starter
– CFEReader – Polls RTU’s
– SCADA – Contains the database
– ISD – Connects the FEP to the EMS
– Ancillary processes:
– AlarmPager (provides method to pass FEP alarms to EMP)
– SpaceSaver
– DBUpdate

 Key: No FEP’s = No Data entering the EMS

Real Time Systems

9
Real Time Systems
10
Traditional 4-wire communications

Real Time Systems

11
Traditional 4-Wire Communications

Real Time Systems

12
FEP Diversity Project

Real Time Systems

13
 Project Motivators

 Issue identified by NERC as a disaster recovery limitation

 Loss of DataCenter, either from a physical or communications
perspective, essentially eliminates the EMS system.
 Upgrade the method of communications to critical transmission
substation RTU’s.
 Network change without causing system software changes
– Keep serial connected FEP ports and serial connected
RTUs
 Improve Communications Network Reliability

Real Time Systems

14
 Project Overview
 Proposal to operate selected critical transmission RTU’s from
physically diverse FEP’s
 3rd/4th Quarter 2005 performed R&D proof of concept testing
 Production deployment began 1st quarter of 2006
 Initial sites include interconnection and critical transmission
sites
 By the end of 2006 we successfully cut over 54 production sites
 By the end of 2007 we successfully cut over 104 production
sites – All 345 and 138kV Tx sites in FE’s MISO area
 111 Transmission sites in PJM are scheduled for completion
though 2012
 All new FE RTU sites are being installed on this technology

Real Time Systems

15
Pilot Objectives - What we wanted to prove?
 Demonstrate that the AREVA system can support data
acquisition using diversified FEP’s at the same level of reliability
as currently observed with co-located FEP pairs.
 Verify the proposed network communications architecture will
support geographically distributed FEP’s.
 Demonstrate that in using diversified FEP’s there is no
degradation in AREVA system level performance (FEP and
EMP).
 Demonstrate that alarm processing performance is equivalent
to that found with legacy designs - throughput.
 Verify system responds properly to the loss of inter-FEP
heartbeat.

Real Time Systems

16
Required work activities in R&D Pilot period

 Build new servers (Windows and Unix)

 Develop a point lists for each RTU
 Develop software configuration for each RTU
 Develop a strategy for modeler configurations
 Develop a standard network hardware configuration
 Configure AREVA Systems
 Configure, debug, and test pilot system
 Develop Cut-over strategies for production

Real Time Systems

17
 Measure of success in all pilot phases?
 Burn in test using all RTU’s for a one week period of local
continuous polling is successful
– No FEP process shutdowns
– Minimal missed scans
 Alarm propagation and data integrity - 24 hour test
– All status points change state at least once per scan interval
– All analog points change each scan interval
– No missed scans or missed alarms
 System stability is maintained for following failover scenarios:
– Loss of one FEP
– Loss of inter FEP heartbeat
– Loss of one EMS server

Real Time Systems

18
FEP Diversity Pilot Phases

 Phase 1 - 10/3/05-10/13/05
– Performed local proof of concept testing using new FEP’s and 55
logical RTU’s. Using DYMEC supplied network hardware, failure
analysis was be conducted and logical RTU’s polling statistics
were gathered. FRAME network is all mocked up locally at
DataCenter B.

– End Result- Satisfactory test results based on operational

performance and failure analysis. All testing performed locally at
DataCenter B.

Real Time Systems

19
FEP Diversity Pilot Phases -CONT.

 Phase 2 - 10/14/05-10/25/05
– DYMEC Equipment was extended to DataCenter A to allow
controlled diversified FEP testing. Required relocating a FEP
server to DataCenter A, and the associated FRAME network
connectivity.

– End Result- Satisfactory test results based on operational

performance and failure analysis.

Real Time Systems

20
FEP Diversity Pilot Phases - CONT.

 Phase 3 -10/26/05-11/16/05
– Selected substations (6 site’s) will be dual ported and connected to
digital circuits. Digital FRAME circuits will appear at DataCenter A
and DataCenter B where the diversified FEP’s and EMS servers
reside.
– Includes site’s on the FE, and the Telco (leased) FRAME network’s

– End Result- Satisfactory test results based on operational

performance and failure analysis.

Real Time Systems

21
Results of Pilot
 Project team successfully installed and configured the new versions of the
SMP software on FEP pair.
– Latest Windows O/S
– Latest AREVA SMP Software
– New hardware components
 Verified successful RTU communications using current serial
communications.
 All three phases of RTU testing completed successfully, including phase
III’s real-life field tests
 Developed Cutover Plan

Real Time Systems

22
Production Implementation Project

 Order communication circuits as required

– A majority of circuits require High Voltage Protection (Positron)

 Make necessary RTU configuration changes

– Baud Rate / Modem settings

 Complete AREVA modeler work to allow cutover to occur

– Run ManageSiteAssignments.pl script to use Primary/Secondary SITE ID
– Make PATH changes for new communication circuit

 Cutover RTU’s to Diversified FEP

– Perform full station check-out on new circuit

Real Time Systems

23
Project Timeline

Real Time Systems

24
FEP Diversity Architecture

Real Time Systems

25
 FEP
DIVERSIFICATION
DYMEC
(Physical)

8 PORT TERMINAL SERVER

(SERIEL CONNECTED max.: 38400
DEVICES i.e: RTU, DFR, bps
LINE SWITHCES...etc)

10/100
- DSU/CSU (Digital Circuit Interface) 5 PORT ETHERNET SWITCH
FRAME RELAY (T1-Fractional, 64k)
DYMEC (IP ONLY DEVICES i.e.: PC,
NETWORK IP CAMERA...etc)
-FRAD (Frame-Relay Access Device)

- ROUTER
-SUBSTATION HARDENED
(-40C -+75C) no fans
- BUILT-IN DATA ANALYZER
- FRAME/SERIAL ENCAPSULATION RTU PROTOCOL

Network Services Engineering

Confidential

Real Time Systems

TITLE

FEP DIVERSIFICATION PILOT PHASE #1

26
FILENAME REVISED PAGE DRAWN BY

9/14/2006 1 OF 1 REX DUDAS

FULL FILENAME
 FEP
DIVERSIFICATION
DYMEC
(LOGICAL)

- FRAME/SERIAL ENCAPSULATION RTU PROTOCOL

(NON-ROUTABLE, ENCRYPTED)

DataCenter A FEPA RTU MAINTENANCE PORT

RTU

DataCenter B FEPB 8 PORT TERMINAL SERVER

FRAME RELAY (SERIEL CONNECTED
NETWORK DEVICES i.e: RTU, DFR,
PVC 1 LINE SWITHCES...etc)

PVC 2
DYMEC
PVC 3 (ROUTER)

5 PORT ETHERNET SWITCH

(IP ONLY DEVICES i.e.: PC,
IP CAMERA...etc)
IP ROUTABLE PROTOCOL REMOVED OVER WAN
CORP. TRAFFIC LINK
(FIREWALL, ENCRYPTED)

Network Services Engineering

Confidential
TITLE

FEP DIVERSIFICATION PILOT PHASE #1

Real Time Systems FILENAME REVISED

9/14/2006
PAGE

1 OF 1
DRAWN BY

REX DUDAS
FULLFILENAME

27
Cut-over Strategy (part1)
 Need to have both existing and diversified RTU modeled for
cutover in AREVA EMS
– Model an additional TFE structure for new diversified FEP
– Changes to BAUD, PTMO1, PTMO2 and NoReplyTimeout
– Additional RTUC Record pointing to same RTU record as 4-wire circuit
in production
– Utilize SITE_* and SITE2_* fields to facilitate the cutover
– Each RTU has potentially hundreds of POINT/ANALOG/COUNT
records associated with it.
– These all need their SITE_* field set to the ID_SITE of the new diverse
FEP.
– SITE2_* field set to the ID_SITE of the current 4-wire FEP in production
– Manage those individual data items that do NOT allow automatic
SITE/SITE2 selection.
– Selection is done automatically by SCANNER based on quality. Items
with no status, (CTRL only) do not have a good quality code to follow
(TAP Changers)
 Areva TFE tree changes made, No RTU tree changes made.
– Limit the risk of the change – No production affects

Real Time Systems

28
Cut-over Strategy (part2)
 RTU Changes
– New FRAME T1 Circuit can be installed at any time on
spare pairs
– Make RTU configuration changes and load onto spare D20
processor board
– Swap out board for cutover, if there are issues original board
can be immediately put back
– No Point mapping changes made

Real Time Systems

29
POST DESIGN BENIFITS
 DR Environment
– Remote RTU can be pulled from either DataCenter
 Eliminates costly – Out-of-date Analog 4-wire circuits
 Increase RTU speed/poll interval
– 1200bps to 9600bps (max.: 38400 bps)
 Ability to pull byte and bit-oriented RTUs
 Remote Support
– Allow for mobile workforce support/configuration/data analysis – Secured IP Based
 Security
– Serial Connectivity – Isolation of corporate substation traffic
 HP/OV Support
– Complete Network end-to-end alarming
 “Smart” Substation platform
– Ability to add additional PVC’s for other needs
– IP enabled Security Cameras for monitoring site
– Swipe card entry systems
 CIP Cyber Security full compliance
– Non-Routable Transport

Real Time Systems

30
FEP Diversification Overview

Real Time Systems

31
 FirstEnergy FEP Diversity: Proposed IP Based Network Architecture A
GMS DACs GMS DACs EMS FEPs EMS FEPs EMS FEPs
DC-A DC-B DC-A DC-B DC-C

GMS SONET/RPR EMS SONET/RPR

Backbone Backbone

ElectronicSecurityPerimeter

Dynamic Dynamic
RTS FW Failover RTS FW
Failover RTS FW
Corp FW DC-A DC-B DC-C
DC-A
Electronic Security Perimeter
FE Corp Network
RPR Backbone

Consolidate RTU Communication over an

Substation RTU Network IP Based FE RPR Wan and/or Public
RPR Exchange WAN Carrier Frame/MPLS Networks.
BG #TBD

Corp FW
Sub-RTR Dynamic Sub-RTR Dynamic Sub-RTR
DC-D DC-A Failover DC-B Failover DC-C
Fa s ard
ML C

T1
Substation Aggregate to T1
tE t o

RTU Router stE ard T1

RTU a
F LC tE t o Substation
Site #1a Site #1
M Fa s ard RTU
Fractional T1 ML C Public Carrier Site #5
Substation FEIsolatedWAN FR/MPLS
RTU TCP/IP
Site #1b
SONET/RPR Substation
Aggregate Network RTU
RTU Router Site #4
Fa s ard
ML C
Ca o

Site #2
ML stE t
rd

tE t o
Fa

Aggregate Substation
Substation
RTU Router RTU
Aggregate Substation RTU Substation
Site #3 Site #3
RTU Router RTU Router Site #1 RTU
Site #4 Site #5 Site #2

Real Time Systems

32
 FirstEnergy FEP Diversity: Proposed IP Based Network Architecture B
GMS DACs GMS DACs
DC-A DC-B EMS SONET/RPR
Backbone

GMS SONET/RPR
EMS FEPs EMS FEPs EMS FEPs
Backbone DC-A DC-B DC-C

DYMEC DYMEC DYMEC

ElectronicSecurityPerimeter

Dynamic Dynamic
RTS FW Failover RTS FW
Failover RTS FW
Corp FW DC-A DC-B DC-C
DC-A
Electronic Security Perimeter
FE Corp Network
RPR Backbone

Consolidate RTU Communication over an

Substation RTU Network IP Based FE RPR Wan and/or Public
RPR Exchange WAN Carrier Frame/MPLS Networks.
BG #TBD

Corp FW
Sub-RTR Dynamic Sub-RTR Dynamic Sub-RTR
DC-D DC-A Failover DC-B Failover DC-C
Fa s ard
ML C

T1
Substation Aggregate to
tE t o

RTU Router stE ard T1 T1

RTU a
F LC tE t o Substation
Site #1a Site #1
M Fa s ard RTU
Fractional T1 ML C Public Carrier Site #5
Substation FEIsolatedWAN FR/MPLS
RTU TCP/IP
Site #1b
SONET/RPR Substation
Aggregate Network RTU
RTU Router Site #4
Fa s ard
ML C
Ca o

Site #2
ML stE t
rd

tE t o
Fa

Aggregate Substation
Substation
RTU Router RTU
Aggregate Substation RTU Substation
Site #3 Site #3
RTU Router RTU Router Site #1 RTU
Site #4 Site #5 Site #2

Real Time Systems

33
Questions?

Real Time Systems

34