Migration from SONET/SDH to Optical Ethernet in Metro

White Paper

Migration from SONET/SDH to Optical Ethernet In Metropolitan Area

By Jiyang Wang

ATRICA, INC. 3255-3 Scott Blvd. Santa Clara, CA 95051 http://www.atrica.com/

1 of 23

Migration from SONET/SDH to Optical Ethernet in Metro 1. Introduction ............................................................................................................................... 4 2. The Requirements of Metro Ethernet Services ........................................................................ 5 3. A Brief Introduction of NG SONET/SDH ................................................................................ 6 3.1 The challenges that legacy SONET/SDH is facing ............................................................. 6 3.2 A Brief Introduction on Next Generation SONET/SDH ................................................... 7 3.2.1 Virtual Concatenation (VCAT).................................................................................... 7 3.2.2 Generic Framing Procedure (GFP) ............................................................................. 9 3.2.3 Link Capacity Adjustment Scheme (LCAS) ............................................................... 9 3.2.4 A Typical NG SONET/SDH Sub-system ..................................................................... 9 4. What Is Carrier-class Optical Ethernet .................................................................................. 10 4.1 Four Major Enhancements for Ethernet to Become Carrier-class................................. 10 4.1.1 End-to-End QoS.......................................................................................................... 10 4.1.2 Sub-50ms Protection................................................................................................... 10 4.1.3 Ethernet OAM ............................................................................................................ 11 4.1.4 Scalability .................................................................................................................... 11 4.2 Network Management and Service Provisioning ............................................................. 11 4.3 Support for More Services................................................................................................. 12 4.3.1 E-LAN Service ............................................................................................................ 12 4.3.2 Circuit Emulation Service (CES) ............................................................................... 12 4.3.3 Service Multiplexing ................................................................................................... 12 4.4 The Cost-effectiveness of Optical Ethernet....................................................................... 12 5. Metro Ethernet Services The Battle Field for Optical Ethernet and NG SONET/SDH ... 13 5.1 Economic Analysis ............................................................................................................. 13 5.1.1 Capex ........................................................................................................................... 13 5.1.2 Service Price ................................................................................................................ 14 5.1.3 The Cost of Upgrading Existing SONET/SDH Rings............................................... 15 5.2 Technical Limitations of NG SONET/SDH in Supporting Data Services ...................... 15 5.2.2 GFP Multiplexing Has A Critical Limitation............................................................ 15 5.2.3 The Limitations in Multi-point Service and Service Multiplexing........................... 16 5.2.4 The Inflexibility of Bandwidth Change ..................................................................... 17 6. The Positioning and Integration of NG SONET/SDH and Optical Ethernet ....................... 17 6.1 The Positioning of Existing SONET/SDH......................................................................... 18 6.2 How Can Optical Ethernet Plug In Smoothly and Seamlessly........................................ 18 6.2.1 Optical Ethernet on Only One Side ........................................................................... 18 6.2.2 Both Sides Are Greenfields......................................................................................... 19 6.2.3 Scaling The Optical Ethernet Metro Networks......................................................... 20 6.2.4 The Integration of Network Management and Service Provisioning ...................... 20 7. Summary................................................................................................................................... 22 Appendix A: More Details on The Limitations of Virtual Concatenation................................ 23

2 of 23

Migration from SONET/SDH to Optical Ethernet in Metro A-1 The Theoretical Limitations ............................................................................................. 23 A-2 The Limitations of Virtual Concatenation in Reality ..................................................... 23

3 of 23

Migration from SONET/SDH to Optical Ethernet in Metro

1. Introduction
Recently several market research and consultant companies have published reports on Ethernet deployment in Metropolitan Area Networks (MAN) and Ethernet services in the MAN and across the WAN. These reports demonstrate the increasingly important role of Ethernet services in carriers revenues over the next 5 years. These reports also illustrate that Ethernet, as a transport technology, is extending its domination from LAN to MAN. According to a report from RHK in March 2002, Ethernet services will generate $4 billion in service provider revenues in North America by 2006, representing a 70% CAGR (Compound Annual Growth Rate). Another recent report in early 2003 from Ovum, a market research and consultant company, projects 362.6 thousand Ethernet lines to the curb or to the basement of buildings that carry Ethernet services to consumer and business subscribers, with North America, Asia Pacific, Western Europe and the rest of the world sharing 36%, 35%, 25% and 4% respectively. These projections are verified by the progress we have witnessed over the last two years. Many activities are being performed by major incumbent and competitive carriers all over the world from lab tests and field trials on various flavors of metro Ethernet, to real deployment of production Ethernet metro networks. Many of these carriers have started to offer extensive Ethernet services. The key drivers of Ethernet services are the needs for high-bandwidth with low-cost as well as ease-ofuse by enterprises and by service providers themselves. Enterprises are trying to lower their IT costs by centralizing these services to headquarters and one or two recovery centers. Critical resources such as databases and sophisticated devices including servers and security firewalls are dramatically increasing the bandwidth usage between branch sites and central sites. High and service differentiated bandwidth is key to enterprises running a new breed of applications such as ERP, CRM, SAN, video conferencing and on-line training. Many tenders have been issued by enterprises asking for 100Mbps or even 1Gbps bandwidth. Notably all these enterprises prefer Ethernet connectivity for the following reasons: Affordable high bandwidth on an as-needed basis Reduced cost on the access device Easy to scale bandwidth without changing hardware Satisfactory QoS and reliability Internet Service Providers (ISPs), are trying to connect their subscribers in the most cost effective way to the IP service routers located in the main PoPs. Because almost all subscribers use Ethernet in their LANs, and Ethernet interface cards in IP service routers are much cheaper than ATM and PoS cards, ISPs prefer to use Ethernet interface in the CPE routers and in the IP service routers. While Ethernet services seem to have gained tremendous momentum and consensus, opinions on the way to implement them have been divided. One approach to implementation is to take advantage of existing SONET/SDH infrastructure, add new access devices or interface cards that can encapsulate

4 of 23

Migration from SONET/SDH to Optical Ethernet in Metro Ethernet frames in appropriate SONET/SDH payloads and transmit the Ethernet traffic to the other end. This solution is called Next Generation SONET/SDH (NG SONET/SDH). The encapsulation mechanism is also designed to support other traffic types such as storage traffic. Another implementation is to use pure Ethernet at the transport layer, thus taking advantage of the costeffectiveness and ease-of-use of Ethernet, and add carrier-class features. We call this solution Carrierclass Optical Ethernet or simply Optical Ethernet (OE) opposed to the current enterprise-class Ethernet products and solutions. This paper provides a clear definition of Ethernet services, followed by a brief introduction on NG SONET/SDH and Optical Ethernet. The paper then focuses on discussing the disadvantages and limitations of NG SONET/SDH in offering Ethernet services and how NG SONET/SDH can be used to integrate legacy SONET/SDH with Optical Ethernet.

2. The Requirements of Metro Ethernet Services

Metro Ethernet Forum (MEF), an industry forum comprised of over 70 members from ILECs, CLECs and almost all the major equipment vendors who share interest in Metro Ethernet, is working on defining Ethernet services. Current drafts include point-to-point Ethernet Line connection (called Ethernet Line or E-Line) and multi-point Ethernet LAN connection (called Ethernet LAN or E-LAN). E-Line functions as a transparent leased line between two demarcation points. The demarcation point is an interface facing the customer in the access device owned by the carrier. This interface is called UNI (User Network Interface). E-Line can be used to interconnect two customer sites or to connect a customer site to an ISP. E-LAN is usually used to interconnect multiple customer sites and appears to the customer as a transparent LAN. The following service attributes have been defined for these two services (according to MEF Ethernet Services Definition Draft 2.0): Privacy the traffic of the sites that are not on the same Ethernet Line is completely separated, thus maintaining security and privacy. Transparency Ethernet Line does not need to change customers internal network. For example, customers are free to use any VLAN and IP addresses in their LANs. Traffic parameters Committed Information Rate (CIR), Committed Burst Size, Peak Information Rate (PIR) and Maximum Burst Size are defined for Ethernet Line. Other service-related parameters include sub-50ms protection and Ethernet OAM that are defined in other drafts. Priority Traffic prioritization is based on physical port, VLAN ID, IEEE 802.1p bits, IP TOS or DiffServ DSCP bits. Other attributes such as delay, jitter and packet loss rate can also be applied to E-Line and E-LAN services.

5 of 23

Migration from SONET/SDH to Optical Ethernet in Metro

MEF Service Definition Draft also describes a function called Service Multiplexing, in which an E-Line connection is established from each of the multiple remote sites to a central site and the central device aggregates the traffic from those E-Line connections onto a single physical port. E-Line and E-LAN have many applications. They can be used to implement leased line type of services and can also be used for DSL backhaul, SAN interconnection, as well as transport network for IP VPN services in which customer edge (CE) routers are connected transparently to service providers edge (PE) routers via E-Line or E-LAN. Figure 1 shows that the device at site A (a central router, for example) uses a single interface to multiplex the traffic over three E-Line connections from the remote sites. Figure 1: Service Multiplexing
Site B

E-Line 1

Site C Site A

E-Line

Site D

E-Line

The discussions from hereon will take the MEF service definitions into account. This paper will delve into more details on how well NG SONET/SDH and OE implement E-Line and E-LAN.

3. A Brief Introduction of NG SONET/SDH

3.1 The challenges that legacy SONET/SDH is facing As a voice-optimized TDM technology, legacy SONET/SDH is facing critical challenges in dealing with Ethernet-based data traffic: Ethernet traffic is bursty and sporadic. But because SONET/SDH allocates a pipe with fixed bandwidth between a pair of endpoints, even when there is no traffic in this pipe, the congested traffic in other pipes cannot use its bandwidth. This is the fundamental inefficiency of TDM when handling Ethernet traffic. A more efficient technology is statistical multiplexing that allows multiple traffic flows to use the maximum available bandwidth on a physical link at any time.

6 of 23

Migration from SONET/SDH to Optical Ethernet in Metro Ethernet rates are not fixed. They can grow from 1Mbps to 1Gbps smoothly at a step of 1Mbps or less, while legacy SONET/SDH has to allocate bandwidth in a pre-structured manner in larger increments (for example, if a 51Mbps pipe is not enough, the next jump is to 155Mbps). Ethernet rates do not match SONET/SDH rates. In order to support full-speed Ethernet traffic, legacy SONET/SDH has to allocate a bigger pipe, which leads to bandwidth inefficiency as shown in Table 1. Table 1: Ethernet Rates vs. SONET/SDH Rates
Ethernet Rate
10Mbps Ethernet 100Mbps FE 1Gbps GE

SONE T Rate
STS-1 STS-3c STS-48c

SDH Rate
VC-3 VC-4 VC-4-16c

Effective Payload Rate

~48.4Mbps ~150Mbps ~2.4Gbps

Bandwidth Efficiency
21% 67% 42%

3.2 A Brief Introduction on Next Generation SONET/SDH Next Generation SONET/SDH intends to address the challenges legacy SONET/SDH faces by introducing the following three new technologies.

3.2.1 Virtual Concatenation (VCAT)

By virtual concatenation, data traffic is carried in a number of parallel small-payload containers (called Synchronous Payload Envelope or SPE) that are individually transported through the network and reassembled at their destination. In other words, virtual concatenation allows arbitrary channel concatenation instead of contiguous channel concatenation where each channel within a Virtual Concatenation Group (VCG) may be on different paths. It allows flexible bandwidth allocation by choosing the right size and right number of payload containers. VCAT supports both low-order container at VT-1.5 and high-order containers starting from STS-1. Figure 2 shows three individual STS-3c channels allocated to carry traffic between two GE interfaces. The traffic is byte interleaved into these three channels, thus allowing an overall throughput of ~450Mbps. VCAT functions similarly to Inverse Multiplexing. Figure 2: Virtual Concatenation
GE
Byte Interleaving

VC Channel
GE

STS-3c-3v

Figure 3 illustrates that SONET/SDH payloads belonging to the same VCG (or sometimes called VCAT Channel) may traverse the network over different paths. As such, virtually concatenated sub7 of 23

Migration from SONET/SDH to Optical Ethernet in Metro channels may incur varied delays and frame buffers are needed at the destination to align the frames from different sub-channels in a VCG. The frame buffers may introduce more latency and jitter, and their implementation constitutes one of the major challenges to system vendors as discussed in Appendix A-2. It is also important to note that the individual payloads in a VCG must be the same size. Figure 3: Discrete SONET/SDH payloads within one VCG traveling on different paths
D SONET Rings

Customer Headquarter A

DCS

DCS B

Customer Remote Office

DCS

DCS

SONET Rings

Enterprise to WAN Box (GE) 200Mbps distributed over STS-1-4v Two of the four STS-1 travel this way The other two travel this way

In order to assembly correctly the data received from different sub-channels at the destination, VCAT utilizes the H4 byte of POH to indicate the sending sequence and the path number a frame belongs to. The sending sequence is denoted as MFI (Multi-Frame Indicator) and the path number is denoted as SQ (Sequence Number). At the destination, MFI is used to align the frames from sub-channels that experience varied delays and SQ is used to decide the order of taking the data from the sub-channels. In theory, VCAT significantly improves the efficiency of bandwidth allocation of SONET/SDH to Ethernet. Table 2 provides examples of how VCAT is used to support full-speed Ethernet and the associated efficiency. Table 2: Ethernet Rates Mapped to SONET/SDH Using VCAT
Ethernet Rate
10Mbps Ethernet 10Mbps Ethernet 100Mbps FE 1Gbps GE 1Gbps GE

SONE T Rate
VT-1.5-7 v VT-2-5 v STS-1-2v STS-1-21v STS-3c-7v

SDH Rate
VC-11-7v VC-12-5v VC-3-2v VC-3-21v VC-4-7v

Effective Payload Rate

~11.2Mbps ~10.88Mbps ~96.77Mbps ~1.02Gbps ~1.05Gbps

Bandwidth Efficiency
89% 92% 100% 98% 95%

8 of 23

Migration from SONET/SDH to Optical Ethernet in Metro

3.2.2 Generic Framing Procedure (GFP)

GFP is an ITU-T standard that provides a framing procedure with low overhead and support for both packet and storage services. Two types of GFP are defined. One is Frame-based GFP or GFP-F, the other is Transparent GFP or GFP-T. In GFP-F, an entire client data frame is received and mapped into one GFP frame without the overhead associated with the line coding and inter-packet gap of the client data. In GFP-T, all the client characters that are line-coded in 8B/10B by physical layer are decoded first and trans-coded to 64B/65B blocks which are then mapped into a GFP frame in pre-determined length, and transmitted immediately without waiting for the entire client data frame. Ethernet traffic is generally mapped to GFP frame by GFP-F to achieve better transmission efficiency and because only 8B/10B line coding is currently supported by GFP-T while 10Mbps and 100Mbps Ethernet use other encoding schemes. Storage traffic of Fiber Channel and FICON/ESCON use GFP-T for low transmission latency. Please note that Gigabit Ethernet can also be mapped by GFP-T because it uses 8B/10B line coding. The decision to use GFP-F or GFP-T is based on the efficiency and latency.

3.2.3 Link Capacity Adjustment Scheme (LCAS)

LCAS is an ITU-standard signaling scheme that allows two endpoints of a VC channel to tune the bandwidth dynamically at the request of the network management system without disturbing traffic. VCAT can be used without LCAS, however LCAS requires VCAT support. This is because the LCAS control commands share H4 bytes with VCAT MFI and SQ, and LCAS needs to assign SQ to the newly added sub-channel and sets other control commands such as EOS (End of Sequence) or NORM (Normal Operating Mode) in H4 bytes of POH. A likely misunderstanding of LCAS is that LCAS allows SONET/SDH to tune a VCAT channel automatically according to the traffic rate. But in fact the network management system needs to send a command to the source node for adding/deleting a sub-channel to/from the existing VCAT channel. The source node uses LCAS commands to notify the destination node of the addition/deletion of the sub-channel.

3.2.4 A Typical NG SONET/SDH Sub-system

The above three elements, VCAT, GFP and LCAS, constitute the essential building blocks of a NG SONET/SDH system. Figure 4 shows a typical architecture of such a system or subsystem. VCAT, GFP and LCAS introduce complexity and additional cost to a SONET/SDH system/network. Though it is possible for a single chip to include all three components or even the Ethernet MAC controller, in reality the chips available today only provide some of the components, for example, GFP without VCAT. Many of them do not yet support LCAS. Cost is the limiting factor for not providing

9 of 23

Migration from SONET/SDH to Optical Ethernet in Metro all the components. It would be more cost effective to customize the functionalities according to the applications. Figure 4: A typical Architecture of NG SONET/SDH Subsystem
RAM
SPI-3/ SPI-4

PPP RPR FC

SONET /SDH

CDR SerDes

SONET/ SDH Framer

VCAT Block

GFP Mapper

FE/GE MAC

MII/GMII

Ethernet

LCAS

4. What Is Carrier-class Optical Ethernet

Carrier-class Ethernet is a data-oriented technology that is going to play an important role in future Metro networks. This technology addresses four major challenges that enterprise-class Ethernet faces. 4.1 Four Major Enhancements for Ethernet to Become Carrier-class

4.1.1 End-to-End QoS

Carrier-class optical Ethernet enables service providers to deliver CIR (Committed Information Rate) and PIR (Peak Information rate) to each traffic flow that is classified physically (based on interface, for example) or logically (based on customer VLAN or type of application as an example) and guarantees the least latency and jitter for delay-sensitive traffic. This level of QoS is close to that of SDH/SONET private line and has better support for data traffic because of PIR which allows subscribers to burst their traffic at the rate as high as wire speed. The granularity of CIR and PIR is 1Mbps or as low as 64Kbps depending on the traffic policing capability of the Ethernet device. Carrier-class optical Ethernet also has an effective way of handling congestion in the network in order to maintain CIR for traffic flow under congestion. Further more, it uses MPLS to achieve better traffic engineering.

4.1.2 Sub-50ms Protection

Carrier-class optical Ethernet achieves sub-50ms protection by implementing MPLS Fast Reroute in hardware and it does not involve any software for the convergence of the network. Another advantage of this protection mechanism is that it works in any topology, not just on a ring. Spanning Tree Protocol or routing protocols, such as OSPF, involve software implementations and their convergence time is far more than 50ms and is not deterministic.

10 of 23

Migration from SONET/SDH to Optical Ethernet in Metro

4.1.3 Ethernet OAM

The original Ethernet had no OAM capabilities. This is acceptable for a LAN but not for a MAN that spans a large area and supports a large number of users. In a MAN, trouble-shooting is more difficult and OAM becomes a necessity. There has been a lot of work on defining Ethernet OAM in the IEEE 802.3 Working Group and Metro Ethernet Forum. Some vendors have already implemented prestandard OAM functions into their products such as Ethernet loop-back, BER detection, SLA Measurements, and alarms for critical problems.

4.1.4 Scalability
Enterprise-class Ethernet has intrinsic limitations on scalability when used as a public network. These limitations include the number of VLANs per network, the number of MAC addresses that need to be learned and stored in the device, and the long and non-deterministic convergence time of Spanning Tree Protocol, that is associated with the number of network elements, the number of VLANs in each element and the complexity of the network topology. The usage of MPLS by optical Ethernet with the right system architecture enables operators to address the scalability of the network and that of the services such as E-LAN. 4.2 Network Management and Service Provisioning Carriers are concerned about the network management of Ethernet network and how Ethernet services are provisioned. In fact, many carriers are still using inventory-based service provisioning, i.e., the network planners use a spreadsheet or database to maintain information on the channels, used or unused, between any two nodes as well as plan the path and allocate a channel. Operators can then log into each network element and plan and configure the channel manually. It would be a nightmare if carriers were required to manage an Ethernet network and provision services in the same way. Atrica provides a point-and-click service provisioning system called ASPEN (Atrica Service Provisioning for Ethernet Network) which enables the operators to establish an E-Line or E-LAN service in just seconds. The operator simply needs to point-and-click on the endpoints of an E-Line or E-LAN service on the management station screen and set the SLA parameters. The provisioning system can determine the best path in the network to meet the SLA requirement of an E-Line or ELAN service, and then it will establish the service automatically. The service provisioning system also maintains inventories for network elements, physical links, bandwidth allocated and still available on each link, subscribers names, locations and their services, as well as the status of each service. The system also displays the path of a service in an easy-to-understand diagram. The operator can explicitly choose the path on the screen and force the service to pass through this path, provided there is enough bandwidth for the service (either E-Line or E-LAN) on that path. ASPEN also provides OSS interface and the integration with billing system, fault management, SLA management and 3rd party network management.

11 of 23

Migration from SONET/SDH to Optical Ethernet in Metro 4.3 Support for More Services

4.3.1 E-LAN Service

E-LAN is a unique service of optical Ethernet, utilizing the Layer 2 switching capability of Ethernet to interconnect multiple sites transparently. In other technologies such as SDH or ATM, multiple pointto-point channels or virtual connections are needed among those sites, which may lead to scalability issues. Carriers need to address three issues on E-LAN service, scalability, QoS and protection. As far as scalability is concerned, Atricas solution for E-LAN can support 256 or even more sites per E-LAN connection, and each core device supports 3.84 million MAC addresses. The E-LAN service also has the same SLA parameters as E-Line service including CIR, PIR and sub-50ms protection.

4.3.2 Circuit Emulation Service (CES)

CES is a technology that emulates TDM circuit over Ethernet. CES can be used to interconnect PBX and existing SDH/SONET networks over Ethernet, or provide TDM-based private line to interconnect routers with legacy interfaces. Atricas CES implementation is based on IETF PWE3 Working Group draft standard.

4.3.3 Service Multiplexing

Optical Ethernet supports service multiplexing by taking advantage of 802.1Q VLAN tagging. For example, a single GE interface in a router can support up to 4096 remote sites for VPN or Internet access. This solution takes advantage of a feature called VLAN Translation that allows the remote sites to maintain their VLAN tags when they share a single router interface; but at ISPs PoP their VLAN tags will be translated to a new VLAN tag so that the ISP router can distinguish traffic from different sites. VLAN Translation is described in the MEF white paper Metro Ethernet Service. 4.4 The Cost-effectiveness of Optical Ethernet Looking back at the history, technically Ethernet has never appeared to be the best technology, even in Local Area Networks, but it beat all the other technologies such as Token Ring, FDDI and ATM and it currently dominates at least 95% of LAN market. The major reasons for Ethernets success are lowcost and ease-of-use. These reasons still remain valid for Ethernet in the metropolitan area. Figure 5 shows the price per Mbps of various metro technologies. A pure Ethernet solution has the lowest cost with a large difference over the other technologies. The figure also illustrates why it is reasonable for some Ethernet carriers to push a movement called one dollar per Mbps. For example, the price of an Ethernet aggregation or access device with 24 Fast Ethernet ports and two Gigabit Ethernet uplinks may be around $2,400 (not including the price of optical transceivers, if

12 of 23

Migration from SONET/SDH to Optical Ethernet in Metro needed). With the emerging optical technologies, optical Ethernet can reduce the CapEx per subscriber to about $1,000, to-date an unmatched price by any other technology. Figure 5: Price per Mbps of Various Ethernet Technologies (Source: RHK, March 2002)

Many agree that optical Ethernet is the right solution for greenfield. But should a carrier with a large SONET/SDH installed base deploy optical Ethernet in metro? To answer this question, more aspects, both economic and technical, need to be revealed.

5. Metro Ethernet Services The Battle Field for Optical Ethernet and NG SONET/SDH
Many carriers, including the incumbents, have started providing Ethernet services in the metro area such as extended to the national and international domains. To shorten the time to market and leverage their existing infrastructures, most of the incumbent carriers use Ethernet over SONET/SDH with or without VCAT and GFP, and are now facing fierce competition from the competitive carriers using pure optical Ethernet solutions. 5.1 Economic Analysis To better understand how optical Ethernet competes with NG SONET/SDH, we can study a simple but real business case.

5.1.1 Capex
Figure 6 shows the cost of building a network for Ethernet services using NG SONET and optical Ethernet based on the average market price. The fact is that the cost ratio between NG SONET and optical Ethernet, on a per subscriber basis, is about 6:1.

13 of 23

Migration from SONET/SDH to Optical Ethernet in Metro Figure 6(a): NG SONET

DCS

Figure 6(b): Optical Ethernet

Core

OC-48 1 x OC-48 16 x OC-3 $38,000

OC -3

GE 2 x GE 24 x FE $9,000

DCS

Aggregator

OC-3

3 CO

FE

FE

FE

1 x OC-3 2 x FE $3,500

ADM

ADM

ADM

Demarc

Demarc

Demarc

1 x FE 1 x FE $500

5.1.2 Service Price

For example if an enterprise is using an E1 leased line from service provider A to interconnect two offices in the same city and it may pay $1,600 per month. Now service provider B comes and offers a new service based on optical Ethernet. The price tag may look like this: For each Mbps of CIR, the price is $300 per month. For each Mbps of PIR, the price is $10 per month. PIR is not guaranteed but the customer has good chance to burst its traffic at this rate because in the network the utilization of the bandwidth is well below 50% even with full CIR subscriptions due to the fact that other traffic is also bursty. So the enterprise decides to buy 2Mbps CIR and 100Mbps PIR from service provider B and pays (2 x 300) + (100 x 10) = $1,600 per month. The customer pays the same amount but gets more bandwidth up to 100Mbps. In this case, the enterprise is guaranteed 2Mbps and sub-50ms protection, just as before. Service provider B takes advantage of two things: the low-cost of optical Ethernet so that CIR is cheaper than E1 leased line, and statistical multiplexing of Ethernet that allows a traffic flow to take as much bandwidth as it can (up to PIR). Even with the lower service price, service provider B can claim a significant profit margin both from CIR and PIR. On the other side, service provider A has to allocate an STS-1-2v channel in order to support 100Mbps. This is far more expensive than that of service provider B, thus service provider A has a much lower profit margin. Point-to-point Ethernet service or E-Line as defined by MEF is the basis for many other value added metro services such as Internet Access, in which E-Line connects business subscribers to their ISP, IP VPN, in which E-Line connects CE routers to PE routers, and SAN Outsourcing, in which E-Line

14 of 23

Migration from SONET/SDH to Optical Ethernet in Metro connects enterprises to their Storage Service Provider. In DSL backhaul network, E-Line connects Ethernet-based or IP DSLAM to BRAS.

5.1.3 The Cost of Upgrading Existing SONET/SDH Rings

Many SONET/SDH rings in metropolitan areas are OC-3/STM-1 and OC-12/STM-4 that can only support a very limited number of Fast Ethernet service ports. As such, these rings need to be upgraded to a higher rate such as OC-48/STM-16 which appears to be becoming the most popular rate in metro. Carriers must pay special attention to the cost of this upgrade because in many cases the upgrade of the access rings also requires the upgrade of the core rings and the overall expenditure on the upgrade may be more than the cost of building a new data-centric optical Ethernet network. 5.2 Technical Limitations of NG SONET/SDH in Supporting Data Services Many carriers with a large installed base of SONET/SDH believe that they can offer Ethernet services whenever and wherever the demand originates. This service is nothing more than a SONET/SDH channel with Ethernet access interfaces to subscribers. With VCAT, carriers can allocate the right sized bandwidth according to customers requirements. Plus, GFP-F allows traffic multiplexing from several Ethernet interfaces into one VCAT channel, a feature similar to statistical multiplexing, and GFP-T supports other services like FICON and ESCON. All this sounds very promising indeed, but carriers would want to understand the limitations of NG SONET/SDH in order to position their offerings properly.

5.2.1 The Efficiency of Data Transmission Is Not Improved

Data traffic is bursty and peak rates of multiple traffic flows usually do not occur at the same time, so it is very possible for these traffic flows to utilize as much bandwidth as needed and travel through the network in minimum time. NG SONET/SDH can only improve the utilization of the bandwidth by allocating the channels with finer granularity and more flexibility. It does not improve the efficiency of data transmission. Very often, the channel size is selected for the peak rate that a customer requests. This makes NG SONET/SDH significantly different from optical Ethernet. For example, an OC-48/STM-16 ring can support up to two GE connections with the help of VCAT. But Ethernet metro carriers feel very comfortable using a single GE ring to support two GE customers because they know that the heaviest traffic from both customers is rarely simultaneous, and both customers will have a maximum rate of roughly 1Gbps. Even when bursty traffic does occur simultaneously, Ethernet can use flow control and traffic shaping functions to avoid packet loss. Apparently a GE ring is much cheaper than an OC48/STM-16 ring.

5.2.2 GFP Multiplexing Has A Critical Limitation

Taking advantage of GFP, NG SONET/SDH may support statistical multiplexing on a per channel basis. This is often referred as GFP multiplexing illustrated by Figure 7. 15 of 23

Migration from SONET/SDH to Optical Ethernet in Metro Figure 7: GFP Frame Multiplexing

2 1

GFP Mapper

GFP DeMapper

2 1

Source node
Encapsulated client frame
1

Sink node
Channel ID

GFP Frame

Within GFP Extension Header, which is optional in GFP frame, there is an 1-byte field called Channel ID or CID. The sink node can use CID to identify the destination Ethernet interface, so it is possible for multiple Ethernet interfaces at the source node to share the same virtually concatenated channel. The effect is similar in GFP-F mode as to statistical multiplexing over that channel. In GFP-T mode the traffic that flows from each interface of the source node still maintains a steady rate (i.e., when no data come from an interface of the source node, GFP mapper will insert dummy 64B/65B control codes into GFP frames for that interface), so the effect is not statistical multiplexing but simply flow multiplexing. GFP multiplexing has a critical drawback. The traffic from the interfaces at the source node that share a VC channel must go to the same sink node. This means that only when several customers are in the same building and their traffic flows to the same destination can GFP multiplexing be utilized. This restriction makes GFP multiplexing of little value to E-Line services in production networks.

5.2.3 The Limitations in Multi-point Service and Service Multiplexing

Multi-point-to-multi-point services require Layer 2 switching within the carriers network. SONET/SDH technology, as a pure transport and point-to-point technology, is not well suited to support multi-point services unless a full mesh of VC channels are established among the multiple endpoints. This is of course not a scalable or cost effective solution. Some carriers provide point-to-multipoint Ethernet service in a hub-and-spoke manner. This service relates to service multiplexing as seen in Figure 1. In fact, this is not a new service to SONET/SDH. The channelized PoS interface in routers is just for this purpose. Unfortunately the channelized SONET/SDH interface card with GFP support in routers is not available today. The major limitation of multi-point service, in SONET/SDH environment, is the very small number of spoke sites supported by the hub site. Today, most of the NG SONET/SDH based MSPP support 6 to 24 remote sites by one hub site, while optical Ethernet can support over 10-times the number of spoke sites. As in SONET/SDH hub-and-spoke is built by multiple point-to-point VC channels taking into account the peak rate by each site and it has very low bandwidth utilization.

16 of 23

Migration from SONET/SDH to Optical Ethernet in Metro Hub-and-spoke can only address some customers needs. If a large amount of traffic needs to flow among spoke sites then hub-and-spoke is not the right solution.

5.2.4 The Inflexibility of Bandwidth Change

One restriction on setting up a SONET/SDH circuit, virtually concatenated or not, is that the subchannels that form that circuit must be the same size and follow the rule as shown in Table 3. Table 3: The Rule of Bandwidth Allocation of VCAT Order Low order High order Bandwidth Range 1.5Mbps - 100Mbps 100Mbps - 10Gbps Minimum subchannel VT1.5 STS-1 Maximum number of subchannels 64 256 Bandwidth Granularity ~1.5Mbps ~51Mbps

If a customer starts with less than 100Mbps (30Mbps using VT-1.5-20v, for example) but wants to upgrade bandwidth to more than 100Mbps (say 150Mbps using STS-1-3v), service would be disturbed because the operator of that network would be required to recreate the route, by deleting the low order circuit and reestablishing a high order circuit. At the high order circuit, the bandwidth increment is not as smooth as at the low order. Low order and high order sub-channels cannot be mixed to form a VCAT channel. Optical Ethernet simply does not have any of these restrictions.

6. The Positioning and Integration of NG SONET/SDH and Optical Ethernet

We have seen the advantages of optical Ethernet and the limitations of NG SONET/SDH in offering data services. Few would disagree that in a greenfield the best solution for Ethernet services or even data service in general is optical Ethernet. Some carriers have already deployed optical Ethernet and the benefits are obvious. In fact, one of Atricas customers achieved positive EBITDA in 6 months and ROI in only 12 months. Many carriers with a large SONET/SDH installed base have already realized the benefits of optical Ethernet. Specifically the low-cost of optical Ethernet may help improve their profitability significantly. But it is neither practical nor economical to rebuild a whole new and separate network. The best solution is to leverage the existing SONET/SDH and plug in optical Ethernet smoothly and seamlessly. The two technologies should be positioned at the right place and for the right services. The management integration and end-to-end service provisioning should be part of the migration strategy. Fortunately NG SONET/SDH can help to make the smooth migration and integration happen.

17 of 23

Migration from SONET/SDH to Optical Ethernet in Metro 6.1 The Positioning of Existing SONET/SDH Although NG SONET/SDH enables carriers to enter the Ethernet service market in a short time, as illustrated earlier in this paper, SONET/SDH is not cost-effective and bandwidth-efficient for Ethernet services in the metropolitan area and thus is not a viable long-term solution. In addition, most of the bandwidth today is in the long-haul networks where DWDM is widely used. In most of the metros, carriers may have many dark fibers but the bandwidth ready for Ethernet services is not abundant. Usually carriers will only light up the fibers when they have enough demand from customers. Moreover, the bandwidth in SONET/SDH can be fully utilized only when the traffic flows at a continuous and steady rate. This happens when enough traffic is accumulated at the aggregation locations. So while SONET/SDH continues to support voice services and interconnect backbone routers in the long-hual networks, carriers can also use SONET/SDH to interconnect multiple metro networks. With VCAT and GFP the connection of multiple metro networks can be created and maintained easily and effectively. 6.2 How Can Optical Ethernet Plug In Smoothly and Seamlessly Carriers can start deploying optical Ethernet in the greenfield where they have dark fibers and when they want to connect new customers. E-Line service, as an example in an enterprise, can connect two offices in the same city or as well as in different cities. This can be done in two situations: i) one office already has a SONET/SDH multiplexer but the other office has nothing except dark fiber, ii) both offices have only dark fibers. Figure 8 illustrates this scenario.

6.2.1 Optical Ethernet on Only One Side

In the greenfield, the carrier uses an optical Gigabit Ethernet ring with a number of Ethernet access devices to connect the new customers. The maximum number of the devices on the same ring is 32. The optical GE ring is then connected to the GE interfaces of the existing digital cross-connect. This is a very cost effective solution for greenfield. One benefit the carrier gets from NG SONET/SDH is that it can connect optical Ethernet with existing SONET/SDH more seamlessly. If the existing digital cross-connect or MSPP has Gigabit Ethernet interface card, optical Ethernet can be connected directly to the digital cross-connect or MSPP. Otherwise, an intermediate box (not shown in Figure 8) with GE and OC-48/STM-16 interfaces can be used in between the optical Ethernet and the existing SONET/SDH network. Since 2002, tens of vendors have been selling this kind of box.

18 of 23

Migration from SONET/SDH to Optical Ethernet in Metro Figure 8. Optical Ethernet on Only One Side
Greenfield Installed Base
GE GE DCS GE T1/E1 DCS DCS
OC-3/STM-1
M-4 2/ST OC-1

ADM

T1/E1

Atrica GE Ring

OC-48/STM-16
OC3/ ST M 1

DCS FE FE

Atricas optical Ethernet access device A-2100 is a key component in this scenario. A-2100 is just 1U high and supports all the carrier-class features such as stringent QoS, sub-50ms protection, Ethernet OAM, various interfaces and media on the subscriber side, including Fast Ethernet and Gigabit Ethernet, over UTP, multi-mode fiber or single-mode fiber, as well as T1/E1 and OC-3/STM-1 CES (Circuit Emulation Service) interfaces. It can be dedicated to a single business subscriber or shared by multiple subscribers. The device has two GE uplinks that support the distance between two adjacent devices from 10km to 120km depending on the pluggable optical transceivers used. The GE access ring in fact gives 2Gbps overall bandwidth because traffic can be sent in two directions (clockwise and counter-clockwise) each of which has 1Gpbs throughput. Spanning Tree Protocol is not used for protection and loop prevention. Atrica has developed a sub-50ms protection mechanism for the GE ring using the same concept of MPLS Fast Reroute. While one office in the greenfield is connected by the GE ring, the other office needs to use NG SONET/SDH ADM that supports OC-3/STM-1 or OC-12/STM-4 uplink and FE or GE access interfaces. Now the two offices can communicate to each other. Is TDM-based application such as PBX interconnection still possible for this customer? The answer is yes, because A-2100 supports Circuit Emulation over Ethernet and it provides T1/E1 and OC-3/STM1 interface on the subscriber side. As shown in Figure 8, the customer can use T1/E1 interface to connect its PBX together.

6.2.2 Both Sides Are Greenfields

The solution for this scenario is quite simple, because the carrier just needs to replicate what it has done in the greenfield, as in Figure 8. Figure 9 shows the solution.

19 of 23

Migration from SONET/SDH to Optical Ethernet in Metro Figure 9: Both Sides Are Optical Ethernet
Greenfield Installed Base Greenfield
GE GE

DCS GE DCS DCS

OC-3/STM-1

GE

T1/E1

T1/E1

Atrica GE Ring

OC-48/STM-16
OC-3/STM-1

Atrica GE Ring

DCS FE

FE

6.2.3 Scaling The Optical Ethernet Metro Networks

When the carrier gets more and more customers in the greenfields, it may need to scale the optical Ethernet networks. Figure10 shows how the networks are scaled. Figure 10: Scaling optical Metro networks in Greenfields
Greenfield
GE

Installed Base

Greenfield
GE

DCS T1/E1 GE A-8100 DCS OC-48/STM-16 DCS GE A-8100 T1/E1

DCS FE

FE

One or more optical Ethernet core devices A-8100 is added to each greenfield so that a large number of GE access rings can be interconnected by the core devices in each greenfield. A-8100 is connected to the digital cross-connect or MSPP via GE interface, or 10GE WAN PHY in the future if the SONET/SDH ring between greenfields is OC-192/STM-64.

6.2.4 The Integration of Network Management and Service Provisioning

The Integration of Network Management and Service Provisioning is a critical step towards the full integration of optical Ethernet with SONET/SDH and may need tremendous effort. Fortunately Atrica provides OSS interface in ASPEN that enables carriers to easily integrate ASPEN into their existing OSS systems.

20 of 23

Migration from SONET/SDH to Optical Ethernet in Metro Initially when the optical Ethernet network is still not too big, carriers can provision E-Line or E-LAN service by only two steps. The first step is to provision one or more SONET/SDH VC channels that function as tunnels or virtual GE links between Ethernet clouds. Then E-Line or E-LAN connections are provisioned end-to-end from ASPEN. At this moment, OSS may just need to integrate the element management and fault management and be the single user interface to the operators. Eventually when optical Ethernet network grows, operators may want to manage the services (i.e., connections) from end-to-end by OSS. See the example in Figure 11. Figure 11: The End-to-End Service Management by OSS

End-to-End Service Management

ASPEN

OSS SONET/S DH Provisioning

Ethernet Service Provisioning

Greenfield
GE

Installed Base

Greenfield
GE

OIF UNI
T1/E1 DCS A-8100

OIF UNI
OC-48/STM-16
DCS A-8100 T1/E1

VC channels

FE

GE interface mapped to VC channel

FE

In this example, OIF UNI is used between optical Ethernet and SONET/SDH to signal the establishment of a VC channel within the OC-48/STM-16 circuit already in place and the mapping of a GE interface to that VC channel. Each VC channel is a tunnel or a virtual GE link between the two A8100s and may contain a certain number of E-Line or E-LAN connections established by ASPEN. ASPEN will provide service ID to the OSS for path correlation so that the operators can view each individual E-Line or E-LAN connection end-to-end. ASPEN can also calculate the bandwidth information on each VC channel allocated for a GE interface thus avoiding oversubscribing CIR on that GE interface.

21 of 23

Migration from SONET/SDH to Optical Ethernet in Metro

7. Summary
Carrier-class optical Ethernet is an ideal solution for metro networks. It is a much cheaper solution than SONET/SDH, a simple fact that holds true for now and the foreseeable future. It has all the functions needed for many types of metro services, it perfectly supports data traffic, it meets the requirements of packetized voice traffic, and it is easy to use. On the other hand, the existing SONET/SDH networks should continue to play its critical role in voice services and long-hual connections. The current enhancement of virtual concatenation, GFP and LCAS to SONET/SDH allows carriers to integrate optical Ethernet with the existing SONET/SDH smoothly and seamlessly so that they are able to offer Ethernet services in the most effective way, thus increasing their profitability and competition strength.

22 of 23

Migration from SONET/SDH to Optical Ethernet in Metro

Appendix A: More Details on The Limitations of Virtual Concatenation

A-1 The Theoretical Limitations There are two theoretical limitations in Virtual Concatenation. One is the maximum number of member channels per Virtual Concatenation Group (VCG), which is determined by Sequence Number (SQ) embedded in SONET/SDH multi-frame Path Overhead (the H4 byte). For high order path (STS-1/VC-3 and STS-3/VC-4), SQ has 8 bits and thus defines maximum 256 members for each VCG. For low order path (VT-1.5/VC-11 to VT-6), SQ has 6 bits and thus defines up to 64 members for each VCG. Another one is the maximum differentiated path delay, which is determined by Multiframe Indicator (MFI) also embedded in SONET/SDH multi-frame Path Overhead. MFI, both for high order path and for low order path, allows maximum 256ms differentiated delay across all the members of a VCG. These two theoretical limitations do not constitute practical concerns. The real concerns to carriers and service providers are from the implementations of virtual concatenation mapper chip (VC Mappers) that is the key part of NG SONET/SDH device. A-2 The Limitations of Virtual Concatenation in Reality Due to the technical difficulties of integrating many buffers on the VC mapper chip, the differentiated path delay provided by the chip itself is small, typically around +/-25ms or less. This leaves a big challenge to system vendors to support more general differentiated path delay by using external memories. As the transfer rate of the external memory has to be fast enough, only SRAM can be used. Compared with DRAM and SDRAM, SRAM has less capacity and is more expensive. Carriers and service providers need to pay special attention to the differentiated path delay in their networks when deploying NG SONET/SDH.

Copyright Protected, 2003 Atrica, Inc. All rights reserved.

23 of 23