Technical White Paper For Netstream: Huawei Technologies Co., LTD

Technical White Paper for NetStream
Huawei Technologies Co., Ltd.
Table of Contents
1 Introduction .......................................................................................................................... 1 2 Technical Overview .............................................................................................................. 2 3 Key Technique ..................................................................................................................... 3
3.1 3.2 3.3 3.4 Packet Format of NetStream............................................................................................. 3 Three Methods of Stream Output of NetStream ............................................................... 6 Collection and Output of NetStream Statistics Information............................................... 8 Capability of NetStream .................................................................................................. 10
4 Typical Applications............................................................................................................ 12
4.1 4.2 4.3 4.4 4.5 Deployment of NDE......................................................................................................... 12 Security Monitoring.......................................................................................................... 12 AS Domain Planning ....................................................................................................... 13 Multicast Traffic Measurement and Planning .................................................................. 14 Traffic-based Settlement between ISPs.......................................................................... 15
5 Conclusion ......................................................................................................................... 15 Appendix A Abbreviations and Acronyms................................................................................ 16
Copyright 2007 Huawei Technologies Co., Ltd. All Rights Reserved. http://datacomm.huawei.com

Abstract: As a measurement and release technique based on network stream information, NetStream can categorize and measure the traffic on the network and the utilization of resources. It performs management and charging for various services and based on different QoS. NetStream is composed of the NetStream Data Exporter (NDE), NetStream Collector (NSC), and NetStream Data Analyzer (NDA). The NDE is responsible for traffic collection and transmission. The NSC is responsible for collecting and storing the traffic statistics from the NDE. The NDA analyzes the statistics to provide the basis for network charging, network planning, network monitoring, application monitoring and analysis. This document focuses on the NDE, while the NSC and NDA are covered in detail in another document. Key Words: Stream, Sampling, Aggregation, Traffic collection, and Traffic sending
1 Introduction
The rapid development of Internet offers users with larger bandwidth and predictable QoS. On the part of the users, they need more careful network management and charging. Therefore, there must be an appropriate technique to support such needs. NetStream is just such a measurement and release technique based on network stream information. It can categorize and measure the traffic on the network and the utilization of resources, and it performs management and charging for various services and based on different QoS. Thus, the following applications are provided: ChargingNetStream provides accurate data for charging based on resources (for example, line, bandwidth, and time segment) utilization. These data include IP addresses, packets, bytes, time, TOS and application types. Internet service providers can use such information to enforce flexible charging strategies, for example, based on time, bandwidth, application, and QoS. Enterprise customers can use such information to calculate the expenses of each department or amortize the costs, for more efficient use of resources. Network planning and analysisNetStream can provide advanced network
management tools with key information, to achieve the best network performance and reliability at the lowest operation cost by optimized network design and planning.
Network monitoringNetStream can provide nearly real-time network monitoring. RMON, RMON-2 and stream-based analysis techniques can be used to visually represent the traffic model of a single router and the whole network, and provide proactive fault detection, efficient troubleshooting and rapid problem solution. Application monitoring and analysisWith NetStream, detailed network application information can be obtained. For example, the network administrator can view the percentages of the traffic occupied by Web, FTP, Telnet and other well-known TCP/IP applications. Internet contents and service providers can plan and allocate the network resources based on such information to meet the users needs. User monitoring and analysisNetStream enables the network operator to obtain the detailed information regarding the users utilization of the network and application resources, and uses such information to effectively plan and allocate resources and guarantee safe operation of the network.
2 Technical Overview
NetStream is based on stream. A stream is composed of the packets from the same sub-interface, with the same source and destination IP address, protocol type, and same source and destination protocol port, and same ToS (usually referred to as the quintuple). NetStream will record the statistics of a stream, including the time stamp, packets, and bytes. NetStream is composed of the NDE, NSC, and NDA. The relationships among them are shown in the following diagram.
Figure 1 Roles of the components of NetStream The NDE is responsible for traffic collection and transmission. The NSC is responsible for collecting and storing the traffic statistics from the NDE. The NDA analyzes the statistics to
provide the basis for network charging, network planning, network monitoring, application monitoring and analysis. (1) NetStream Data Exporter It analyzes and processes the network stream, takes the stream statistics that meets the specified conditions, and outputs it to the NDC. Before the output, the NDE also perform some processing to the data, for example, aggregation. (2) NetStream Collector As a UNIX application running on Solaris, it parses the packets of the NDE and stores the statistics to the database, for further analysis by the NDA. The NSC can collect the data outputted from multiple NetStream devices, and filter and aggregate them. (3) NetStream Data Analyzer As a network traffic analyzer, it takes the statistics from the NDC for subsequent processing, and provides the basis for various services (for example, network planning and attack monitoring). With the graphical user interface, it is easy to use, allowing the users to easily obtain, display and analyze the data collected by the NSC.
3 Key Technique
3.1 Packet Format of NetStream
The statistics of the network stream collected by the NDE is encapsulated in the UDP packets for output to the NSC/NDA. One UDP packet can carry multiple statistics records. The formats of these statistics records are determined by the NDE equipment. After the NSC and NDA receive the statistics messages from the NDE, they first check the version of the records. Records of different versions have different formats. Currently, three versions of records are supported: V5, V8, and V9. The packet format is shown in the following diagram:
Figure 2 Packet format of NetStream Table 1 Comparison between three packet formats Version Format description Advantage Disadvantage A great variety of fields The format is fixed, and not are outputted. All the The packet format is fixed, possible to expand; fields of the stream and hence difficult to There are so large a volume of records before expand; data, that the NSC cannot store aggregation can be It is the original data stream them for long; outputted to the NSC; generated based on the The contents are rich; The NSC and NDA are under septet. The load of the equipment great pressure. is low. The format is fixed, and not The volume of data is possible to expand; The equipment must perform relatively small; The contents born are aggregation, which is a heavy The packet format is fixed, slightly simpler, and they load, so this version is only used and hence difficult to are suitable for specific to output the converged stream expand; analysis; information to the NSC. New aggregation modes The new aggregation mode is can be added. added, but this requires the upgrade of the host and NSC. The packet format is based It is the most flexible on the template, and hence output format, which is easy to expand; alterable; Two types of data can be It can be used to output -outputted. One is statistics the stream records both data and the other is option before and after the data. aggregation
V5
V8
V9
V5 is used to output the details of the stream to the NSC/NDA.
V8 is used to output the aggregated stream information to the NSC/NDA. The common disadvantage of V5 and V8 are: As the users have their new needs, the NDE must expand the output information. At the same time, the software of the NSC/NDA must be modified to suit the change of the NDE. However, this will cause the NSC/NDA software of different manufacturer or even the different versions of the same manufacturer to fail to parse the statistics packets of the NDE. The most noticeable difference between V9 and previous versions is that it is based on template. The template provides the flexible and expandable packet output format, which allows new stream measurement services to be added easily without changing the basic record format. Advantages of the V9 template: 1) Flexibility: It allows the needed domain statistics to be outputted separately, without outputting all the IP stream information to the NSC. This reduces the volume of stream output, and the possible memory requirements and bandwidth requirements of the NDE and NSC. 2) It allows new domains to be added in the output record without changing the output packet format. For previous versions, on the contrary, the addition of a new domain means a new version of output protocol format. To parse this new format, the NSC needs to provide new support. 3) Since the information is outputted in the old template, the NSC can still explain the stream record, even if it does not understand the real semantics of the domain added. The stream data template and stream data of V9 are shown in the following diagram:

Data template format
According to the mapping from FlowSetID to TemplateID, the attribute of the Value is parsed by Type+Length. Stream data
0?indicates the data template packet. These two must match
Figure 3 Stream data template and stream data of V9
3.2 Three Methods of Stream Output of NetStream

The NDE can output streams in one of the three methods: Stream-by-stream, sampling and aggregation. Stream-by-stream is the process to output the information of each stream to the NSC equipment. Sampling is the process to output the stream statistics information to the NSC by ratio, and the NSC will restore the information to the original statistics based on the sampling rate. The advantages of the stream-by-stream method are that the NSC can obtain the details of the stream, and can perform more flexible subsequent processing to these stream records. However, its disadvantage is as obvious in that the network bandwidth and CPU utilization are increased, enormous storage media space is required to store such information, and probably the users do not need the information in such great detail. Sampling can reduce the pressure on the storage media space, but the statistics information will suffer a certain degree of distortion. Is there is solution that solves the information explosion of the stream-by-stream method, while reducing distortion? Thus, aggregation is proposed. Aggregation is the process to
combine the statistics information of the streams with the same attributes, for example, all the packets between two autonomous domains, and the packets to the same destination. On the NDE equipment, first the streams are aggregated, and then outputted to the NSC. This way, the network bandwidth, CPU utilization and storage media space are reduced greatly. At present, eleven aggregation modes are supported. See the following table for their details.
Table 2 Ten Aggregation Modes of NetStream

Aggregation mode as Description Autonomous system aggregation: The streams are classified according to the four key values (NetStreams source/destination autonomous system number, input interface index, and output interface index), and the streams with the same key values are combined into an aggregated stream, which corresponds to an aggregation record. Autonomous system TOS aggregation: The streams are classified according to the five key values (NetStreams source/ destination autonomous system number, input interface index, output interface index, and ToS), and the streams with the same key values are combined into an aggregated stream, which corresponds to an aggregation record. Protocol-port aggregation: The streams are classified according to the three key values (NetStreams protocol number, source, destination autonomous system number), and the streams with the same key values are combined into an aggregated stream, which corresponds to an aggregation record. Protocol-port-TOS aggregation: The streams are classified according to the five key values (NetStreams protocol number, source/destination autonomous system number, ToS, input interface index, and output interface index), and the streams with the same key values are combined into an aggregated stream, which corresponds to an aggregation record. Source-prefix aggregation: The streams are classified according to the four key values (NetStreams autonomous system number, source mask length, and input interface index), and the streams with the same key values are combined into an aggregated stream, which corresponds to an aggregation record. Source-prefix-ToS aggregation: The streams are classified according to the five key values (NetStreams source autonomous system number, source mask length, source prefix, ToS, and input interface index), and the streams with the same key values are combined into an aggregated stream, which corresponds to an aggregation record. Destination-prefix aggregation: The streams are classified according to the four key values (NetStreams destination autonomous system number, destination mask length, destination prefix, and output interface index), and the streams with the same key values are combined into an aggregated stream, which corresponds to an aggregation record. Destination-prefix-ToS aggregation: The streams are classified according to the five key values (NetStreams destination autonomous system number, destination mask length, destination prefix, ToS, and output interface index), and the streams with the same key values are combined into an aggregated stream, which corresponds to an aggregation record. Prefix aggregation: The streams are classified according to the nine key values (NetStreams source/destination autonomous system number, source/destination mask length, source/destination prefix, and input/output interface index), and the streams with the same key values are combined into an aggregated stream, which corresponds to an aggregation record. Prefix-ToS aggregation: The streams are classified according to the nine key values (NetStreams source/destination autonomous system number, source/destination mask length, source/destination prefix, ToS, and input/output interface index), and the streams with the same key values are combined into an aggregated stream, which corresponds to an aggregation record. The streams are classified according to the 13 key values (4 label values of each level label, 4 corresponding EXPs of each label, 4 bottom label identifiers and remote PE IP address corresponding to the top label). The streams with the same key vlaues convergence to a MPLS aggregation stream which corresponds to one aggregation record.
as-tos
protocol-port
protocol-port-tos
source-prefix
source-prefix-tos
destination-prefix
destination-prefix-tos
prefix
prefix-tos
Mpls aggregation
3.3 Collection and Output of NetStream Statistics Information

Copyright 2007 Huawei Technologies Co., Ltd. All Rights Reserved. http://datacomm.huawei.com 8
The NDE collects and outputs the statistics information as shown in the following diagram.
Statistics information processing unit AS buffer AS-TOS buffer Statistics information taking Assembly and output unit
Sending
Forwarding
Other buffers
IP packet
Figure 4 Extraction and Output of NetStream Statistics Information The statistics information collection unit identifies the streams and outputs those that meet the pre-set conditions to the statistics information processing unit, which can aggregate the statistics information, and obsolete the stream records. The obsolete streams are assembled and outputted by the assembly and output unit. The statistics packets of the NDE can be outputted to a single NSC, or to two NSCs, which serve as mutual backup. This way, it becomes less likely for packets loss during the transmission from NDE to NSC. The NDA obtains the aggregate of the statistics information from the two NSCs, to have the relatively accurate statistics records, as shown in the following diagram.
Figure 5 Data output of NetStream

Copyright 2007 Huawei Technologies Co., Ltd. All Rights Reserved. http://datacomm.huawei.com 9
3.4 Capability of NetStream

1) Interfaces supported
The NetStream function can be enabled on the physical interfaces and sub-interfaces, to measure the traffic of the Ethernet ports (FE, GE), ATM port, and POS port. The NetStream function can also be enabled on the inbound direction of an interface bound to the VPN. 2) Information available with NetStream a) b) IPv4 information Source IPv4 address Destination IPv4 address Source prefix mask Destination prefix mask Source AS Destination AS Source interface Destination interface IP protocol Inbound interface Outbound interface ToS Time stamp of the first packet Time stamp of the last packet Next hop router IP address IPv6 information Source prefix mask Destination prefix mask Source AS Destination AS Source interface
10 http://datacomm.huawei.com
Copyright 2007 Huawei Technologies Co., Ltd. All Rights Reserved.
3)
Destination interface IP protocol Inbound interface Outbound interface ToS Time stamp of the first packet Time stamp of the last packet Next hop router IP address Source IPv6 address Destination IPv6 address IPv6 stream table VPN identifier
Traffic measurement on the inbound/outbound direction a) b) c) d) e) f) Measures the unicast traffic on the inbound direction of the interface; Measures the unicast traffic on both inbound and outbound direction of the interface; Measures the multicast traffic on the inbound direction of the interface; Measures the unicast traffic on the outbound direction of the interface; Measures the multicast traffic on the outbound direction of the interface; Measures the multicast traffic on both inbound and outbound direction of the interface;
4)
Sampling, that is, measuring the traffic of an interface by percentage a) b) Sampling based on the number of packets (the maximum ratio can be set to that one is taken among 215 packets) Supports the sampling of random number of packets
5)
Traffic sending
The traffic measurement result can be sent to the two NM servers. 6) Supports aggregation
Currently, eleven types of aggregation are supported. Each aggregation supports output in two formats (V8/V9). If the output is in the V9 packet format, separate template parameters
can be used for configuration, with double destinations supported. For details, see the table provided in the section on the three methods of NetStream stream output.
4 Typical Applications
4.1 Deployment of NDE
On the network equipment on the access layer, convergence layer and core layer of the network, the NetStream service board can be inserted to enable NetStream. Based on the requirements for the volume and direction of traffic, appropriate strategies can be taken to enable NetStream. Table 3 Strategies for Enabling NetStream on NDE Deployment position Access layer Convergence layer Core layer Application Application monitoring, attack monitoring Charging, AS Peer monitoring Traffic analysis, traffic engineering
4.2 Security Monitoring
Identifies attackers by abnormal traffic Dos, SYN Flood
Detailed records on the internal users?access of the network (IP+PORT+Traffic+Time)
Figure 6 Security monitoring The NetStream feature can be enabled at the incoming interface of the router on the access or convergence layer to have the detailed records of the users network access. Network attack and its source can be identified based on the abnormal traffic. These records can be used as the basis based on which network security strategy can be established and implemented.
4.3 AS Domain Planning
Figure 7 AS Domain Planning At the AS interface of the egress router on the corer layer, the NetStream feature is enabled, and outgoing/incoming traffic measurement is also enabled at the same time. To suit the practical needs, the stream attribute source/destination AS can be configured to Original AS or Peer AS, and the stream statistics of the NetStream can provide the data for inter-carrier settlement and network planning.
4.4 Multicast Traffic Measurement and Planning
Traffic of multicast source
Traffic of multicast to subscribers
Figure 8 Multicast traffic measurement and planning On the incoming interface of the multicast source of the multicast router, the NetStream feature can be enabled to measure the traffic of the multicast source. On the user access interface, the NetStream can be enabled to measure the traffic copied for multicast, providing the basis data for the network expansion and equipment expansion for multicast services such as IPTV, stream media/VOD.
4.5 Traffic-based Settlement between ISPs
According to the destination IP and mask, the traffic to the two ISPs can be measured through destination-prefix aggregation
Figure 9 Settlement is made between ISPs by traffic based on the IP prefix On the ISP access interface on the MAN convergence layer or backbone layer, the NetStream feature can be enabled, and the outgoing/incoming traffic measurement can be started. By the destination IP and mask and with the destination-prefix, it can measure the traffic of the access to different ISPs, providing the data for settlement between them.
5 Conclusion
Performing traffic collection and aggregation based on stream, the NetStream can provide accurate data for charging based on resources (line, bandwidth, and time segment) utilization, and provide advanced network management tools with key information, to achieve the best network performance and reliability at the lowest operation cost by optimized network design and planning. It can implement nearly real-time network
monitoring and network-wide traffic, provide proactive fault detection, efficient troubleshooting and problem solution, and provide applications and analysis such as security monitoring. NetStream will continuously push the development of the network traffic/direction analysis technology, and provide data support for the charging settlement, network planning, and network operation and maintenance of vast carriers.
Copyright 2007 Huawei Technologies Co., Ltd. All Rights Reserved. 15 http://datacomm.huawei.com
NetStream Technology White Paper
Appendix A Abbreviations and Acronyms
Abbreviation/Acronym NDE NSC NDA
Full Name NetStream Data Exporter NetStream Collector NetStream Data Analyzer
Copyright 2007 Huawei Technologies Co., Ltd. All Rights Reserved

Technical White Paper For Netstream: Huawei Technologies Co., LTD

Uploaded by

Copyright:

Available Formats

Technical White Paper For Netstream: Huawei Technologies Co., LTD

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Technical White Paper For Netstream: Huawei Technologies Co., LTD

Uploaded by

Copyright:

Available Formats

Technical White Paper for NetStream

Huawei Technologies Co., Ltd.