CAPACITY PLANNING ANALYSES OF LARGE DATA NETWORKS: A CASE STUDY

Paul Ybarra International Network Services and Tom Mee United Services Automobile Association
Forecasting the performance of a Local Area Network or Wide Area Network has been difficult due to extremely large variances in utilization. Large numbers of network devices can compound the difficulty. An aggregation technique that takes into account the historical utilization of these devices is examined, and the reliability of forecasts based on them is discussed. The accuracy of trending the highest weekly hourly averages for aggregations of devices versus individual network devices is presented. A strategy for reducing the choice of metrics to track for each resource is also discussed as an alternative to the exhaustive measurement of each resource and device.

1. Introduction For the purposes of this discussion, we assume that Capacity Planning is the activity of monitoring resource utilization and estimating the appropriate time for acquisition of additional resources. This paper investigates an aggregation technique that combines data from several network devices to create a group forecast. An aggregation methodology was sought to reduce the amount of time required to analyze thousands of network devices and tens of thousands of interfaces. Next, a simple and generic model was explored to determine whether it was scalable to a large number of devices and to compare it, in terms of accuracy, with the aggregation technique. This paper presents a model of capacity planning that may be applied to complex systems, such as large data networks. The model was developed in response to the installation of a new network at the United Services Automobile Association (USAA). (USAA is a San Antonio based insurance and diversified financial services company, whose primary customers are members of the U.S. military and their families.) 2. Network Topology In 1996, USAA installed a switched, fast ethernet network in response to traffic growth in excess of 125% per year. The USAA network consists of three main components (see Figure 1): an Internet Protocol (IP) backbone, an Advanced Peer-to-Peer Networking (APPN) backbone, and a series of user networks (Electronic Communities). Each component consists of a backbone core of Asynchronous Transport Mode

(ATM) switches. An ATM switch is a high-speed network device that transports higher layer protocols (for example, IP and APPN) with very little delay. The IP backbone is called the IP Core because it handles all the IP-related traffic to and from enterprise servers. The APPN backbone is called the APPN Core and handles all traffic to and from SNA servers. The APPN protocol allows SNA data to be routed to networks outside the core. The routers allow each of these backbones to communicate with each other. They also make decisions based on the message destination and preferred network path. User networks are called Electronic Communities (ECs). The USAA network consists of 20 ECs. As

Figure 1. USAA's three major network components consist of an IP and an APPN backbone along with 20 user networks called Electronic Communities.

with the IP and APPN cores, an EC incorporates an ATM backbone and a network of Ethernet Switches (see Figure 2). Ethernet is a data-link protocol that allows for communication over copper and fiber cables. The layer of switches connected to the users' workstations is called the Bay layer. These switches connect directly to Ethernet cards on each of the users' computer units. Note in Figure 2 that there are two levels of Ethernet switches. The cascade of switches

2.1 Measurement All data is collected using a Simple Network Management Protocol (SNMP) based collector, and then stored in a SAS IT Service Vision performance database. This database stores daily, weekly, monthly and yearly statistics on the collected data. Information is collected from switches and routers on CPU, memory, interface and port utilization. Data are collected in 15-minute intervals. More frequent collection rates are preferred; however, the centralized data collection scheme and the large number of devices prohibits a shorter collection interval. A distributed data collection system is planned and will improve this condition. 2.2 Naming Convention Key to the aggregation technique discussed in this paper is the USAA naming convention. Each device name is based on its type and precise physical location. This allows devices to be aggregated into classes, such as routers, ATM switches, Electronic Communities and so forth. 3. Approach Data is available to forecast utilization for virtually every network device and interface. Our initial approach, aggregation, was to attempt to simplify the task by creating classes of devices and interfaces and treating these as single entities. For example, Bay switches may be treated as a single class. Aggregated data for these switches forms the basis for characterization of the class. The object of capacity planning, then, shifts from the individual devices to a specific class. The goal is to monitor class utilization and forecast when the class will run out of capacity. To elaborate, let us assume that the link utilization of a Bay switch is being monitored. Assume also that each switch has two uplinks and forty-eight downlinks. The highest hourly average is determined each week for each device. This is the highest weekly hourly average (HWHA) or peak hour for the device. The peak hour for all other devices is determined and combined to produce the class average for the week. For example, using five switches, Table 1 illustrates peak hour utilizations for a single week.

Figure 2. A typical Electronic Community consists of a fully-meshed network of ATM switches connected to Distribution Switches that then connect to user-connected Ethernet switches called Bay Switches. between the Bay and the ATM core serves to increase the port density per EC. This layer of switches is called the Distribution layer. The USAA network is highly redundant and faulttolerant. There are approximately 150 ATM Switches with over 4000 ports, 1000 Ethernet Switches with over 40,000 ports, and 200 routers with over 600 interfaces. The most highly utilized devices in the network are the routers. They must handle traffic from high-speed ATM switches, make a best path decision, and then route it to the appropriate interface. For LAN routers, the router's CPU is the highest utilized resource. For WAN routers, the interface can be the most highly utilized resource. For Ethernet and ATM switches, the available number of physical ports is the most important statistic to monitor. The data collection matrix represents an enormous number of devices (1350), ports (44,600) and metrics (4 for routers, 2 for switches). However, this can be significantly reduced by tracking only router CPU and memory for LAN routers (72), router interface and memory utilization for the WAN routers (128) and port count (40,000) for Ethernet and ATM switches. The matrix is then reduced from 90,400 (44600*2 + 600 + 200*3) elements to 40,400 (72*2 + 128*2 + 40000) elements. This reduction is possible due to the extremely low utilization of the other resources in each network unit.

Week #1 Switch #1 Switch #2 Switch #3 Switch #4 Switch #5 Class Average

Uplink 8% 12% 9% 18% 8% 11%

Table 1. Utilization for Class of Bay Switches in EC01 for Week No. 1 Monday Uplink 01:00 1 Week Uplink #1 02:00 1 Switch #1 8% 03:00 1 Switch #2 12% 04:00 2 Switch #3 9% 05:00 2 Switch #4 18% 06:00 5 Switch #5 8% 07:00 5 Class 08:00 5 11% Average 09:00 6 10:00 8% Table 1: Utilization for Class of Bay Switches in 11:00 EC016for Week No. 1 12:00 7 13:00 7 14:00 7 15:00 5 16:00 5 17:00 5 18:00 4 19:00 3 20:00 3 21:00 3 22:00 2 23:00 2 24:00 2 Table 2. Hourly Utilization for Switch #1 in the Class of Bay Switches for EC01 on a Monday

Table 2 shows the utilizations for Switch #1 for an entire day. The highest daily average is 8%, as shown in bold type. If this value is greater than or equal to the daily averages for the other days, then it will be the HWHA sample used for the device. (It should be noted at this point that the peak hour may occur on any hour of any day of the week.) When sufficient data have been collected, the capacity planner may forecast trends using appropriate statistical techniques. Selection of the appropriate intervals is important, as is the decision to use class

averages or peaks. These issues are discussed in the next section. 3.1 Highest Weekly Hourly Average There are several ways to use trend data. One can use daily averages, weekly averages, monthly averages, daily peaks, weekly peaks or monthly peaks. In addition, averages of averages, averages of peaks, or peaks of averages can be used. Averaging has the effect of "smoothing out" the peaks. The larger the average time period, the smoother the peaks appear. If the time period is too large, the averaging minimizes peak influences. If the time period is too small, the averaging maximizes peak influences. Thus, averaging is a frequency filter in which peaks of short duration are removed when larger window sizes are used. The maximum allowable peak influence is site dependent based on comfort level. There may be instances when a peak value that exceeds a certain threshold is intolerable. Network traffic is inherently bursty and has many peaks within a given observation period. This burstiness implies contention for network devices and has a significant effect on the performance of these devices. The highest daily hourly average (HDHA) attempts to incorporate these influences. A measure of burstiness is defined in [MEN98] by the 2-tuple (a, b). "A" is the peak to average ratio and "b" is the percentage of the time the packet arrival rate is greater than the mean arrival rate. An HDHA is a similar measure in that a burstier hour will contribute more to the daily average than the other hourly averages. An HDHA is not the peak sample for the day but the highest hourly average for the day. Note that an HWHA will always be greater than or equal to the weekly average but less than the weekly peak. Additionally, an HWHA will also be greater than or equal to an HDHA for each day of the week. Daily averages do not vary significantly from day to day. They are easy to trend, and models can be used to accurately predict average utilization. Unfortunately, average utilization can be up to 10 times less than the peak values for the day. Peak values vary significantly from day to day and are difficult to trend. However, peaks do not vary significantly from week to week. Tracking averages is not very useful if the effects of peak influences are desired. An HWHA is a means of tracking average utilization while incorporating the effects of peaks. Averages tend to be constant; therefore, large week to week fluctuations can be attributed predominantly to an increased burstiness, which may indicate more network users or a new application. If this increase is sustained and unplanned, the effects on performance will be significant. Such events must be dealt with by determining the cause and minimizing the likelihood of a recurrence. Otherwise, accumulated events such as these could be deleterious

to user response-times, especially if they occur concurrently. Typical measurements of HWHA show weekly fluctuations of less than 10 percent. Fewer than 20% of samples fall outside this range. A significant increase outside this 10 percent range indicates an increase in peaks in either number and/or strength. This is a direct result of additional users and/or application usage. Forecasting a baseline trend for a class, however, does not tell us when the class will exceed a capacity threshold. We cannot assume that the threshold for the class is identical to the threshold for the devices that comprise the class. For example, the class average may be well below the threshold, although several devices in the class may have exceeded it. Users may experience performance degradation or service outages even when the class appears to have ample capacity. A clear sign of this event becomes readily apparent when an ordinary least squares trend is used on the peak hourly data for a few devices (Figure 3). Note that the slopes for each of these devices vary significantly from each other. Efforts to forecast the aggregate threshold data will result in a significant standard error. This unfortunate result has been observed with more than 40 weeks of data.

Figure 3. Comparison of CPU HWHA slopes for various routers. However, all is not lost. The HWHA data that was to be used to predict the utilization of a class of devices will be used to predict the utilization of each individual device instead. The processing time required to determine these thresholds is in the order of seconds. The aggregation technique presupposed that the resources of both the planner and the computing system would be severely taxed if all devices and interfaces required individual examination. A surprising result of the aggregation attempt was how the individual devices appeared to be following a well-behaved linear trend. The standard error for these devices averaged

only 8 percent from the predicted (the median was about 4). The forecasts can be determined programmatically, and the standard error for each device computed. (Figure 4 shows HWHA for several devices.) The time investment by the planner is now shortened to include only analysis of those devices expected to exceed the threshold. For USAA, the number of such devices was in the 20's. Automatically generated graphs representing actual and estimates along with expected errors and number of samples are easily and quickly made for such a small number of devices. These results were satisfying since both management and non-technical staff (such as budget planners) readily understand the results of an ordinary least squares analysis. 4. Limitations Several important notes must be made concerning the peak hour data that USAA capacity planners use to forecast. All resource metrics, such as CPU and memory, are compiled and stored using the HWHA method. This time interval represents an observational period of one week, although the data is just an hourly average. Current performance of network devices cannot be determined from this data. USAA's performance monitoring group deals with time periods significantly smaller than the weekly hourly averages that have been considered in this analysis. Performance monitoring is concerned more with highresolution network data and views it with a more microscopic lens. Capacity planners use a more telescopic lens that takes a larger picture and is less concerned with the finer-resolution, rapidly changing data. Despite this difference in analytical approach, there exists an important symbiotic relationship both in implementing the tools and in dealing with significant events between both groups. Capacity planners rely exclusively on the performance group to collect the network data. This data is then filtered and stored in a separate capacity planning database for trend and analysis. Conversely, decisions affecting the performance of the network, such as the purchase of new equipment, can have a significant effect on the performance of other parts of the network. Consequently, any plans to increase capacity of resources need to be coordinated with the performance management teams. The a posteriori analysis presented in this paper presumes that traffic grows linearly. No assumptions were made concerning marked changes in this growth pattern. This type of analysis requires keeping abreast of significant changes in software use and user populations. Y2K issues also affect planning. Thresholds are adjusted to account for these special events. 5. Port, Memory, and Interface Utilization

The results presented above for router CPU utilization are similar to those for memory and interface data-linear with a small standard error. Since port availability
Router 1 CPU
40 % 30 20 10 0 1 5 9 13 17 21 25 29 33 37 41 Actuals Estimates

Week

Router 2 CPU
70 60 50 % 40 30 20 10 0 1 5 9 13 17 21 25 29 Week 33 37 41 Actuals Estimates

Router 3 CPU
100 90 80 70 %l 60 50 40 30 20 10 0

Actuals Estimates

13 17

21

25

29

33

37 41

Week

does not vary significantly from week to week, no averaging is necessary. Forecasting of the data directly is possible while retaining a low standard error. 6. Conclusion The primary driver responsible for the growth seen in the network data is USAA's employee growth. Annualizing the slope of the curves in Figure 4 shows a yearly growth of about 5%. USAA employee growth is about 8%.

Although aggregation was not as productive as anticipated, the linear model that was discovered for individual devices using HWHA justified the effort. Implementation of the tools discussed in this paper can be realized using any database that allows external data to be imported and the highest weekly hourly average to be stored. In addition, this database should have the capability of being analyzed with automated scripts or tools. By using the HWHA technique, planners now have a tool to accurately plan network capacity. To improve this model, the number of samples per hour can be Figure 4. Highest Weekly Hourly Averages for select devices by week. Displayed are the actual data and the Least Mean Square estimates.

increased for a more accurate HWHA representation. In addition, the HWHA window can be decreased to allow higher frequency peaks to be better represented. References [MEN98] D. A. Menasce, V. A. F. Almeida, Capacity Planning For Web Performance Metric, Models, & Methods, NJ: Printice Hall, (1998), pp. 223-227.