SCADA Water Upgrade
SCADA Water Upgrade
SCADA Water Upgrade
Audit #02-13
Prepared by
Office of Inspector General
Allen Vann, Inspector General
Tim Beirnes, Lead Consulting Auditor
John Lynch, Lead Information Systems Auditor
MGT 08-06F
This audit was performed pursuant to the Inspector General’s authority set
forth in Chapter 20.055, F.S. The audit focused on assessing the
appropriateness of the selected technology for the District's SCADA
system and analyzing the reasonableness of the 10-year cost estimates
for the SCADA capital outlay and operating costs. Field work was
conducted between April 12, 2002 and July 24,2002. Mr. John T. Lynch,
Lead Information Systems Auditor and Mr. Tim Beirnes, Lead Consulting
Auditor, prepared this report.
Sincerely,
Allen Vann
Inspector General
AV/
Enclosure
c: Henry Dean
John Fumero
INTRODUCTION .....................................................................................................1
BACKGROUND .......................................................................................................2
EXECUTIVE SUMMARY..........................................................................................4
During the April 10, 2002 Governing Board Workshop the Information
Technology Staff made a presentation on the “Cost Model and 10-Year
Projects” for the District’s Water Management System (WMS). This
presentation covered the 10-year cost projections for the
upgrade/replacement of the District’s supervisory control and data acquisition
(SCADA) system software, hardware, data collection and communications
equipment. Staff estimated the cost for this planned project, including
ongoing support, would be $402,481,801 spread over the 10-year plan period.
During the Regular Governing Board meeting on April 11, 2002, the Board
requested that the Inspector General’s Office audit the SCADA plan as
presented by staff and report the findings to the Board.
• The various data collection technologies (CR10’s) and others into the
SCADA system.
• Non-recurring,
• Recurring, and
• Reimbursable Costs.
3. Interview project managers, implementation/maintenance support staff,
and data collection/processing staff.
4. Review the criteria and selection process for the MOSCAD RTU’s.
New Spending Of the $292 million for the eleven-year period from FY2002
Only 19% through 2012 (inclusive), only $55 million (19%) is due to
new capital and maintenance spending. Approximately
$43 million, or 15%, is due to inflation.
Conditions Reported
KEMA Recommendations
1
KEMA Consulting. South Florida Water Management District, Water Management, Needs Analysis.
May 2000; 1.1 – 1.5.
• The original estimate included indirect cost of $81.8 million, which includes
$20.3 million for fringe benefits and $61.5 million in central services
overhead. In our opinion, fringe benefits should be included; however, the
SCADA initiatives will not materially impact the current level of central
service overhead costs and thus is irrelevant for the purposes of this
2
The backup material presented to the Governing Board also contained an estimate of $320 million that
excludes fringe benefits and indirect cost.
• Field site maintenance cost was estimated at $3,600 per site per year.
Our analysis indicates current annual site maintenance cost is about
$2,520 per site. Our analysis was based on a review of current outsourced
contracts for site maintenance and historical repair frequency. This
reduced the site maintenance cost estimate by $12.9 million.
3
Management's Response and Comments at Appendix C, Note 2 (Page 16), states that the "draft report
inadvertently applies productivity increase to all included EMA salaries rather than to data QA/AC alone".
Using this methodology, however, also decreases the estimated cost per sensor. Additional analysis by the
Office of Inspector General considering staff's methodology indicates that the cost estimate would be about the
same. We additionally concluded that the Customer Support and Legally Mandated units workloads do not
increase proportionately with the number of new sensors.
Recommendations:
Management Response:
(2) The site survey should be validated to ensure that surveyed sites
are necessary and do not duplicate existing or projected new
sites.
Our analysis indicates the estimated cost to develop and operate the SCADA
system is about $292 million for the eleven-year period from FY2002 through
• The District budgeted $17.6 million for SCADA initiatives in fiscal year
2002. Thus, the District would expend $194 million over the 11-year
period if expenditures continued at the current level and if prices did not
increase.
• Only $55 million (19%) is due to proposed new spending beyond the
current level of expenditures at current price levels.
• About $43 million (15%) is due to escalating current and new spending for
inflation.
Total Cost
$292M
Current
$194 $55 $43 New
SCADA Total
Inflation
The following graph shows a breakdown of cost between capital outlay and
operating cost by the above categories:
$85 $109
Current Cost
Five District programs account for 99% of all new field site installation costs
as shown in the following graph:
ECP Other
Planning 1%
6%
9% CERP
$7 $5 $1 39%
$10 $29
Big Cypress
13%
$24.
Flood Control
32%
Three District programs, representing 58% of new site installation costs, are
funded through special revenue funds. These expenditures are specifically
identifiable with these programs; accordingly, the cost for these new sites
should be charged directly to the following programs.
• The $29 million estimated cost for CERP project sites should be eligible for
in-kind credit towards the District’s 50% cost share. Also, the U. S. Army
Corps of Engineers (USCOE) will likely pay directly for some of these costs
since approximately 40% of site installation cost is for items that are
generally part of the structural construction contract (such as wiring,
equipment housing, and building accesses to the equipment).
• ECP already has dedicated funding sources for its $5 million estimated
new field site costs.
Recommendation:
Note
Staff IG
Estimates Adjustments Estimates
Non-Recurring Costs
New Installations - Outsource $ 65,964,984 $ 10,021,767 a $ 75,986,751
Field Infrastructure
Site Upgrades - Outsource $ 5,901,377 $ - $ 5,901,377
Site Upgrades - Other Direct 1,266,743 - 1,266,743
Total Field Infrastructure $ 7,168,120 $ - $ 7,168,120
Communications Infrastructure
System Expansion - Outsource $ 7,662,342 $ (355,239) $ 7,307,103
System Expansion - Other Direct 12,847,706 - 12,847,706
Total Communications Infrastructure $ 20,510,047 $ (355,239) b $ 20,154,809
SCADA Central
Water Management SCADA System - Outsource $ 2,821,747 $ (1,771,851) $ 1,049,896
Water Management SCADA System - Other Direct 525,000 1,101,487 1,626,487
Total SCADA Central $ 3,346,747 $ (670,364) c $ 2,676,383
Information Management
Water Management Information System - Outsource $ 6,570,000 $ (2,125,000) d $ 4,445,000
Water Management Information System - Other Direct 500,000 - 500,000
Other EMA Direct 1,518,400 (650,000) e 868,400
Total Information Management $ 8,588,400 $ (2,775,000) $ 5,813,400
Total Non-Recurring Costs $ 105,578,298 $ 6,221,165 $ 111,799,463
Recurring Costs
Field Infrastructure
Operations and Maintenance - Outsource $ 47,705,935 $ (12,890,115) h $ 34,815,820
Operations and Maintenance - Other Direct 33,036,794 - 33,036,794
Operations and Maintenance - Salaries 23,056,091 7,377,949 f 30,434,040
Indirect 28,234,489 (28,234,489) g -
Total Field Infrastructure $ 132,033,309 $ (33,746,655) $ 98,286,654
Communications Infrastructure
Operations and Maintenance - Outsource $ 2,002,471 $ - $ 2,002,471
Operations and Maintenance - Other Direct 7,358,465 - 7,358,465
Operations and Maintenance - Salaries 5,768,151 1,845,808 f 7,613,960
Indirect 6,301,705 (6,301,705) g -
Total Communications Infrastructure $ 21,430,794 $ (4,455,897) $ 16,974,897
SCADA Central
Operations and Maintenance - Outsource $ 5,266,335 $ - $ 5,266,335
Operations and Maintenance - Other Direct 128,078 - 128,078
Operations and Maintenance - Salaries 5,242,210 1,266,604 f 6,508,815
Indirect 5,727,115 (5,727,115) g -
Total SCADA Central $ 16,363,738 $ (4,460,510) $ 11,903,228
Information Management
Operations and Maintenance - Outsource $ 48,304,200 $ (42,270,373) i $ 6,033,827
Operations and Maintenance - Other Direct 6,562,850 - 6,562,850
Operations and Maintenance - Salaries 30,641,013 9,805,124 f 40,446,137
Indirect 41,567,598 (41,567,598) g -
Total Information Management $ 127,075,662 $ (74,032,847) $ 53,042,814
(b) Equipment lease payments were inadvertently escalated for inflation where the payments are
fixed amounts over the lease term. Adjustment reflects removal of the inflation from the lease
payments.
(c) During the course of the audit, the Governing Board approved a contract for the Central SCADA
system in the amount of $1,328,368. This system was originally estimated to cost $2,000,000.
Thus, estimate was reduced to reflect actual contract cost.
(d) Costs for software and implementation of the new Water Management Information System
(WMIS) and database upgrades were based on preliminary numbers in a draft report KEMA
Consulting prepared. Costs were adjusted to reflect the estimate in KEMA Consulting's final
report, which were significantly less then the draft report estimates.
(e) Adjustment represents cost relating to the new WMIS that were double counted.
(f) Staff estimate added indirect cost, which includes fringe benefits as well as central services
overhead. In our opinion, only the fringe benefits portion of indirect cost should be included in the
analysis. All indirect costs were removed (see Note g) and fringe benefit costs were added to
base salaries (at the rate of 32%).
(g) Indirect costs include central services overhead and salary fringe benefits. The level of SCADA
system activity will not materially impact central services overhead; however, fringe benefits are
variable expenditures. Hence, all indirect costs were removed and fringe benefit costs were
added to the base salaries (see Note f). The effect of removing central services overhead
reduces the ten-year cost estimate by $61,535,422.
(h) Original estimate was based on an annual cost of $3,600 for preventative maintenance and
emergency repairs for each field site. We performed an analysis of field site maintenance
expenditures, which showed cost to be approximately $2,520 per site annually. Cost estimates
were adjusted accordingly.
(i) Investments in the SCADA system and the Environmental Data and Analysis System should
provide significant efficiency gains by standardizing the data collection process and incorporating
program logic to validate data that is currently examined manually. The new system will only
necessitate staff manually reviewing exceptions the system generates. The staff estimate
assumes that data quality assurance cost will increase linearly with new sensor installations - plus
3% inflation. The staff estimate also assumes that external contractors will be engaged to handle
the increased workload the new sensors generate. Our analysis indicates that the new system
will approximately double internal staff's efficiency, thereby eliminating most of the outsourcing
needs.
Note: The Inspector General estimates are based on the number of new sites per the "site
survey", which is subject to revision based on staff review. Any significant changes to the
number of estimated sites would materially change cost estimates.
These definitions were developed by District staff or were drawn from the "Free On-line
Dictionary of Computing," by Dennis Howe @ Web Site: http://www.foldoc.org.
backup
A spare copy of a file or system of files, usually kept on magnetic tape or other removable
medium such as compact disc, for use in the event of failure or loss of the original files or
system.
CR 10
Campbell Scientific, Inc. remote terminal unit (RTU) data logger.
hardware
The physical, touchable, material parts of a computer or other system. The term is used to
distinguish these fixed parts of a system from the more changeable software or data
components.
MOSCAD
Motorola SCADA remote terminal units (RTU’s). MOSCAD RTU’s are programmable units
with multiple communication paths that have the capable of controlling gates, pumps, and
monitoring data sites.
on-line
Accessible directly via a computer (or terminal), rather than on paper or other removable
medium such as magnetic tape or CD.
password
An arbitrary string of characters chosen by a user or system administrator and used to
authenticate the user when he attempts to log on in order to prevent unauthorized access to
his account.
platform
Specific computer hardware. It may also refer to a specific combination of hardware and
operating system.
real-time
An application which requires a program to respond to stimuli within some small upper limit of
response time (typically milli- or microseconds). Process control at a chemical plant is the
classic example. Such applications often require special operating systems (because
everything else must take a back seat to response time) and speed-tuned hardware.
recovery
The process of restoring computer data file with a backup copy usually after a crash or
accidental deletion of a file.
software
Computer programs, as opposed to the computers on which they run (the "hardware").
telemetry
Transmission and collection of data obtained by sensing conditions in a real-time
environment.
user(s)
The people who either use computers directly, or use the information they provide; also
called computer users or end users.