Return On Security Investment (ROSI) - A Practical Quantitative Model
Return On Security Investment (ROSI) - A Practical Quantitative Model
Return On Security Investment (ROSI) - A Practical Quantitative Model
Organizations need practical security benchmarking tools in order to plan effective security
strategies. This paper explores a number of techniques that can be used to measure security within
an organization. It proposes a new benchmarking methodology that produces results that are of
strategic importance to both decision makers and technology implementers. The approach taken
reflects a work-in-progress that is a combination of practical experience and direct research.
Keywords: Return on Security Investment, benchmarking, security strategy, security metrics,
management, measurement, standardization, economics, algorithms
ACM Classification: H.1.1 (Models and Principles: Systems and Information Theory – Value of
Information, K.6.5 (Management of Computing and Information Systems: Security and
Protection), K.6.0 (Management of Computing and Information Systems: General – Economics),
H.4.2 (Information Systems Applications: Types of Systems – Decision Support) eg. MIS))
1. INTRODUCTION
In a world where hackers, computer viruses and cyber-terrorists are making headlines daily, security
has become a priority in all aspects of life, including business. But how does a business become
secure? How much security is enough? How does a business know when its security level is
reasonable? Most importantly, what’s the right amount of money and time to invest in security?
Executive decision-makers don’t really care whether firewalls or lawn gnomes protect their
company’s servers. Rather, they want to know the impact security is having on the bottom line. In
order to determine how much they should spend on security, they need to know:
• How much is the lack of security costing the business?
• What impact is lack of security having on productivity?
• What impact would a catastrophic security breach have?
• What are the most cost-effective solutions?
• What impact will the solutions have on productivity?
Before spending money on a product or service, decision-makers want to know that the
investment is financially justified. Security is no different – it has to make business sense. What
decision-makers need are security metrics that show how security expenditures impact the bottom
Copyright© 2006, Australian Computer Society Inc. General permission to republish, but not for profit, all or part of this
material is granted, provided that the JRPIT copyright notice is given and that reference is made to the publication, to its
date of issue, and to the fact that reprinting privileges were granted by permission of the Australian Computer Society Inc.
Manuscript received: 12 April 2005
Communicating Editor: Julio Cesar Hernandez
Journal of Research and Practice in Information Technology, Vol. 38, No. 1, February 2006 45
Return on Security Investment (ROSI) – A Practical Quantitative Model
line. There’s no point in implementing a solution if its true cost is greater than the risk exposure.
This paper will present a model for calculating the financial value of security expenditures, and will
look at techniques for obtaining the data necessary to complete the model.
(1)
To calculate ROI, the cost of a purchase is weighed against the expected returns over the life of
the item (1). An overly simplistic example: if a new production facility will cost $1M and is
expected to bring in $5M over the course of three years, the ROI for the three year period is 400%
(4x the initial investment of net earnings).
A simple equation for calculating the Return on Investment for a security investment (ROSI) is
as follows:
(2)
Let’s see how this equation works by looking at the ROI profile for a virus scanner. ViriCorp
has gotten viruses before. It estimates that the average cost in damages and lost productivity due to
a virus infection is $25,000. Currently, ViriCorp gets four of these viruses per year. ViriCorp expects
to catch at least three of the four viruses per year by implementing a $25,000 virus scanner.
(3)
The virus scanner appears to be worth the investment, but only because we’re assuming that the
cost of a disaster is $25,000, that the scanner will catch 75% of the viruses and that the cost of the
scanner is truly $25,000. In reality, none of these numbers are likely to be very accurate. What if
three of the four viruses cost $5,000 in damages but one costs $85,000? The average cost is still
$25,000. Which one of those four viruses is going to get past the scanner? If it’s a $5,000 one, the
ROSI increases to nearly 300% – but if it’s the expensive one, the ROSI becomes negative!
Coming up with meaningful values for the factors in the ROSI equation is no simple task. At the
time of writing, there is no “standard” model for determining the financial risk associated with
security incidents. Likewise, there are also no standardized methods for determining the risk
mitigating effectiveness of security solutions. Even methods for figuring out the cost of solutions
can vary greatly. Some only include hardware, software and service costs, while others factor in
internal costs, including indirect overhead, and long-term impacts on productivity.
46 Journal of Research and Practice in Information Technology, Vol. 38, No. 1, February 2006
Return on Security Investment (ROSI) – A Practical Quantitative Model
There are techniques for quantitatively measuring risk exposure, but the results tend to vary in
accuracy. For most types of risk, the exposure can be found by consulting actuarial tables built from
decades of claims and demographic statistics. Unfortunately, similar data on security risk does not yet
exist. Furthermore, the variability in exposure costs can lead to misleading results when predicting
based on actuarial data. In the ViriCorp example, the exposure cost is misleading – the average cost
of $25,000 doesn’t reflect the fact that most incidents cost very little while some cost quite a lot.
Is there any point to calculating ROSI if the underlying data is inaccurate? Apparently so, since
some industries have been successfully using inaccurate ROI metrics for decades. The advertising
industry is one such example. Ads are priced based on the number of potential viewers, which is
often extrapolated from circulation data and demographics. The ad buyers assume that the true
number of ad viewers is directly correlated to the number of potential viewers; if the viewer base
doubles, roughly twice as many people will probably see the ad. Therefore, even though they may
never know the true number of viewers, ad buyers can nonetheless make informed purchasing
decisions based on other more reliable measurements.
If the method for determining ROSI produces repeatable and consistent results, ROSI can serve
as a useful tool for comparing security solutions based on relative value. In the absence of pure
accuracy, an alternate approach is to find consistent measurements for the ROSI factors that return
comparably meaningful results. This task is much easier, and breaks through the barrier of accuracy
that has kept ROSI in the domain of academic curiosity.
KEY POINT: Repeatable and consistent metrics can be extremely valuable – even if they’re
“inaccurate”.
It’s very difficult to obtain data about the true cost of a security incident (the SLE). This is
because few companies successfully track security incidents. Security breaches that have no
immediate impact on day-to-day business often go completely unnoticed. When a breach does get
noticed, the organization is usually too busy fixing the problem to worry about how much the
incident actually costs. After the disaster, internal embarrassment and/or concerns about public image
often result in the whole incident getting swept under the rug. As a result of this “ostrich response”
to security incidents, the volume of data behind existing actuarial tables is woefully inadequate.
Currently, the “best” actuarial data comes from efforts such as the annual survey of businesses
conducted by the Computer Security Institute (CSI) and the U.S. Federal Bureau of Investigation
(FBI). The businesses are asked to estimate the cost of security incidents for various categories over
the course of a year. Unfortunately, the methods used to calculate these costs vary from business to
business. For example, one business might value a stolen laptop based on its replacement cost.
Another might factor in the lost productivity and IT support time, and yet another might factor in
Journal of Research and Practice in Information Technology, Vol. 38, No. 1, February 2006 47
Return on Security Investment (ROSI) – A Practical Quantitative Model
lost intellectual property costs. As a result, some businesses value a laptop theft at $3000; others put
it down as $100,000+. The final number is more likely to be influenced by business factors (how
much will insurance reimburse, what are the tax implications, what impact will a large loss have on
the stock price) than by financial reality.
For the purposes of ROSI, the accuracy of the incident cost isn’t as important as a consistent
methodology for calculating and reporting the cost, as previously discussed. It would be quite
challenging to get companies to agree upon a standard technique for tabulating the internal cost of
a security incident. Therefore, the focus must be on cost factors that are independently measurable
and directly correlate to the severity of the security incident.
One potentially significant cost is the loss of highly confidential information. In organizations
valued for their intellectual property, a security breach resulting in theft of information might create
a significant loss for the business yet not impact on productivity. The cost of a security incident in
this case is the estimated value of the intellectual property that is at risk, using industry-standard
accounting and valuation models. For most industries, analysts are already externally measuring
this value. If an organization doesn’t already estimate the value of its IP assets, it probably doesn’t
need to consider this cost.
Another significant cost is the productivity loss associated with a security incident. For many
organizations the cost in lost productivity is far greater than the cost of data recovery or system
repair. Security can be directly connected to an organization’s financial health by including lost
productivity in the cost of a disaster. This approach automatically forces security projects to
improve business efficiency and eliminates those projects justified solely by fear of the unknown.
Lost productivity can have a serious impact on the bottom line. Just ten minutes of downtime a
day per employee can quickly add up to a significant amount, as illustrated in Table 1.
1000 employees
* 44 Hours/year security related “downtime”
* $20 per hour average wage
= $880,000 per year in lost productivity
48 Journal of Research and Practice in Information Technology, Vol. 38, No. 1, February 2006
Return on Security Investment (ROSI) – A Practical Quantitative Model
Measuring employee perception of downtime can be accomplished with a survey. If the survey
is correctly constructed, there will be a strong correlation between the survey score and financial
performance. Specifically, if a department shows a decrease in perceived downtime, it should also
show an increase in productivity on the internal balance sheets.
A good survey will ask the employees questions that have coarse quantitative answers, or
answers that imply a quantitative value. For example, one question might be, “How much spam do
you receive each day?” The employee might have to choose between four answers: less than 10,
10–30, 30–50 or more than 50. Average minutes of downtime can be associated with each answer.
For example, dealing with 30–50 spam messages per day can cause up to ten minutes of downtime,
especially if it’s hard to tell the difference between spam and desired messages.
The key to getting consistent results from a survey that measures employee perception is to
ensure that the questions are quantitative, clear and answerable without too much thought. For
example, a bad question would be “Estimate the amount of downtime you had this month,” since
few people could answer this without logging events as they happen. A better question is to ask,
“How often is the fileserver unavailable for more than 10 minutes (daily, weekly, monthly, rarely)”.
A person who experiences weekly fileserver problems is unlikely to put down “daily” unless the
problem is extremely frequent.
Once the survey answers are scored, the result will be an indication of monthly downtime. This
can be converted into a dollar amount of lost productivity by using salaries expressed as hourly
rates. For example, if the average salary for a department is $75/hour and the average downtime is
30 hours per month, then the company is losing $2250 in non-productive time per employee due to
security-related issues. In a professional service firm, these employees might also generate revenue.
The hourly billable rate multiplied by the revenue realization rate and the monthly downtime gives
an additional quantification of lost revenue opportunity. Tuning the productivity survey so that the
calculated loss exhibits stronger correlation with internal financial measurements of profit and loss
can increase accuracy.
Journal of Research and Practice in Information Technology, Vol. 38, No. 1, February 2006 49
Return on Security Investment (ROSI) – A Practical Quantitative Model
KEY POINT: With a good survey and scoring system for productivity, combined with
external measurements of intellectual property value, it becomes possible to quantify risk
exposure in a repeatable and consistent manner.
KEY POINT: There are a number of ways in which lost productivity can provide a
meaningful estimate of risk exposure, any of which can be used to calculate ROSI.
50 Journal of Research and Practice in Information Technology, Vol. 38, No. 1, February 2006
Return on Security Investment (ROSI) – A Practical Quantitative Model
The following argument has been used to justify a simple, fixed percentage for risk mitigation:
• A security solution is designed to mitigate certain risks.
• If the solution is functioning properly, it will mitigate nearly 100% of these risks (85% to be
conservative).
• Therefore, the amount of risk mitigation is 85%.
Unfortunately, there are a number of serious problems with this “logic”:
• Risks are not isolatable — a well-locked door mitigates 0% of risk if the window next to it is
open.
• Security solutions do not work in isolation - the existence and effectiveness of other solutions
will have a major impact.
• Security solutions are rarely implemented to be as effective as possible due to unacceptable
impact on productivity.
• Security solutions become less effective over time, as hackers find ways to work around them
and create new risks.
A better approach is to conduct a security assessment and “score” the assessment based on some
consistent algorithm. This score can represent the amount of risk currently being mitigated. By
1 Based on aggregate SecureMark results and analysis
Journal of Research and Practice in Information Technology, Vol. 38, No. 1, February 2006 51
Return on Security Investment (ROSI) – A Practical Quantitative Model
evaluating risk mitigation within the context of the network’s overall security, the two problems of
isolation mentioned above are avoided. A good assessment will also capture the impact of
implementation choices made for the sake of usability and productivity. Likewise, a good scoring
algorithm will factor in the time impact on solution effectiveness.
When evaluating a security solution, the assessment can be conducted as if the solution were
already in place. The difference between this score and the actual score is the amount of risk being
mitigated due to the solution. When calculating ROSI, the predicted score (not the difference)
should be used as the overall risk mitigation.
The accuracy of the score as a measurement of mitigated risk is dependent on the quality of the
assessment and scoring algorithm. Following assessment guidelines published by standard-setting
groups such as the International Security Forum (ISF), National Institute of Standards in
Technology (NIST), and the International Standards Organization (ISO) will lead to the creation of
good assessments. Artificial Neural Networks can be used to create particularly good scoring
algorithms, the details of which will be discussed in a forthcoming paper.
KEY POINT: Even with an inaccurate scoring algorithm, using a scored assessment as a
method of determining risk mitigation is effective because the scores are repeatable and
consistent, and therefore can be used to compare the ROI of different security solutions.
Problem Average
Downtime
Application and system related crashes 10 Mins
Bandwidth efficiency and throughput 10 Mins
Over-restrictive security policies 10 Mins
Enforcement of security policies 10 Mins
System related rollouts and upgrades from IT 10 Mins
Security patches for OS and applications 10 Mins
Trouble downloading files due to virus scanning 10 Mins
Compatibility issues – hardware and software 15 Mins
Too many passwords/permissions security problems 15 Mins
Table 3: Productivity Loss Due to Security Solutions
52 Journal of Research and Practice in Information Technology, Vol. 38, No. 1, February 2006
Return on Security Investment (ROSI) – A Practical Quantitative Model
For example, implementing a firewall might require a network restructuring. The new structure
might solve serious bandwidth problems that were previously creating extensive downtime.
This productivity impact can be measured by re-running the productivity surveys used to
estimate risk exposure. The given answers are adjusted to assume that the solution has been put into
place. The difference between the current and projected productivity is the impact factor that needs
to be included in this calculation.
Let’s factor productivity into our earlier example with ViriCorp’s virus scanner. We can see that
if cost of the solution exceeds $60,000, the ROI is 0% and therefore it’s not worth purchasing.
Assuming the full cost of the system remains at $30,000, there’s a margin of $30,000. For 100
employees earning an average of $20/hour, that margin equates to 3.5 minutes per day of downtime.
If implementing the virus scanner creates more than 3.5 minutes of downtime each day, it’s more
cost effective to not purchase the scanner. On the other hand, if the scanner can eliminate downtime
by minimizing the impact of viruses, it could make the scanner quite attractive in terms of ROI.
KEY POINT: The cost of a solution must include the impact of the solution on productivity,
since this number is often large enough to make or break the viability of a given solution.
Unfortunately, nobody can predict when a security device will prevent a problem. As a result,
one solution is to spread the savings out across the predicted lifetime of the device. You could also
“front-load” the savings, under the assumption that the device will be most effective at the
Journal of Research and Practice in Information Technology, Vol. 38, No. 1, February 2006 53
Return on Security Investment (ROSI) – A Practical Quantitative Model
beginning of its life, and lose effectiveness as the years progress and hackers figure out how to
bypass the device:
The problem with using Net Present Value for security investments is that accuracy is quite
critical to obtaining comparatively meaningful results. While ROSI doesn’t factor in the time value
of money, it can at least provide comparable figures with inaccurate (but consistent) data. This may
be a case where it’s better to be meaningful than precise.
54 Journal of Research and Practice in Information Technology, Vol. 38, No. 1, February 2006
Return on Security Investment (ROSI) – A Practical Quantitative Model
on organizational productivity. This influence can be significant, and must be factored into the cost
of the solution. SecureMark can estimate the impact a given solution will have on overall
productivity. This impact is factored in when prioritizing underlying problems and their respective
solutions.2
The resulting SecureMark scorecard gives all the factors necessary to calculate the Return On a
Security Investment: Risk Exposure expressed in dollars of lost productivity, and the percentage of
risk currently mitigated expressed as a SecureMark Score. The analysis indicates the top problems
prioritized by their impact on risk exposure and lost productivity. Likewise, the solutions presented
are selected based on their predicted ability to mitigate risk and minimize lost productivity.
In a few years, the data accumulated by SecureMark will allow an unprecedented amount of
accuracy in its scoring and analysis. For now, we have not yet collected enough data to begin
eliminating subjectivity from SecureMark’s scoring and analysis. That said, our system is still
consistent, which allows for meaningful comparison of solutions. It also allows for meaningful
industry comparisons – a company can tell if its score is above or below industry average. Until the
system can automatically provide accurate results, SageSecure security experts review all scores
and analyses to ensure consistency and accuracy. The result is the only automated, repeatable and
consistent ROSI benchmarking system available to date.
3. CONCLUSION
In this paper we’ve presented an analysis of the problem of determining a meaningful Return on
Security Investment for security expenditures. We presented a model for calculating ROSI, and then
showed how the various factors could be obtained. Some unique approaches to measuring Risk
Exposure and Risk Mitigation were explored, specifically those that focused on lost productivity as
a critical factor. The importance of factoring productivity into both exposure and solution cost was
stressed. The suitability of using Net Present Value in this context was explored, and a real-world
implementation of the entire model (SecureMark) was examined.
We hope the concepts discussed in this paper will encourage further research into the connection
between productivity and security. We feel that this is one of the most promising areas in which a
strong connection can be made between security and financial performance.
2 It might appear that the productivity impact of a security solution is getting factored in twice: once because the Risk
Mitigated * Risk Exposure gives a $ figure for productivity savings, and a second time when factored into the cost. These
are actually two different ways in which productivity affects ROSI. The first shows that any security improvement will
minimize the chance of productivity draining incidents, and therefore reclaims some lost productivity, proportional to the
increase in risk mitigation. The second way is the impact that the solution itself will directly have on productivity loss.
For example, implementing a spam filter will marginally improve overall security by stopping a number of different
email-borne threats. This will impact on overall productivity by minimizing downtime due to these threats. This impact
will be captured by the increase in risk mitigation. The spam filter may also save employees up to 15 minutes per day by
improving their email usage efficiency. Factoring the productivity impact into the cost of the solution will capture this
gain. In some cases there is a small amount of overlap between the two influences, but this is generally inconsequential
and can be further minimized by adjusting the scoring system.
REFERENCES
A GUIDE TO SECURITY RISK MANAGEMENT FOR INFORMATION TECHNOLOGY SYSTEMS PUBLISHED BY
THE GOVERNMENT OF CANADA COMMUNICATIONS SECURITY ESTABLISHMENT (1996): See: www.cse.
dnd.ca/en/documents/knowledge_centre/publications/manuals/mg2e.pdf
BERINATO, S. (2002): Calculated Risk, CSO Magazine, December. See: www.csoonline.com/read/120902/calculate.html
BRAITHWAITE, T. (2001): Executives need to know: The arguments to include in a benefits justification for increased
cyber security spending. In Information Systems Security, Auerbach Publications, September/October.
BUTLER, S.A. (2002): Security attribute evaluation method: A Cost-benefit approach, Computer Science Department,
Carnegie Mellon University. See: www2.cs.cmu.edu/~Compose/ftp/SAEM-(Butler)-ICSE_2002.pdf
Journal of Research and Practice in Information Technology, Vol. 38, No. 1, February 2006 55
Return on Security Investment (ROSI) – A Practical Quantitative Model
BIOGRAPHICAL NOTES
Wes Sonnenreich is a co-founder of SageSecure, LLC. Prior to SageSecure, Wes served as co-
founder and Chief Technology Officer for Glocal Communications, an Internet strategy and
solutions company dedicated to the pharmaceutical industry. At the time, Glocal represented over
60% of the global pharmaceutical industry with offices in Boston, Washington DC, London, Basel
and Tokyo. Wes has authored a number of books on security and network technologies, most
recently including ‘Network Security Illustrated’, published in October 2003 by McGraw-Hill. He
holds a B.S. in Computer Science and Music from M.I.T. and attended Harvard Business School’s
Program for Global Leadership, a 10 week long intensive Executive Education program for senior
managers of companies with extensive international operations.
Jason Albanese has years of experience as a successful entrepreneur and business leader and
is the co-founder of SageSecure, LLC. Prior to SageSecure, Jason was the founder and CEO of
Jumar Technologies, a business-to-business software company based in New York City. Jason led
Jumar Technologies from its inception, providing vision and management for a 30-person
workforce. Jumar Technologies was spun-off of JP Consulting Group, a database software
consulting company, also founded by Jason. At JP Consulting he created technology strategies for
large organisations and managed a team of highly skilled consultants. Jason co-authored
‘Network Security Illustrated’, published in October 2003 by McGraw-Hill. He holds a B.A. in
Economics from Union College.
Bruce Stout CPA is the founder and president of The Rainmakers’ Forum, a mentoring and
business networking organisation for consultants, CEOs, entrepreneurs, executives, professionals
and financial or service company representatives. He is a recognized expert on coaching, practice
development and strategic planning. Bruce is a widely read author, consultant and professional
speaker, appearing on CNN, CNBC and Preferred Lifestyles (his own talk show).
56 Journal of Research and Practice in Information Technology, Vol. 38, No. 1, February 2006