Measuring Control Effectiveness

Measuring Control Effectiveness
John Mitchell PhD, MBA, CEng, CITP, FBCS, CFIIA, CISA, CGEIT, QiCA, CFE
LHS Business Control Tel: +44 (0)7774 145638

47 Grangewood
Potters Bar
Hertfordshire
EN6 1SL [email protected]
England www.lhscontrol.com
GRC 2.0 - Breaking Down

rd The Silos
© John Mitchell ISACA Ireland Conference – 3 October 2014
CMM & ISO 15504 Levels
CMM ISO 15504

5 – Optimised 5 - Optimised
4 – Managed and Measurable 4 – Predictable
3 – Defined 3 – Established
________________________________________
2 – Repeatable 2 - Managed
1 – Ad Hoc 1 - Performed
0 – Non existent 0 - Incomplete
GRC 2.0 - Breaking Down The Silos

© John Mitchel rd
ISACA Ireland Conference – 3 Slide # 2 2014
October
Components of the Control Environment
Monitoring
Control
Activity
Control
Risk Objectives
Information &
Communication Analysis

© John Mitchel rd
ISACA Ireland Conference – 3 October 2014
Generic Risk Management Process
H
i
g E
h
L
Senior Management I
D Inherent
Attention K
E Risk
Local Management L
I
Attention C Likelihood
H
O Reduction
No Action O
D
B Residual
Risk Consequence
L Reduction
o A
w
A B C D E
Low CONSEQUENCE High

rd The Silos
© John Mitchel ISACA Ireland Conference – 3 October 2014
Which Risk
Would You Want Assurance Over?
Residual
Inherent Risk Controls
Risk
Risk 1 None
Risk 2 Some
Risk 3 Lots

rd The Silos
What Is This Control Stuff?
Anything which monitors or modifies a process to ensure its

predictability
A control is basically a test against a prediction
You can only test for what you can predict
Sometimes the prediction is absolute

(gender must be ‘F’)
Sometimes the prediction is variable

(within the range of 50 to 50,000)

rd The Silos
Control Classifications
Class Ability to detect the event and take recovery action Type
1 Prevents the event, or detects it as it happens and prevents further Preventive

impact
2 Detects the event and reacts fast enough to fix it well within the
specified time window
3 Detects the event and reacts just fast enough to fix it within the Detective
4 Detects the event but cannot react fast enough to fix it within the
5 Fails to detect the event but has a partially deployed business
continuity plan
6 Fails to detect the event but does have a business continuity plan Reactive
7 Fails to detect the event and does not have a business continuity
plan
Source: D Brewer & W List GRC 2.0 - Breaking Down

rd The Silos
Anatomy of a Control
Design
Implementation
Monitoring
Evaluation

rd The Silos
Measuring Control Design
How well the control should work, in theory, if it is always

applied in the way intended:
3 – designed to reduce risk aspect entirely
2 – designed to reduce most aspects of risk
1 – designed to reduce some areas of risk
0 – very limited or badly designed, even where used correctly
provides little or no protection

rd The Silos
Measuring Control Implementation
The way in which the control performs in practice:

3 – control is always applied as intended
2 – control is generally operational but on occasions is not
applied as intended
1 – control is sometimes correctly applied
0 – control is not applied or applied incorrectly

rd The Silos
Measuring Control Monitoring
How we know that the control is continuing to operate

(embedded monitor):
3 – operation is always monitored
2 – operation is usually monitored, but on occasions is not
1 – operation is monitored on an ad-hoc basis
0 – operation is not monitored at all

© John Mitchel rd
Measuring Control Evaluation
How frequently control effectiveness & efficiency is

evaluated:
3 – control is regularly evaluated for effectiveness/efficiency
2 – control is occasionally evaluated for
effectiveness/efficiency
1 – control is evaluated very infrequently
0 – control is never evaluated

rd The Silos
Scoring Control Effectiveness Example
(No Weighting)
Apply DIME:
Design = 2 (3)
Implementation = 3 (3)
Monitoring = 2 (3)
Evaluation = 1 (3)
TOTAL = 8 (12) = 0.75 (75% total effectiveness)
NOTE: If either Design, or Implementation is zero then

total score becomes zero

rd The Silos
Risk & Control Documentation
LHS Business Control 47 Grangewood, Potters Bar, EN6 1SL, England +44 (0)1707 851454 [email protected] www.lhscontrol.com
RISK & CONTROL DOCUMENTATION

Company:
Division:
Location:
Business Area/Activity: Score the Effectiveness of the

Controls in Mitigating the Risk
N/A 1 2 3 4 5
A Controls for managing the risk of
B As a minimum these should include the Is it performed? How

following standard controls Contr. Contr. Who/what performs it? Often? How is it evidenced?
Class Score
N/A Yes No
1) Control 1
2) Control 2
3) Control 3
4) Control 4
C Where the answer to a minimum requirement is Is it performed? Who/what performs it? How How is it evidenced?
NO: Contr. Contr.
Class Score
Please give details of any alternative controls N/A Yes No
providing assurance
D Where the score for control effectiveness is < 3 Contr. Proposed Pot. Who/what will perform How How will it be evidenced?
Class Implementation Score it? Often?
Please detail the control which is to be Date
implemented to improve the result

rd The Silos
Assessing Overall
Control Effectiveness
Analyse each control to arrive at an overall score for all of

the controls mitigating a risk
1 = Poor level of control - management attention required

2 = Very basic control - enhancement required
3 = Adequate level of control - scope for improved
effectiveness
4 = Good control - scope for increased efficiency
5 = Excellent control - no improvement possible

rd The Silos
Control Effectiveness Reporting
H
i
g E 8
h
L
I 3,4,5,6,7,9, 12) Power
D 2,18
10,11,13,14
12
K Loss
E
L 14) 3rd Party
I Support
C 16
H
O
O
D
B 1
L 15) Loss of
o A 17 15 Data Centre
w
A B C D E
Low CONSEQUENCE High

rd The Silos
Graphical Representation
(Multiple Risk Areas)
M/F Ops.
Network
Changes in control over the Internet
Disaster Rec.
Change Control
4.5
Internet
3.5 EPOS
HR
2.5 EDI
Sys. Dev.
1.5
Sys. Maint.
0.5 Help Desk
2010 2012 2013 Mngt. Info.
Cap. Plan.
Tech. Support
© John Mitchel 17 GRC 2.0 - Breaking Down
rd The Silos
Summary
 Whether you use CMM or ISO 15504 you still need to assess control
effectiveness
 Evaluation should be against the controls mitigating a risk
 Evidence must be available that the control is effectively working
 The evidence must show who/what operates the control and the
frequency of operation
 Control effectiveness can be consistently assessed by applying the

DIME method

rd The Silos
Questions?
John Mitchell
PhD, MBA, CEng, CITP, FBCS, CFIIA, CISA, CGEIT, QiCA, CFE
LHS Business Control

47 Grangewood
Potters Bar
Hertfordshire EN6 1SL
England
Tel: +44 (0)7774 145638
[email protected]
www.lhscontrol.com

© John Mitchel rd

Measuring Control Effectiveness - John Mitchell

Uploaded by

Copyright:

Available Formats

Measuring Control Effectiveness - John Mitchell

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Measuring Control Effectiveness - John Mitchell

Uploaded by

Copyright:

Available Formats

LHS Business Control Tel: +44 (0)7774 145638

GRC 2.0 - Breaking Down

CMM ISO 15504

GRC 2.0 - Breaking Down The Silos

GRC 2.0 - Breaking Down The Silos

GRC 2.0 - Breaking Down

GRC 2.0 - Breaking Down

Anything which monitors or modifies a process to ensure its

A control is basically a test against a prediction

You can only test for what you can predict

Sometimes the prediction is absolute

Sometimes the prediction is variable

GRC 2.0 - Breaking Down

1 Prevents the event, or detects it as it happens and prevents further Preventive

Source: D Brewer & W List GRC 2.0 - Breaking Down

GRC 2.0 - Breaking Down

How well the control should work, in theory, if it is always

GRC 2.0 - Breaking Down

The way in which the control performs in practice:

GRC 2.0 - Breaking Down

How we know that the control is continuing to operate

GRC 2.0 - Breaking Down The Silos

How frequently control effectiveness & efficiency is

GRC 2.0 - Breaking Down

TOTAL = 8 (12) = 0.75 (75% total effectiveness)

NOTE: If either Design, or Implementation is zero then

GRC 2.0 - Breaking Down

RISK & CONTROL DOCUMENTATION

Business Area/Activity: Score the Effectiveness of the

B As a minimum these should include the Is it performed? How

GRC 2.0 - Breaking Down

Analyse each control to arrive at an overall score for all of

1 = Poor level of control - management attention required

GRC 2.0 - Breaking Down

GRC 2.0 - Breaking Down

 Evaluation should be against the controls mitigating a risk

 Evidence must be available that the control is effectively working

 Control effectiveness can be consistently assessed by applying the

GRC 2.0 - Breaking Down

LHS Business Control

Tel: +44 (0)7774 145638

GRC 2.0 - Breaking Down The Silos

You might also like