Guideline for Follow-Up of Performance Audit

Recommendations

Prishtina, December 2021

1
Content

1. INTRODUCTION ......................................................................................................... 4
2. PERFORMANCE AUDIT FOLLOW-UP ACTIVITY ................................................ 5
2.1 The aim and purpose of follow-up in performance audit ....................................... 5
2.2 Follow-up of the implementation of recommendations given in the audit report .. 6
3. FOLLOW-UP OF THE IMPLEMENTATION OF RECOMMENDATIONS ............. 7
3.1 Follow-up procedures ............................................................................................. 7
3.2 Follow-up planning ................................................................................................. 8
3.3 Monitoring of progress of recommendation implementation ................................. 8
3.4 Subsequent audits ................................................................................................... 9
3.5 Reporting the results of the follow-up .................................................................. 10
4. OUTLINE OF AUDIT RECOMMENDATION FOLLOW UP SYSTEM ................ 11
4.1 Database for follow up audit recommendations ................................................... 11
Annex 1, Table 1: Database of Follow-up Audit Reccomendations ............................... 15
Annex 2, Table 2: Audit reports with implementation of recommendations and
implementation in progress as of date 20xx .................................................................... 16
Annex 3. Flow Chart of NAO Follow-Up audit process ................................................... 1

2
Abbreviations
ISSAI International Standards of Supreme Audit Institutions
NAO National Audit Office
SAI Supreme Audit Institution

3
1. INTRODUCTION

The audit process does not end with the issuance of the audit report or with the preparation
of the action plan by the audited entity to address the recommendations as these actions do
not ensure that these recommendations will be implemented. For this purpose, it is
necessary to carry out monitoring of implementation of the recommendations on a regular
basis. Monitoring is a follow-up process during which auditors evaluate the appropriateness,
effectiveness, and timelines of actions undertaken by the management of the audited
organization to address each audit recommendation. A follow-up process will facilitate the
effective implementation of audit findings and recommendations. It will also provide
feedback to the SAI, the legislature and the government on performance audit effectiveness
and the improvements made by the audited entity (ISSAI 3200). The auditor shall follow
up, as appropriate, on previous audit findings and recommendations and the SAI shall
report to the legislature, if possible, on the conclusions and impacts of all relevant corrective
actions (ISSAI 3000/136). Follow-up on the audit report is an important tool to strengthen
the impact of the audit and improve future audit work, and is therefore a valuable part of
the audit process (ISSAI 3200).

After the issuance of the performance audit report, the auditee is required to provide a
written action plan specifying time and the ways of implementing the recommendations
given. Timely implementation of recommendations is one of the goals of NAO as provided
in the audit reports. In this regard, NAO should establish a system that stores the previous
audit findings and/or recommendations of the audit and related activities that the auditee
has received. Monitoring activities provide feedback to the SAI, the government and
parliament on the results of the audits conducted. The auditor shall focus the follow-up on
whether the audited entity has adequately addressed the problems and remedied the
underlying situation after a reasonable period (ISSAI 3000/139).

After a period of time, the SAI may undertake additional evaluation of progress made by
the auditee in relation to the audit findings and recommendations given to the
implementation and report on the accompanying review. This means that based on action
plan for implementation of recommendations after the publication of the report, the
following points are checked:

 What are the reactions and what are the comments of various stakeholders?
 What else can we learn from the audit and the entire process - how can we make
improvements?
 Are there indications of a lack of quality of audits conducted, and if so, what could be
reasons and explanations?
 What has been done so far in connection with the audit and its recommendations,
what are the results?
 Are there arguments in connection with the audit and its recommendations,
considering that taking actions is more than necessary?

4
2. PERFORMANCE AUDIT FOLLOW-UP ACTIVITY

Follow-up of performance audit recommendations enhances the effectiveness of the

activities of the supreme audit institution, including impact of the audit, and also enhancing
the accountability towards the public. It is important to maintain good communication with
the management of the audited entity or entities that are related to the audited activities.
Good communication ensures the effectiveness of the audit process. The audit team should
explain to management the importance of each recommendation and the consequences for
the entity if it fails to fulfill them. The impact of the audit may be identified through the
effect of corrective action taken by the responsible parties, or through the influence of the
audit findings and conclusions over governance, accountability, the understanding of the
problem addressed or the approach towards it.

2.1 The aim and purpose of follow-up in performance audit

When conducting follow-up of audit reports, the aim is to determine whether actions taken
on findings and recommendations have remedied the underlying conditions. This means
that both positive and negative reactions regarding the audit and the audit report need to
be examined by the auditor. It is therefore important to adopt an unbiased approach.
Follow-up activities after the publication of the performance audit report for the
implementation of recommendations contributes to the effective carrying out of
recommendations from the audited institutions.

Following up on audit findings and recommendations may serve four main purposes:

a) Identifying the extent to which audited entities have implemented changes in

response to audit findings and recommendations;
b) Determining the impacts which can be attributed to the audits;
c) Identifying areas that would be useful to follow up in future work; and
d) Evaluating the SAI’s performance. Follow-up provides a basis for assessing and
evaluating SAI performance and may contribute to better knowledge and improved
practices in the SAI. In this respect a follow-up of audit reports is also a self-
assessment tool.
Follow-up of the implementation of the recommendations in the NAO should ensure to:

 Increase the effectiveness of audit reports, the primary reason for follow-up the
performance audit report is to increase the likelihood that recommendations will be
implemented;

 Help the executive and legislative bodies, follow-up reports may be of importance for
influencing the actions of legislative bodies;

5
  Provide basis for evaluating and assessing performance of given recommendations
and the SAIs itself; and

 Serve as an incentive for learning and development activities, performance follow-up

can contribute to better knowledge and improving audit practice.
In the process of follow-up the performance audit team and management of the NAO
participates, and the actions following the publication of the report can be divided into the
activities of the audit team and management activities of the NAO1.

The activities of the audit team are reflected in the following:

 Compare the work plan with the actual work performed;

 Recommendations for follow-up activities; and

 Implement subsequent audits.

SAI management activities are reflected in the following:

 Deciding on the policy of supporting activities;

 Determine the date for the commencement of the supporting activities; and

 Method of reporting the results of supported activities.

It is known that it is very difficult to isolate the impact of the audit report in the context
of other significant changes, so the auditor should rather focus on the fact that have
corrected the weaknesses rather than on whether specific recommendations are
implemented or not. The key factor remains the question of whether the audit
recommendations are implemented, and this factor can often prove to be the only
measurable indicator of impact.

2.2 Follow-up of the implementation of recommendations given in the audit

report

Recommendations in performance audit report should be monitored to determine their

progress, what measures are taken by the competent institution to implement these
recommendations and identify the causes if the recommendations were not implemented.
Time and extent of follow-up activity depends on the management decisions of the NAO.
Depending on the type of follow up, a separate letter should be submitted to the audited
entity.

Supporting activities for the recommendations can be done in several different ways. The
most common are: direct contacts with audited entity or through formal requests for
information and subsequent audits. The simplest way is to hold meetings with the client
after a certain time to determine what steps the client has taken to improve its efficiency

1 Annex 3 Flow Chart of NAO audit Follow-Up process

6
and to check that recommendations were implemented and provide evidence of
implementation where possible.

After the issuance of the performance audit report, consideration and adoption of reports
by the Parliament, the auditee may be requested to submit a response to the legislature or
SAI-in regards of issues related to implementation of recommendations given in the
performance audit report.

3. FOLLOW-UP OF THE IMPLEMENTATION OF

RECOMMENDATIONS

The decision to conduct follow-up activities on influence of the recommendations should

be based on an analysis of feedback results supporting activities for recommendations. If
there are indications that the competent institutions are slow in implementation of
recommendations given in the performance audit report, or weak actions are taken in
coordinating their actions, or have difficulty in understanding that all actions are necessary,
there are good reasons for supporting activities to influence recommendations and
subsequent audit.

SAI activities relating to measuring the effects of the recommendations require much effort
and analysis. Actually it is a new performance audit, which should cover how the
recommendations are implemented, and many other issues that may affect the results.
Impact assessment in every case is the essence of this activity, and provides the basis for
the effectiveness of audit recommendations given in the performance audit report.
Procedures for follow-up of audit recommendations include:

 Confirmation of the time frame within which management's responses to audit

findings and recommendations are required;

 Evaluation of management responses; and

 Follow-up audit (if required).

3.1 Follow-up procedures

Different methods may be used to perform follow-ups on the findings and the
recommendations made. Some of the examples are shown below:
a) Arrange a meeting with the responsible parties after a certain time has elapsed to find
out what actions have been taken to improve performance and to check which
recommendations have been implemented.
b) Request from the responsible parties to inform the NAO in writing about the actions
they have taken to address the problems presented in the audit report.

7
 c) Use phone calls or limited field visits to collect information on the actions taken by the
audited entity.
d) Keep up to date on reactions from responsible parties, the legislature and the media,
and analyze whether problems identified have been appropriately addressed or not.
e) Request financial audit to collect information about actions taken as part of their audit
procedures.
f) Carry out a follow-up audit, resulting in a new performance audit report.
The methods to apply will depend on the priorities established by the NAO during the
strategic and annual planning process for performance auditing. They are also influenced
by the importance of the identified problems, the actions expected to be implemented, and
the external interest in information on the actions taken.

3.2 Follow-up planning

To properly plan follow-up activities, auditors should complete the database, which
summarizes the recommendations from previous audits. The timeline for monitoring the
fulfillment of recommendations is noted in this database so that implementation activities
are timely monitored for the implementation of each recommendation.
Planning follow-up and how to apply this process it depends on the following factors:

 The importance of the audited process and discovered weaknesses;

 Costs and efforts associated with improving the audited process;

 The risk of occurrence of any eventual event, unless corrective measures are taken;

 Scope of the corrective action to ensure that all relevant entities apply the necessary
improvements; and

 The time frame for implementing the changes.

3.3 Monitoring of progress of recommendation implementation

Follow-up can be done through monitoring or through follow-up audits.

Monitoring will be appropriate when:

 The audited process or activity is of lesser importance and does not represent a
serious obstacle to achieving the organization's main objectives;

 Weaknesses, errors, deficiencies or irregularities identified are not of paramount

importance;

 The recommendations given are easy to implement; or

 Corrective action is not complicated.

8
Auditor can monitor progress through:

 Receiving and evaluating management responses to audit findings within a

reasonable period after the audit results have been communicated;

 Periodic admission of new data from the management of audited organizations, in

order to assess the state of affairs to correct the reported weaknesses; and

 Reporting on the status of responses to audit findings.

3.4 Subsequent audits

When conducting follow-up of an audit report, the auditor needs to concentrate on findings
and recommendations that are still relevant at the time of the follow-up. Insufficient or
unsatisfactory action by the audited entity may call for a further audit by the NAO. The
subsequent audit is similar to traditional performance audit, although the objectives and
scope have narrowed to focus on the shortcomings noted in the previous report. In many
cases, audit work will be sufficient to follow up the recommendations without conducting
subsequent audits. However, if there is a management decision to implement subsequent
audits, it is necessary to know that a subsequent audit planning just as important as
planning and other studies and requires the same form as the work plan and the main
study. Subsequent audits should be included in the strategic audit plan, and depending on
priority given is decided by the management of the NAO.
Subsequent audit is applied in any of circumstances as following:

 When serious errors and deficiencies/irregularities have been identified in the prior
audit;

 When there is a high risk that the management fails to undertake the corrective
actions;

 Where the implementation of the recommendations requires the preparation of

further internal rules and regulations;

 When the required change refers to the activities of one or more entities; and

 When multiple resources are needed to introduce changes to the entity.

Follow-up audit contains the same planning, performance and reporting procedures as a
regular audit, adding several special procedures as follows:

 Reviewing the audit findings in the previous report, to determine the scope of the
follow-up audit;

 Designing adequate auditing tests and procedures for evaluating corrective actions;

 Conducting field audit work and document the results of the audit work performed;

 Verifying and reviewing certain dates for implementation if required; and

 Issuing the follow-up audit report.

9
During the follow-up audit, conclusions should be drawn as to whether the management's
actions have had an impact on reducing the risks identified in the previous audit and
whether they have improved the functioning of the organization towards achieving its
goals.

The scope of subsequent audits should be described in the work plan, and determined on
the basis of assessment of:

 Relevance of the initial conclusions;

 Corrective actions taken by the competent institutions;

 Action and influence of others who are not subject to audit; and

 Relevance of external factors and other issues.

Reporting on subsequent audits is conducted in accordance with general principles of
reporting. Reporting on the subsequent audit may not be required, and about the need for
such activities, it is decided by the management of the SAI. The decision depends on the
NAO assessment of the follow-up audit findings and conclusions.

Deviations and improvements identified in the subsequent audit should stand and should
always be reported to the subject of audit. Positive action in implementing the audit
recommendations should always be emphasized, because they are of merit of the NAO.
Follow-up the execution of the performance audit is important for evaluating the benefits
and progress that occurred as a result of the performance audit. It helps to better estimate
the effectiveness of performance audit in terms of improving services and the overall effects
of the public sector, and is useful when NAO seeks additional funding for their work.

3.5 Reporting the results of the follow-up

Whichever method is used, the results from the follow-up need to be reported
appropriately in order to provide feedback to the legislature. It is good practice to report
deficiencies and improvements identified in audit follow-up to the responsible parties or
the legislature.

Follow-up may be reported individually or as a consolidated report. If several follow-ups

are reported together, they may include an analysis of different audits, possibly
highlighting common trends and themes across a number of reporting areas. Whatever the
form, the follow-up reports must be balanced and findings presented objectively and fairly.

10
4. OUTLINE OF AUDIT RECOMMENDATION FOLLOW UP
SYSTEM

Responsibility for the follow-up of audit recommendations and reporting on the

implementation results shall lie with the audit team leader or another official of the NAO
appointed by the head of the audit department and in coordination with the Auditor
General. The follow-up of audit recommendations and reporting on recommendation
implementation results is governed by the Procedure for the provision and follow-up of
performance audit recommendations approved by an order of the Auditor General.

The person appointed to follow the recommendations and reporting on the implementation
results might be appointed also to monitor or coordinates closely with the person that
monitors the execution of resolutions and decisions of the Parliament of the Republic of
Kosovo and its committees according to the audit material.

4.1 Database for follow up audit recommendations

The best way to keep and manage with the follow-up database for implementation of
recommendations is electronically. Back-ups should be made on a regular basis and stored
on a separate drive or other relevant software to be kept in a protected place from fire, flood
or other force major.

The key activities that relate to the database for tracking the implementation of the
recommendations are shown in steps which are presented below.

Step 1: Development of database for Performance Audit recommendations

NAO should develop a database containing audit recommendations per each audit report2.
This database should be updated by the director of respective department who is
responsible for maintaining and developing statistics based on answers received from the
auditees. Database is drawn in excel sheets and a key is applied in sensitive columns which
will be kept only by managers of this database to make sure that no unauthorized changes
will be made in the database.

Status of audit recommendations are as following:

 Implemented – The recommendation is successfully implemented.

 Started Implementation – Some measures have been taken towards the

implementation of recommendation, but the progress of implementation is still in an
early stage.

2 Annex 1 of this follow-up PA Guidelines

11
  Final Stage Implementation - Rightful measures have been taken from the auditee to
implement the recommendation and the implementation is at the final phase to be
fully completed.

 Not started Implementation – No measures have been taken by the auditee to

implement the recommendation so far but an action plan has been presented.

 Not addressed – No measures were taken and no action plan has been presented for
the implementation of recommendations.

 No longer applicable – Because of the changes within the auditee structure or for any
other reason, the recommendation is no longer relevant to the auditee.

Performance Audit Reports will be followed-up in semi-annual bases, while summarized

reports will be produced in annual bases. Database contains the following information:

Number of recommendations shows the recommendations order number;

Audit report number shows the number of specific report in order to enable follow-up of
respective audit;
Recommendations in the data base should be written in a summarized comprehensive and
understandable form for the users of the database showing clearly what the aim of the
recommendation is. This column will help the NAO management to organize follow-up
process by filtering specific audit reports to follow-up and also have a clear implementation
picture of recommendations of respective audit.
Responsible entity shows the entity responsible for implementation of specific
recommendation. If the recommendation is issued to a group of institutions (ie. All
municipalities), then follow-up should be performed with all municipalities. This column
will help the NAO management to organize follow-up process by filtering entities or
groups of entities to follow-up and also have a clear picture about their performance.
Due date it has two columns separating original due date and current due date. Original
due date shows the due date set in action plan after the audit report was issued while
current due date shows subsequent due dates if the entity has failed to implement
recommendation in the original due date.
Status prior report shows the status of specific recommendation from the previous semi-
annual follow-up process.
Status current report shows the status of specific recommendation if failed to be
implemented in previous semi-annual follow-up process.
Delays: Delays in the implementation of the recommendations are presented by specifying
the number of months (a month when the delay is for more than a month) or years of the
specific delay.
Management response column is used to show management response if there were any
reasons for not implementing recommendation or actions in the process to implement the
recommendation.
Comments by Auditors is open for the auditors in charge for follow-up where is shown
the auditors opinion on the management response and the status of implementation.

12
Tables in the summary of recommendations status based on audit reports are linked with
formulas in order to automatically generate the status of each followed-up
recommendations calculating automatically the number of open status per each
recommendation3. The overall ‘open’ status shows that respective audit report needs
further follow-up until all recommendations are implemented or no longer applicable. In
order to easy handle with the database during follow-up work, each status of
recommendation implementation is indicated with a specific color representing the
respective status.

Step 2: Issuance of Performance Audit Report

NAO Letter accompanying report issued to auditee(s) should request auditee(s) to provide
NAO response within 30 days of report issuance on actions being taken to address each
audit report recommendations. The auditee response for each recommendation should:
a) Advise whether or not agrees with the report recommendation;
b) If it does not agree with the recommendation explain why;
c) For the agreed recommendations, explain what actions have been or will be taken to
implement the recommendation and, if fully implemented, provide appropriate
documentation to support implementation of the recommendation;
d) For those recommendations it agrees with but have not been fully implemented,
explain the status of the implementation, what additional actions are needed to fully
implement the recommendation, and provide a target date as to when the
recommendation will be fully implemented; and
e) Identify a contact point within the organization who can respond to questions the NAO
may have concerning the status of the recommendations. Include contact info: name,
phone number and email address.
Note: Preliminary Information concerning the audit recommendations should be entered into the
data base upon report issuance. Some information, such as the status and target date for
implementation will not be available at this time.

Step 3: Receipt of Response from Auditee Regarding Actions Taken to Address

Audit Recommendation
a) Information provided by the auditee needs to be reviewed to determine the status of
the recommendation and whether or not the recommendation can be considered
implemented (or closed). Suggest that the performance audit team responsible for the
audit report in question make this determination.

3 Annex 2 of this follow-up PA Guidelines

13
 b) Based on the initial submission of information by the auditee and the review of this
information by the audit team, the team should determine the status of the audit
recommendations in question and provide this status information to whoever is
responsible for maintaining the audit recommendation data base.
c) If determination is made that additional information is needed from the auditee, the
audit team should contact the designated auditee contact person and request the
additional information, explaining why additional information is needed, and giving a
date when the information should be provided to NAO.
d) Once the additional requested information is received, the audit team should review
that information and make a determination as to whether the information is adequate
and the status of the recommendation in question. If information submitted by the
auditee is still not considered adequate, the audit team should again contact the auditee
and explain why the provided information is not adequate and what additional
information is needed and when. (At this point it would probably be advantageous for
the audit team to meet in person with the auditee to resolve any issues).

Step 4: Semi-annual follow-ups on outstanding recommendations

a) At the end of each calendar semi-annual, the status of audit recommendations will be
followed while a report should be produced in annual bases listing the audit
recommendations (by report and responsible entity) that are still outstanding (not
implemented).
b) For those audit recommendations that have been outstanding more than six months
and for which the NAO has no information on the status of implementation for the last
six months. Performance audit department either methodology department should
contact the auditee (either in writing by email or letter or by phone) and request the
auditee provide updated information on actions being taken to implement the
recommendation and the target date for completion.
c) Information obtained during the semi-annual follow-up should be used to update the
status of recommendations contained in the data base.

Step 5: Issuance of semi-annual reports on status of performance audit

recommendations
a) Performance Audit Department and/or the Methodology Department annually
should issue summarized reports on the status of performance audit recommendations
(these reports are described in Step 1 above).

14
Annex 1, Table 1: Database of Follow-up Audit Reccomendations

As of 30.01.20XX As of 30.06.20XX
No Audit Recommendation Responsible Report Due date Recommendation Overdue Management Comments by Recommendation
Report Entity date (Action Status Response Auditors Status
Title Plan) (Evidence when
needed
1 12345678 A MMPH 01.01.20XX 01.01.20XX Started 12 months Comment Comment Implemented
Implementation
2 12345678 B MMPH 01.01.20XX 01.01.20XX Not Addressed More than 2 Comment Comment Not Addressed
years
3 91011121 C TGjK 01.01.20XX 01.01.20XX Not started 5 months Comment Comment Implemented
Implementation
4 91011121 D TGjK 01.01.20XX 01.01.20XX Started Implemented Comment Comment Implemented
Implementation on time
5 14512389 E KOb 01.01.20XX 01.01.20XX Final Stage Implemented Comment Comment Implemented
Implementation on time
6 14512389 F Kgja 01.01.20XX 01.01.20XX Started 2 months Comment Comment Final Stage
Implementation Implementation
7 45983587 G KPr 01.01.20XX 01.01.20XX Not Addressed More than 3 Comment Comment No Longer
years Applicable
8 155468864 H KOb 01.01.20XX 01.01.20XX Not started 3 months Comment Comment Implemented
Implementation
9 155468864 I KOb 01.01.20XX 01.01.20XX Not Addressed More than 2 Comment Comment Not started
years Implementation
10 155468864 J KOb 01.01.20XX 01.01.20XX Final Stage Implemented Comment Comment Implemented
Implementation on time
11 155468864 K KOb 01.01.20XX 01.01.20XX Not Addressed 12 months Comment Comment Implemented

Source: Follow up database, Follow up Tab, Detailed List of Recommendation Follow Ups

15
Annex 2, Table 2: Audit reports with implementation of recommendations and implementation in progress as of date 20xx

Recommendation
status as of Jan
30, 20xx Recommendation status as of June 30, 20xx
Final Stage
Total no. of Started Not started Not No Longer
Audit No. Auditee Overall Status Implemented Implementati Overall Status
Recommendations Implementation Implementation Addressed Applicable
on
12345678 MMPH 2 OPEN 1 0 0 0 1 0 OPEN
14512389 KOb 2 OPEN 1 1 0 0 0 0 OPEN
45983587 KPr 1 OPEN 0 0 0 0 0 1 OPEN
91011121 TGjK 2 OPEN 2 0 0 0 0 0 COMPLETE
155468864 KOb 4 OPEN 3 0 0 1 0 0 OPEN
#N/A 0 0 0 0 0 0 0 0 0
#N/A 0 0 0 0 0 0 0 0 0
#N/A 0 0 0 0 0 0 0 0 0
#N/A 0 0 0 0 0 0 0 0 0
#N/A 0 0 0 0 0 0 0 0 0
#N/A 0 0 0 0 0 0 0 0 0
#N/A 0 0 0 0 0 0 0 0 0
#N/A 0 0 0 0 0 0 0 0 0
#N/A 0 0 0 0 0 0 0 0 0
#N/A 0 0 0 0 0 0 0 0 0
#N/A 0 0 0 0 0 0 0 0 0
#N/A 0 0 0 0 0 0 0 0 0
#N/A 0 0 0 0 0 0 0 0 0
#N/A 0 0 0 0 0 0 0 0 0
#N/A 0 0 0 0 0 0 0 0 0
#N/A 0 0 0 0 0 0 0 0 0
#N/A 0 0 0 0 0 0 0 0 0
#N/A 0 0 0 0 0 0 0 0 0
#N/A 0 0 0 0 0 0 0 0 0
#N/A 0 0 0 0 0 0 0 0 0
Total 11 0 7 1 0 1 1 1
Source: Follow up database, Summary Tab, Summary of recommendations status based on Audit Report - Generated list from Follow_Up Tab

16
ZYRA KOMBЁTARE E AUDITIMIT –NACIONALNA KANCELARIJA REVIZIJE - NATIONAL AUDIT OFFICE

Annex 3. Flow Chart of NAO Follow-Up audit process

17