09 GBT LR Ver VI Ver 1nov2018 Final Adjusted

WHO GLOBAL BENCHMARKING TOOL (GBT) FOR EVALUATION
OF NATIONAL REGULATORY SYSTEM OF MEDICAL PRODUCTS
NRA LOT RELEASE (LR): INDICATORS AND FACT SHEETS

Revision VI version 1
November 2018
WHO global benchmarking tool rev. VI ver. 1, NRA Lot Release (LR): indicators and fact sheets
Page 2 of 33

Function: 09‐ NRA LOT RELEASE (LR)
Description: National Regulatory Authority (NRA) lot release (also called official authority batch release) is a non‐common
regulatory function that does not apply to all medical products. Lot release is a system specifically established
for the regulatory release of specified biological products. The goal of the regulatory function is to ensure the
quality, safety and efficacy of biological products through a regulatory release system. Lot release is done on a
lot‐by‐lot basis and takes into account the nature and inherent variability of these products.
NRAs should have the legal mandate to perform independent lot release. They should develop and implement
the necessary policies, guidelines, procedures and forms in line with World Health Organization and major
international guidelines.
In the case of vaccines, different approaches are currently used for conducting lot release. The options
include: review of the summary protocols only, review of the summary protocols combined with independent
testing (i.e., either full or selected testing), and recognition and acceptance of lot release certificates from the
responsible NRA or National Control Laboratory (NCL). The NRA or NCL has the responsibility to decide on an
appropriate strategy for each vaccine. The decision should take into consideration the nature of the vaccine,
the post‐marketing experience for each vaccine (including production history and safety profile), and the
availability of other independent evidence of product quality.
Indicator: LR01 Legal provisions, regulations and guidelines required to define regulatory framework of independent lot release
by the NRA.
Objective: The objective of this indicator is to ensure that NRA activities are backed by legislation and provisions for independent lot release.
The lot release of vaccines by regulatory authorities is part of the regulation of vaccines and involves the independent assessment
of each lot of a licensed vaccine before it is released onto the market. The impact of using substandard lots may not be known for
a very long time (i.e., years). Similarly, safety issues with a particular lot may not be known immediately (i.e., within a few hours)
after administration. Due to these delays, there could be a drastic impact if a large number of healthy persons receive a vaccine
Page 3 of 33

before a problem is recognized. For these reasons, a careful, independent review of manufacturing and quality control data on
every lot is necessary before a lot is marketed.
Category: 01. Legal provisions, regulations and guidelines
Sub Indicator: LR01.01: Legal provisions and regulations exist to conduct and enforce lot release for all vaccines.
Maturity Level: 1
Scope: 1. Vaccines
Description: The assessor should verify the existence of legal provisions, regulations or other administrative provisions that give the National
Regulatory (NRA) authority to implement and enforce lot release for vaccines. The legal provisions should define the responsible
officer authorized to sign the regulatory lot release certificate.
The assessor should verify that the legal basis to perform lot release applies for all vaccines marketed in the country and includes
Expanded Programme on Immunization and non‐ Expanded Programme on Immunization vaccines as well as imported and
domestically‐ produced vaccines. Current approaches for conducting lot release of vaccines include: review of the summary
protocol only; review of the summary protocol with independent testing (i.e., either full or selected testing); and recognition and
acceptance of lot release certificates from the responsible NRA or National Control Laboratory (NCL). These approaches are not
mutually exclusive and different approaches may be used for different products in the same country. It is the responsibility of the
NRA or NCL to decide on the appropriate strategy for each vaccine, taking into consideration the nature of the vaccine, the post‐
marketing experience (including production history and safety profile) for the vaccine, and the availability of other independent
evidence of product quality.
The assessor should verify that the criteria have been defined when the NRA may elect to follow non‐routine procedures for lot
release. Reasons for exemption from lot release could include, for example, a shortage of a product on the market or the need to
import a non‐authorized product for a defined time. The assessor should verify that legal provisions and regulations permit the use
of a fast ‐track mechanism with specific lot release requirements.
Objective: The objective of this sub‐indicator is to ensure that lot release of vaccines is part of the regulatory framework and involves the
independent assessment of each lot of a licensed vaccine before it is released on to the market.
Each country should establish the national guidelines for lot release. The guidelines should define all required procedures, from
the submission of the lots for release to the issuance of lot release certificates.
All vaccines lots should be released by an NRA or NCL; however, in defined exceptional circumstances (e.g., a public health
Page 4 of 33

emergency), exemptions could be allowed.
Requirement: Lot release regulatory framework and regulations for lot release of all vaccines.
Evidence to review: The assessor should ask for and review:
1. Legal provisions for lot release;
2. Documented procedures and records to ensure that staff participating in lot release function contribute, as appropriate, to
other regulatory functions;
3. Legal provisions and regulations that require lot release for all vaccines;
4. Legal provisions and regulations for exemptions from lot release exist;
5. Legal provisions and regulations defining fast‐track mechanism for lot release;
6. List of products, if any, where the routine procedure for independent lot release may have not been followed.
References: 1. Guidelines for independent lot release of vaccines by regulatory authorities, (42),
(https://www.who.int/biologicals/WHO_ECBS/en/ and http://apps.who.int/medicinedocs/en)
Framework: “Structure/Foundation/Input”
Rating Scale:  NOT IMPLEMENTED (NI): There is no legal basis for lot release.

 ONGOING IMPLEMENTATION (OI): The NRA is preparing to establish the legal basis, but there is no evidence of results from
such activities.

 PARTIALLY IMPLEMENTED (PI): The legal basis was established recently and is at the implementation stage, so this practice is
not consolidated yet.

 IMPLEMENTED (I): The NRA has the legal provisions and also consistently maintains documentation of related activities over
time.
Limitations and remarks: Scoring this sub‐indicator as "not applicable NA" is excluded (i.e. this sub‐indicator will always apply for all benchmarked NRAs).
Sub Indicator: LR01.02: Acceptance policy and criteria for lot release performed by another NRA are documented.
Page 5 of 33

Maturity Level: 2
Scope: 1. Vaccines
Description: If the country does not have the capacity to perform lot release on its own, the assessor should determine whether the country
recognizes certificates from the country of origin or from other competent NRAs and NCLs.
Different processes are used for implementing this recognition. Examples of these processes include establishment of a list of
countries that are acceptable to the importing country or creation of mutual recognition agreements. Establishment of mutual
recognition agreements is a legal approach. Many NRAs and NCLs use such agreements to: enhance international regulatory
cooperation in order to maintain high standards of product safety and quality; reduce the regulatory burden for NRAs and NCLs
and manufacturers; and improve the free flow of goods and increase the accessibility of medical products globally. Reciprocal
mutual recognition of release certificates involves a number of legal aspects that should be addressed. However, the key to
successful mutual recognition is the building of mutual confidence among the interested parties. This requires strong collaboration
and communication among the different NRAs and NCLs and a good level of transparency. Situations may exist where a two‐way
recognition of certificates or test results is not possible, owing to technical or other limitations. However, even in cases where
reciprocity is not attainable, an NRA or NCL may still wish to recognize a release certificate from another NRA or NCL. This should
be possible, provided the releasing NRA or NCL has clearly established procedures that are transparent and relevant to the NRA or
NCL wishing to recognize the certificate or test results.
Objective: The objective of this sub‐indicator is to ensure that, in cases when a lot has already been released by another NRA or NCL, it may
be possible for the NRA to accept that lot for release on the basis of the existing release certificate.
Requirement: Recognition of other NRA’s decision
1. Documented provisions and criteria for recognition of decisions, reports or certificates from other authorities;
2. Documentation for the recognition decision process including its rationale and reasoning;
3. List of NRAs or NCLs that are considered acceptable
Page 6 of 33

Rating Scale:  NOT IMPLEMENTED (NI): There is no evidence of the policies mentioned in the indicator.

 ONGOING IMPLEMENTATION (OI): The NRA is preparing to draft such policies or procedures, but there is no evidence of
results from such activities.

 PARTIALLY IMPLEMENTED (PI): The policy was established recently and is at the implementation stage, so this practice is not
consolidated yet.

 IMPLEMENTED (I): The NRA has such policies and procedures and also consistently maintains documentation of related
activities over time.
Limitations and remarks:  In some counties, the legal framework prevents the NRA or NCL from recognizing decisions from other NRAs. In such cases,
this sub‐indicator should be scored as non‐ applicable.
 The product’s manufacturer should be involved in the establishment of an agreement for sharing product information, since
there are issues of confidentiality that need to be addressed.

Indicator: LR02 Arrangement for effective organization and good governance.
Objective: The objective of this indicator is to establish that structures are in place at the organizational and governance levels to promote
effective intra‐ and inter‐NRA and NCL relationships and efficient management of information traffic.
The quality, safety and efficacy of a medical product, such as a vaccine, are the responsibility of the manufacturer. The regulatory
authority of the country is responsible for establishing procedures to ensure that this responsibility is met. The same requirements
for regulatory oversight should apply to the production of all vaccines, regardless of whether they are intended for domestic use
or for export.
Category: 01. Legal provisions, regulations and guidelines
Sub Indicator: LR02.01: There is a defined organizational structure with clear responsibilities to conduct independent lot release
Page 7 of 33

activities.
Maturity Level: 2
Scope: 1. Vaccines
Description: The assessor should verify that roles and responsibilities for all regulatory entities involved in the independent lot release are
documented and implemented. It is critical that the roles and responsibilities of both the NRA and the NCL are clearly defined,
particularly when they are separate entities. When all elements are available for final evaluation, a formal decision‐making process
should be in place to decide whether the lot can be released.
Objective: The objective of this sub‐indicator is to ensure there are appropriate structures in place, with clearly defined roles and
responsibilities for each entity involved in lot release activities.
Requirement: Roles and responsibilities in regard to lot release function
1. Written document defining organizational structure, with clearly‐defined responsibilities, to conduct independent lot release
activities;
2. Documented evidence designating the post holder responsible for signing lot release certificates;
3. Organization chart including the department, unit, group or post holder responsible for independent lot release.
2. Quality management systems — Requirements. International Standard ISO 9001:2015. International Organization for
Standardization, 2015, (4), (https://www.iso.org/iso‐9001‐quality‐management.html)
Rating Scale:  NOT IMPLEMENTED (NI): There is no defined structure with clear responsibilities to conduct independent lot release activities.

 ONGOING IMPLEMENTATION (OI): The NRA is preparing to establish the legal basis, but there is no evidence of results from
such activities.
Page 8 of 33

 PARTIALLY IMPLEMENTED (PI): The legal basis was established recently and is at the implementation stage, so this practice is

 IMPLEMENTED (I): The NRA has defined a structure with clear responsibilities to conduct independent lot release activities and
consistently maintains documentation of related activities over time.
Sub Indicator: LR02.02: Documented procedures are implemented to ensure coordination and communication among all regulatory
entities involved in independent lot release.
Maturity Level: 3
Scope: 1. Vaccines
Description: The assessor should review the adequacy of the implemented coordination mechanisms.
Lot release is one component of a regulatory framework which includes marketing authorization (MA), good manufacturing
practices (GMP) inspection, and post‐marketing surveillance. The relationship between NRA and NCL varies from country to
country, but in all cases, it is essential that the different entities of the regulatory structure interact and exchange information
effectively.
Good coordination and communication are needed, especially when different regulatory entities are involved in this process.
Objective: The objective of this sub‐indicator is to ensure that documented procedures, structures, and mechanisms are implemented for
proper relationships within and among entities involved in lot release, quality control (QC) laboratory testing, MA and facilities
inspection. These coordination mechanisms will ensure effective and efficient exchange of information for lot release activities.
Requirement: Communication among all regulatory entities relevant to independent lot release.
1. Documentation that defines role and responsibilities;
2. Documentation for established communication channels (i.e., written documents that describe methodology and procedures);
3. Records of relevant communications;
Page 9 of 33

4. Records of regulatory actions taken based on lot release findings
Rating Scale:  NOT IMPLEMENTED (NI): There are no documented procedures to ensure coordination and communication among all
regulatory entities involved in independent lot release.

 ONGOING IMPLEMENTATION (OI): The NRA is preparing to establish the procedure, but there is no evidence of results from
such activities.

 PARTIALLY IMPLEMENTED (PI): The procedure was established recently and is at the implementation stage, so this practice is

 IMPLEMENTED (I): The NRA has documented procedures and also consistently maintains documentation of related activities
over time.
Indicator: LR03 Human resources to perform NRA lot release
Objective: The objective of this indicator is to ensure that all entities within an NRA are adequately resourced with a trained, experienced and
skilled workforce, and are empowered to fully perform the function of independent lot release. This will ensure that NRA lot
release activities are performed in accordance with international best practices.
The objective of this indicator is to evaluate the human resource capacity of the entities with respect to the number of personnel,
the skills and experience of the personnel, and the overall composition the workforce, with the goal of evaluating whether the
workforce possesses the specific expertise required to perform the NRA lot release function.
Category: Resources (HR, FR, infrastructure and equipment)
Page 10 of 33

Sub Indicator: LR03.01: Sufficient competent staff (i.e., education, training, skills and experience) are assigned to perform NRA lot
release activities
Maturity Level: 3
Scope: 1. Vaccines
Description: The assessor should verify that the human resources assigned to perform NRA lot release activities should be sufficient with
respect to numbers and competent with respect to the requisite skills, education, experience and training. There should be
technical documents and standard operating procedures (SOPs) that provide guidance on the required background for NRA lot
release activities and that consider the requirements for educational background, competencies, skills, experience, and training.
The assessor should verify that the NRA estimated the number of staff required to effectively and efficiently perform NRA lot
release function and that the NRA actually recruited that number. In addition, the assessor should verify that these competency
requirements are well‐established and maintained by the NRA. Metrics and statistics on the different activities performed as well
as performance indicators can be used for estimating the adequacy of the number of the assigned staff. The assessor should also
verify that the competency of the assigned staff is built, maintained and improved through recruitment as well as continuous on‐
the‐job training.
Objective: The objective of this sub‐indicator is to ensure the existing human resources for NRA lot release is sufficient, in terms of numbers,
experience, and specific competencies, to perform all the activities along the entire NRA lot release chain.
Requirement: Sufficient number of competent human resources in charge of NRA lot release activities.
1. Evidence that the number of staff members involved in each of the documented activities along the entire NRA lot release
process flow is adequate.
2. Evidence that the systems and structures are in place to ensure appropriate placement of staff with respect to competence and
skills.
3. Evidence that the system and structures have been implemented. The documentation should include the records to verify that
the staff competence is appropriate for the job requirements.
4. Evidence that the professional profiles of the human resources engaged in NRA lot release activities are appropriate with
respect to education, skills, and expertise, to perform a particular function along the NRA lot release chain. Documentation should
Page 11 of 33

include a list of the requisite skills and training for each position.
5. Recruitment plan.
References: 1. National drug regulatory legislation: guiding principles for small drug regulatory authorities. In: WHO Expert Committee on
Specifications for Pharmaceutical Preparations: thirty‐fifth report. Geneva: World Health Organization; 1999: Annex 8 (WHO
Technical Report Series, No. 885), (1), (http://apps.who.int/medicinedocs/documents/s21964en/s21964en.pdf)
Framework: Structure/Foundation/Input
Rating Scale:  NOT IMPLEMENTED (NI): The NRA does not have enough competent staff (i.e., education, training, skills and experience) to
perform NRA lot release activities

 ONGOING IMPLEMENTATION (OI): The NRA has recently developed a plan to recruit adequate competent staff; however, the
plan has not been implemented.

 PARTIALLY IMPLEMENTED (PI): The NRA has initiated the implementation of the human resources development plan;
however, there is need to complete the competency profile.

 IMPLEMENTED (I): The NRA has a sustained number of competent staff (i.e., education, training, skills and experience)
assigned to perform NRA lot release activities
Limitations and remarks:  Assessment of the adequacy and appropriateness of the number of staff members is quite subjective and should be linked to
some process or output indicators. When estimating staff adequacy, the assessor should consider the workload, backlog, and
delays in delivery based on established timeframes.
 Scoring this sub‐indicator as "not applicable NA" is excluded (i.e. this sub‐indicator will always apply for all benchmarked
NRAs).
Sub Indicator: LR03.02: Duties, functions, and responsibilities of the staff in charge of NRA lot release activities are established and
updated in the respective job descriptions
Maturity Level: 3
Page 12 of 33

Scope: 1. Vaccines
Description: The assessor should verify that procedures are in place to maintain a current and updated structure for managing job descriptions
for personnel participating in NRA lot release activities. In addition, job descriptions should address current staff duties,
responsibilities and the requisite competencies. A job description with this format and content should be established and
implemented for all staff. The management of job descriptions should be supported by a guidance document that provides
direction on when and how to update the information, and where the information should be kept for easy access. The guidance
document should present the appropriate duties and responsibilities that are assigned to each member of the organization
involved in NRA lot release activities. Thus, the professional profiles of staff are reflected in their respective roles and
responsibilities within the NRA. There should be procedures to guide responsible persons to document that duties, functions and
responsibilities are revised and kept up to date. In addition, procedures should be available to guide the keeping and documenting
of up to date work schedules and enforcing the implementation of the documented guidelines and procedures.
Objective: The objective of this sub‐indicator is to ensure that duties and responsibilities of the staff are clear and well defined, that job
descriptions are kept up to date with current duties, functions and responsibilities, and that these activities are adequately
documented.
Requirement: Duties, roles and responsibilities of the staff relevant to NRA lot release activities.
1. Procedure and guidelines that guide placement of staff members within the NRA;
2. The professional profiles of staff (i.e., job descriptions) and documentation that they are related to their current roles and
duties;
3. The professional profiles of the external experts and documentation that the profiles provide a composition that is complete
and consistent with that prescribed in the legal provisions;
4. Procedures to guide the documentation of up to date duties and work schedules, and to enforce the implementation of the
documented guidelines and procedures;
5. Job descriptions for designated staff.
Page 13 of 33

Standardization, 2015 sub‐clause 5.1.1, (4), (https://www.iso.org/iso‐9001‐quality‐management.html)
3. Guiding principles for small national drug regulatory authorities. In: WHO Expert Committee on Specifications for
Pharmaceutical Preparations: thirty‐first report. Geneva: World Health Organization; 1990: Annex 6 (WHO Technical Report Series,
No. 790), (5), (http://apps.who.int/medicinedocs/documents/s21964en/s21964en.pdf)
4. Effective drug regulation: A multicountry study. Geneva: World Health Organization; 2002, (7),
(http://apps.who.int/medicinedocs/pdf/s2300e/s2300e.pdf)
Framework: Structure/Foundation/Input
Rating Scale:  NOT IMPLEMENTED (NI): There is no evidence of defined or established duties, functions, responsibilities, respective job
descriptions and necessary required competencies.

 ONGOING IMPLEMENTATION (OI): The NRA has recently drafted or developed the role and responsibilities document but it
has not yet been implemented.

 PARTIALLY IMPLEMENTED (PI): The NRA has initiated implementation of this requirement but it has not been defined or
followed for all staff or the roles and responsibilities documents, including staff job descriptions, are not up to date.

 IMPLEMENTED (I): The NRA has defined and established all required duties, functions, and responsibilities, and respective job
descriptions are up‐to‐date.
Sub Indicator: LR03.03: Training plan developed, implemented and updated at least once a year for staff in charge of NRA lot
release activities.
Maturity Level: 3
Scope: 1. Vaccines
Description: The assessor should verify that training plans are developed, implemented and updated at least once every year to reflect the
current situation by considering education and experience of the staff. The training plan should be complemented with guidelines
or similar documents that guide the development and implementation of training plans. The assessor should ensure that induction
Page 14 of 33

training for new staff as well as continued on‐ the‐job training for staff is planned and implemented. There should be procedures
to approve the training plan and the budget allocated for implementing and updating the training plan. The plans should present
clearly defined training goals and should include training in certain topics and skills to address identified deficiencies. Learning
objectives, training methods and activities, evidence of learning, and evaluation and assessment of training should be
documented. This documentation should confirm that the learning objectives were achieved and were designed to address
weaknesses within the entities. Procedures should be in place to ensure that a training plan is developed, implemented and
updated at least once every year. The assessor should verify that there is a system in place for monitoring the implementation and
effectiveness of the training plan and for documenting the skills acquired in training activities for internal and external experts.
Objective: The objective of this sub‐indicator is to ensure that a training plan for staff exists, and that it is implemented and updated
annually. Through the training plan, NRA can be sure that competency of staff in charge of NRA lot release activities is maintained
and enhanced.
Requirement: Implementation of training plan
1. Guidelines for development, implementation and annual update (i.e., at least once per year) of the training plan. Guidelines
should also provide for a mechanism to measure effectiveness of training.
2. Documentation for the system or structures used to approve the training plan and to evaluate the adequacy of the budget
allocated to the training activities.
3. The current or existing staff training plan (or matrix) for staff. The assessor should assess this in in relation to the respective
individual job descriptions.
4. SOP for developing and maintaining the training plan.
5. Evidence that the NRA has investigated and identified training needs.
6. List of trainings performed.
7. Example records for training activities.
References: 1. Effective drug regulation: A multicountry study. Geneva: World Health Organization; 2002, (7),
2. Quality systems requirements for national good manufacturing practice inspectorates. In: WHO Expert Committee on
Specifications for Pharmaceutical Preparations: thirty‐sixth report. Geneva: World Health Organization; 2002: Annex 8 (WHO
Page 15 of 33

Framework: Process
Rating Scale:  NOT IMPLEMENTED (NI): There is no systematic training program including training plan (or matrix).

 ONGOING IMPLEMENTATION (OI): The NRA has recently drafted or developed the training plan but there is no evidence of
implementation.

 PARTIALLY IMPLEMENTED (PI): The NRA has developed and initiated the training plan implementation. However, the NRA has
not fulfilled all required planned training or has been applying the plan for less than two years.

 IMPLEMENTED (I): The NRA has an updated training plan developed that is supported by adequate records to demonstrate
effective plan implementation, including induction training for new staff and routine on‐the‐job training for recruited staff.
Limitations and remarks:  Training plans must be updated regularly; ideally on an annual basis, but not less frequently that once every two years.
 Some regulatory functions may include many training activities that are not incorporated in the institutional training
programme. Such training normally is offered by invitation. In this case, the assessor should recognize reports from non‐
routine NRA lot release‐relevant training not included in the NRA training plan.
NRAs).
Sub Indicator: LR03.04: The NRA generates and maintains records of staff training activities and training effectiveness verification.
Maturity Level: 3
Scope: 1. Vaccines
Description: The assessor should verify that records of staff training that is performed or organized by the NRA are generated, maintained,
regularly updated. This activity should be supported by guidelines that direct the NRA to generate and maintain records of staff
training activities. Procedures should be in place to document and propose staff training needs and to allocate a budget for
continuous staff capacity building and development. The assessor should check that there is an evaluation or assessment
mechanism to verify the quality of learning, and to confirm that learning objectives are achieved. Documentation should include
an inventory (i.e., soft and/or hard) system that records all impactful and non‐impactful trainings and identifies all staff members
Page 16 of 33

who participated. A system to measure or estimate impact of trainings should be established.
Objective: The objective of this sub‐indicator is to ensure that training organized by the NRA or responsible regulatory authority is adequately
documented and that the training records are adequately maintained and kept. Staff training records are considered an integral
part of staff file and are a tool for measuring and tracking staff competency, development and adequacy.
Requirement: Training records
1. Guidelines or similar documents that guide the NRA to generate and maintain records of staff training activities;
2. Evaluations of training effectiveness;
3. The training inventory, and procedures for completing the inventory;
4. Examples of archived records of staff training, and procedures for the archiving system
Standardization, 2015 sub‐clause 5.1.1, (4), (https://www.iso.org/iso‐9001‐quality‐management.html)
3. Guiding principles for small national drug regulatory authorities. In: WHO Expert Committee on Specifications for
Pharmaceutical Preparations: thirty‐first report. Geneva: World Health Organization; 1990: Annex 6 (WHO Technical Report Series,
No. 790), (5), (http://apps.who.int/medicinedocs/documents/s21964en/s21964en.pdf)
4. Effective drug regulation: A multicountry study. Geneva: World Health Organization; 2002, (7),
Framework: Output
Rating Scale:  NOT IMPLEMENTED (NI): There is no evidence that the NRA generates and maintains records of staff training activities.

 ONGOING IMPLEMENTATION (OI): The NRA has recently initiated plans to generate, document and keep records of staff
training activities, however they are not yet followed.

 PARTIALLY IMPLEMENTED (PI): The NRA has recently initiated plans to generate, document and keep records of staff training
Page 17 of 33

activities but they are not fully followed for all training activities or they have been established for less than two years.

 IMPLEMENTED (I): The NRA generates and maintains records of staff training activities.
Limitations and remarks:  The assessor should note that some NRAs out‐source training including staff capacity development activities. In this case the
assessor should request the identity of the provider, as well as evaluations of the provider. The assessor may request the
professional profiles of tutors or resource persons used to offer training. The assessor may also request records covering the
archiving systems in place.
NRAs).
Indicator: LR04 Procedures established and implemented to perform NRA lot release.
Objective: The objective of this indicator is to ensure documented and implemented procedures to perform independent lot release.
The lot release of vaccines by regulatory authorities is part of the regulation of vaccines and involves the independent assessment
of each lot of a licensed vaccine before it is released onto the market. This assessment is based, at a minimum, on the review of
manufacturer’s summary protocols. Lot release may be supplemented by other documents such as the release certificates from
the responsible NRA or NCL. In some circumstances, this information is also supplemented by product testing that is independent
of the manufacturer’s quality control testing. The summary protocol content should follow existing guidelines from the World
Health Organization Technical Report Series or other internationally‐accepted guidelines. If accepted guidelines are not available,
the country should define the summary protocol template for the product.
Current approaches to conducting lot release of vaccines include: review of the summary protocols only, review of the summary
protocols combined with independent testing (i.e., either full or selected testing), and recognition and acceptance of lot release
certificates from the responsible NRA or NCL. These approaches are not mutually exclusive and different approaches may be used
for different products in the same country. It is the responsibility of the NRA or NCL to decide on an appropriate strategy for each
vaccine, taking into consideration the nature of the vaccine, the post‐marketing experience for the vaccine (including production
history and safety profile), and the availability of other independent evidence of product quality.
Category: 07. Regulatory process
Sub Indicator: LR04.01: Independent lot release is based, at a minimum, on summary lot protocol review and the appropriate
Page 18 of 33

documentation exists.
Maturity Level: 2
Scope: 1. Vaccines
Description: The assessor should verify that summary lot protocols are mandatorily required for the lot release of vaccines.
The assessor should make sure the NRA or NCL issue lot release certificates based, at a minimum, on the review of the summary
lot protocols issued by the manufacturer. The protocol review should be performed by comparing the critical data of each lot
(including testing data) to the licensed product specifications.
The assessor should verify that the SOP for summary protocol review describes the steps and criteria required for a complete
review of the summary protocol. The SOP should cover all review steps up to and including the final conclusion based on review of
the summary protocol. When needed, these steps may include a request for corrections from the manufacturer and a review of
corrected pages. In some cases, the NRA or NCL may elect to conduct an investigation before reaching a conclusion. The NRA or
NCL should produce a formal written conclusion regarding the summary protocol review. A summary decision form should be
filled out to verify that the product complied with approved specifications. This summary decision should be signed by the
responsible staff.
The assessor should determine whether the competent authority’s approach to independent lot release is appropriately described
in the NRA or NCL process charts. Procedures should cover the options used: release upon review of summary protocol only or
release upon review of summary protocol plus independent testing by the NCL. The procedures should also define how and by
whom the final decision is made. The summary decision form should define the specific option used and include a formal written
conclusion. SOPs or documents are necessary to cover the essential elements.
Objective: The objective of this sub‐indicator is to ensure that summary lot protocols are mandatorily required, at a minimum, for vaccines
lot release.
The manufacturers’ summary protocols summarize information taken from the production and QC to ensure that the lot meets
the specifications in the approved MA. In addition, summary protocols submitted to the NRA or NCL should be approved by the
person from the manufacturer who is designated as responsible for quality assurance or QC. In general, the format and content of
the protocol is finalized and approved by the NRA or NCL during the review of the license application. The format of the protocol
should be amended in response to changes in the approved production process. Amendments to the protocol should be approved
by the NRA or NCL.
Page 19 of 33

Requirement: Documentation for national lot release
1. SOP or guideline for performing lot release;
2. List of required documents for performing lot release;
3. Relevant written and enforced SOPs developed as part of the quality management system for reviewing lot release protocols for
each product;
4. Examples of lot release records;
5. Documented evidence of evaluation process.
2. WHO Expert Committee on Biological Standardization (ECBS), Vaccine‐specific standardization, (135),
(https://www.who.int/biologicals/vaccines/en/)
3. Guidelines for EU Official Control Authority Batch Release, European Directorate for the Quality of Medicines and Healthcare
(EDQM), Council of Europe, (104), (https://www.edqm.eu/en/human‐ocabr‐guidelines)
Framework: Process
Rating Scale:  NOT IMPLEMENTED (NI): No procedures for lot release based, at a minimum, on summary protocol review are available.

 ONGOING IMPLEMENTATION (OI): The NRA is preparing the procedures but there is no evidence of results from such activities
(i.e., reports or certificates).


 IMPLEMENTED (I): The NRA has procedures for lot release based, at a minimum, on summary protocol review and consistently
maintains documentation of the results over time.
Page 20 of 33

Sub Indicator: LR04.02: NRA or NCL staff involved in lot release have access to MA relevant files and updates.
Maturity Level: 3
Scope: 1. Vaccines
Description: The assessor should verify that the NRA or NCL staff involved in lot release has access to files and updates relevant to MA.
NRA or NCL lot release should be performed only for medical products that have a valid MA in which specifications have been
approved by the competent NRA or NCL of the country using the vaccine. In addition, the development and adoption of more
effective test methods should be encouraged; however, any changes in testing should be approved by the NRA or NCL. If a
different test method is used by the NRA or NCL, and if there is a discrepancy in test data between the manufacturer and the NRA
or NCL, then the approved test method defined in the MA should be used to resolve the issue.
Objective: The objective of this sub‐indicator is to ensure that specifications as described in the MA are used to judge the test results. Also,
there should be a mechanism in place to allow the testing staff of the NRA or NCL to be aware of the latest version of the
approved MA specifications.
In the decision‐making process for MA, the responsible NRA or NCL staff should be involved in assessing the test methods, validity
criteria and product specifications.
Requirement: Access to MA data for lot release
1. SOPs that define how MA data are considered for lot release;
2. Records of relevant updated information that was communicated to NRA or NCL.
Rating Scale:  NOT IMPLEMENTED (NI): No procedures to define how MA data are considered for lot release are available.

 ONGOING IMPLEMENTATION (OI): The NRA is preparing a procedure, but there is no evidence of results from such activities.
Page 21 of 33


 IMPLEMENTED (I): The NRA has procedures to define how MA data are considered for lot release, and also consistently
maintains documentation of the results of related activities over time.
Sub Indicator: LR04.03: Analysis of lot‐to‐lot consistency is conducted.
Maturity Level: 3
Scope: 1. Vaccines
Description: The assessor should verify that statistical analyses are conducted once sufficient data have been accumulated. Alert limits (i.e.,
warning limits) and action limits should be defined on the basis of statistical assessments and trend analyses of test data. In
general, when data are distributed normally, ±2 and ±3 standard deviations from the mean are set for the alert limits (i.e., warning
limits) and action limits respectively. The variability and precision of the test should be considered when defining the limits. Care
should be taken in interpreting such limits when they are based on small datasets. Trend analyses of key parameters may be
requested from manufacturers or from the responsible NRA or NCL. More complex statistical approaches can be used for trend
analyses when sufficient data and expertise are available, particularly when data are not normally distributed. In addition, a set of
data from a certain period (e.g., 6 months or 1 year) should be analyzed statistically and compared to data from the previous
period, in order to detect any significant differences, shifts, or trends.
In order to conduct an appropriate trend analysis, it is important to have data from an adequate number of release batches for
each product.
When the NRA or NCL does not receive consecutive lots, or when it receives only a small number of production lots, interpretation
of trends may require additional information (e.g., yearly biological product reports).
Objective: The objective of this sub‐indicator is to ensure that all critical quantitative data from QC testing, especially from potency testing,
from the manufacturer or other sources are used for trend analysis as an essential part of lot release.
Page 22 of 33

Requirement: Data monitoring
1. Procedures and mechanisms employed to ensure lot‐to‐lot consistency.
2. Documentation that these procedures are performed by NRA or NCL on regular basis.
References:
Framework: Output
Rating Scale:  NOT IMPLEMENTED (NI): No procedures for analysis of lot‐to‐lot consistency are available.

 ONGOING IMPLEMENTATION (OI): The NRA is preparing a procedure for analysis of lot‐to‐lot consistency but there is no
evidence of results from such activities.

not yet consolidated.

 IMPLEMENTED (I): The NRA has such procedures and also consistently maintains documentation of related activities over
time.
Limitations and remarks:  Information obtained from lot‐to ‐lot consistency analyses could be used to establish criteria for testing by the NCL.
NRAs).
Indicator: LR05 Mechanism for information‐sharing exists to promote transparency and accountability.
Objective: The objective of this indicator is to ensure that the mechanisms for independent lot release, including requirements and timelines,
are made public in a clear and transparent way. In addition, these mechanisms could contribute to informing the public about any
risks, e.g., with respect to access, availability or shortages of some medical products subject to national lot release. Public
transparency of decision‐making and resource‐management is one of the principles of truth and accountability.
Page 23 of 33

Category: 08. Transparency, accountability and communication
Sub Indicator: LR05.01: Results of lot release process are publicly available
Maturity Level: 4
Scope: 1. Vaccines
Description: The assessor should verify the existence of a list of product batches that pass or fail NRA lot release. In addition, the assessor
should verify that the NRA or NCL has the required authority and an available process to publish the list of product batches that
passed or failed NRA lot release. At a minimum, a publicly available website (or another form of communication) should be
available to publish the list of product batches that passed NRA lot release. The assessor should determine the frequency at which
this information is updated.
Objective: The objective of this sub‐indicator is to ensure that processes and procedures for lot release decision‐making, as well as the list of
product batches that pass or fail NRA lot release, are documented and available to the public.
Transparency enhances public trust, permits the timely application of corrective measures, prevents public use of unqualified
products, and promotes the timely identification of potential vulnerabilities due to possible acts of corruption.
Requirement: Results of lot release process available
1. List of product batches that pass or fail NRA lot release;
2. Published list of product batches released;
3. List of product batches released compared with the list of product batches failed;
4. Guideline or SOP that defines the process to publish the list of product batches that passed or failed NRA lot release.
References: 1. Good governance for medicines: Model framework. Updated version 2014. WHO, (110),
(https://www.who.int/medicines/areas/governance/ggm_modelframe_updated/en/)
2. Guidelines for independent lot release of vaccines by regulatory authorities, (42),
Page 24 of 33

Framework: Output
Rating Scale:  NOT IMPLEMENTED (NI): No results of lot release process are available.

 ONGOING IMPLEMENTATION (OI): The NRA is preparing a procedure but there is no evidence of results from such activities.

 PARTIALLY IMPLEMENTED (PI): The procedure or the list was established recently and is at the implementation stage, so this
practice is not yet consolidated.

 IMPLEMENTED (I): The NRA has the list with results of lot release process and consistently maintains documentation of related
activities over time.
Limitations and remarks:  In some countries, the legal framework does not allow publishing for the public results of product batches that fail lot release.
Nevertheless, in case of emergency or risk, the NRA or NCL should inform public in timely manner.
 There may be specific situations to take into consideration, e.g., during recalls or when specifications are different in country
of origin than in country of use.
Sub Indicator: LR05.02: Follow‐up and communication with involved parties, including the manufacturer, on issues of data quality.
Maturity Level: 3
Scope: 1. Vaccines
Description: The assessor should verify the establishment and implementation of communication procedures with involved parties, including
the manufacturer, on issues of data quality.
Good communication with the manufacturer of the product is an important element in developing an effective system. NCLs
should discuss with the manufacturer the transfer of assays, if required. This should begin as early as possible in the MA process,
to allow for transfer, qualification, and validation of the methodology prior to application of the method for lot release testing of
the first lot. It is also necessary to establish documented and approved procedures and guidelines for performance of lot release
testing, both for internal use and for transparency with regard to partners, including other NCLs and the manufacturer of the
product.
A procedure to communicate QC and national lot release issues should be developed by the NRA or NCL. These procedures may
Page 25 of 33

include formal notifications by memorandum or letter, email communications, or minutes of telephone discussions.
Manufacturers’ responses should be reviewed and documented when making the decision on the lot. This response can include
submission by the manufacturer of a corrected page or revision of the summary protocol. These corrections should then be
properly traced by the NRA or NCL as per good documentation practices. Depending upon the nature and severity of the
discrepancies or errors, the manufacturer may be asked to perform an investigation to determine the root cause of the issues, and
to initiate any corrective and preventive actions required to avoid similar problems in the future.
A feedback mechanism from the NCL to the NRA, the GMP inspectorate and the MA staff is highly advisable, in order to coordinate
and optimize regulatory actions (e.g., encouraging license variations or refinements in product specification based on trend
analyses).
Objective: The objective of this sub‐indicator is to ensure that any discrepancies, errors or out‐of‐ specification results found in the summary
protocol submitted are documented and verified before they are communicated to the manufacturer. The manufacturer should be
notified when an out‐of‐specification result is confirmed, and exchanges should ensue to try to identify the cause of the
discrepancy.
Requirement: Follow‐up and communication on issues of data quality
1. Evidence that the NRA or NCL has a suitable mechanism to follow up and communicate on issues related to data quality with all
involved parties such as the manufacturer, importer, wholesaler or user of the products.
2. Written documents that describe the process;
3. Records of any actions.
Framework: Process
Rating Scale:  NOT IMPLEMENTED (NI): No procedures or reports are available regarding follow‐up and communication with involved
parties, including the manufacturer, on issues of data quality.

 ONGOING IMPLEMENTATION (OI): The NRA is preparing a procedure for this but there is no evidence of results from such
activities.
Page 26 of 33


 IMPLEMENTED (I): The NRA has such procedures and also consistently maintains records of the results of related activities
over time.
Limitations and remarks:  For imported lots, communication with the NRA of the producing or releasing country may be required. For producing or
releasing countries, communication with the country inspectorate may be required. Such information exchange can help to
evaluate the corrective and preventive actions introduced by the manufacturer. When needed, confidentiality issues should be
taken into consideration during the communication process.
In some countries, communications among government, manufacturer and NRA are used, for planning purposes, for all batch
release processes and not only for product analysis.
NRAs).
Indicator: LR06 Mechanism in place to monitor regulatory performance and output.
Objective: The objective of this indicator is to ensure the implementation of a system or mechanism for monitoring regulatory performance
and output of the independent lot release function.
Category: 09. Monitoring progress and assessing outcomes and impact
Sub Indicator: LR06.01: Lot release records, reports and certificates available.
Maturity Level: 3
Scope: 1. Vaccines
Description: The assessor should verify that there is requirement to issue a certificate of release for all vaccines that have undergone lot
release. This release certificate is issued by the responsible NRA or NCL on the basis of, at a minimum, a review of the lot summary
protocol for the relevant lot. The assessor should verify that lot‐to‐lot consistency is appropriately analyzed and documented by
Page 27 of 33

the NRA or NCL on regular basis.
Objective: The objective of this sub‐indicator is to ensure that all documentation related to independent lot release is available and readily
accessible to internal staff whenever needed. A general lot release process chart that outlines the lot approval process and
identifies the persons responsible for each activity should be available. Documentation supporting compliance with approved
specifications (i.e., summary protocol review and test reports, if applicable) should also be available.
Requirement: Documentation system
1. SOP describing the acceptance criteria for NCL test results;
2. Records of all the individual test results from the certificates of analysis (i.e., for all lots that have undergone lot release);
3. Records of evaluation of lot release including reports and certificates;

Framework: Output
Rating Scale:  NOT IMPLEMENTED (NI): No lot release records, reports or certificates are available.

 ONGOING IMPLEMENTATION (OI): The NRA is preparing a procedure for lot release but there is no evidence of results from
such activities (i.e., reports or certificates).

not yet consolidated yet.

 IMPLEMENTED (I): The NRA has procedures, records, reports and certificates for lot release and also consistently maintains
documentation of related activities over time.
Page 28 of 33

Sub Indicator: LR06.02: Corrective actions taken in case of deviations due to laboratory error.
Maturity Level: 3
Scope: 1. Vaccines
Description: The assessor should review that appropriate corrective actions are taken by the NCL in case of deviations (i.e. significant
discrepancy between NCL testing results and manufacturer results due to laboratory or operator error and not due to product
quality). Depending upon the nature and severity of the discrepancies or errors, the NCL may be asked to perform an investigation
to determine the root cause of the issues, including steps for corrections, corrective actions and/or preventive actions to avoid
recurrence or similar problems in the future.
Objective: The objective of this indicator is to confirm that corrective and preventive actions are taken, necessary improvements are made,
and the effectiveness of the actions is confirmed. The NRA or NCL should determine opportunities for improvement and
implement any necessary actions in case of a deviation. These should include improving products and services to meet
requirements as well as addressing future needs and expectations. Responsive actions can include corrections, corrective actions,
preventive actions, continued improvements, innovations, and re‐organizations.
Requirement: Corrective and preventive actions system
1. Evidence for availability of procedures for corrections, corrective actions and preventive actions;
2. Examples of corrections, corrective actions and preventive actions.
3. Evidence for implementation of corrections, corrective actions and preventive actions.
Framework: Process
Page 29 of 33

Rating Scale:  NOT IMPLEMENTED (NI): No procedures or reports regarding corrections, corrective actions and continued improvements are
available.

 ONGOING IMPLEMENTATION (OI): The NRA is preparing a procedure for this, but there is no evidence of results from such
activities.


 IMPLEMENTED (I): The NRA has such procedures and also consistently maintains documentation of related activities over
time.
Limitations and remarks:  A selected sample of yearly biological product reports should be reviewed as well. This is a report that is submitted annually to
the NRA or NCL by manufacturers and that contains production information on both bulk and final lots. The report should
include test methods and results, reasons for any recalls and corrective actions taken, and any pertinent post‐marketing
information.
NRAs).
Sub Indicator: LR06.03: Regulatory action taken in case of product non‐compliance.
Maturity Level: 3
Scope: 1. Vaccines
Description: The assessor should verify that in case of non‐compliances, the NRA or NCL should confirm, through appropriate laboratory
investigation, that the non‐compliant results reflect the quality of the lot tested and is not due either to an analytical error by the
NCL or to the influence of variables unrelated to the product. Once confirmed, the manufacturer should be notified as soon as
possible with prompt exchanges of information to try to identify the cause of the discrepancy through manufacturer investigation.
Test reports, including the results and outcomes of all of testing performed, should be prepared. These test reports should be used
in the final evaluation and decision‐making process for the lot (or lots) under consideration. A feedback mechanism from the NCL
to the NRA, the GMP inspectorate, and the MA staff is highly advisable in order to coordinate and optimize regulatory actions
Page 30 of 33

(e.g., batch recall, product withdrawal, MA revocation, encouraging license variations, or refinements in product specifications
based on trend analyses). For imported lots, communications between the NRAs or NCLs of the producing and releasing country
may be required. For producing and releasing countries, communications with the country inspectorate may be required. Such
information exchange can help to evaluate the corrective and preventive actions introduced by the manufacturer.
Objective: The objective of this sub‐indicator is to ensure that the NRA or NCL has an implemented procedure dealing with the actions to be
taken when test results do not comply with the specifications. These procedures should also cover the appropriate regulatory
actions to be taken.
Requirement: Non‐compliance action
1. Written documents that describe regulatory actions (e.g., revoking of authorization, recalling of product, or stopping of
importation) to be taken in case of non‐compliance.
2. Records of actions.
Framework: Process
Rating Scale:  NOT IMPLEMENTED (NI): No procedures or reports regarding regulatory actions taken in cases of non‐ compliance are
available.

 ONGOING IMPLEMENTATION (OI): The NRA is preparing a procedure for this, but there is no evidence of results from such
activities.


 IMPLEMENTED (I): The NRA has such procedures and also consistently maintains records of the results of related activities
Page 31 of 33

over time.
Sub Indicator: LR06.04: Performance indicators for national lot release activities are established and implemented
Maturity Level: 4
Scope: 1. Vaccines
Description: The assessor should verify the existence and implementation of performance indicators for different activities included under the
national lot release functions.
Specifically, the system should define key performance indicators (KPIs) along the entire national lot release activity chain and all
indicators should be adequately justified. For the purpose of clarity and consistency, established KPIs should be supported with
guidelines for monitoring and maintenance of the KPIs. The guidelines in turn should be supported by SOPs and tools that define
the procedures to be used for monitoring and evaluating the performance indicators and that define procedures and timelines for
reviewing and revising the indicators.
Examples of performance indicators for national lot release activities include, but are not limited to: number of released lots per
year, number of rejected lots per year, and average number of days to reach a decision on the release of the received batches.
Established KPIs might be qualitative, quantitative or combination of both. In general, quantitative indicators are preferred to
avoid bias or misinterpretation. However, qualitative indicators are also accepted. Qualitative indicators may or may not include
scoring or scaling to render them semi‐quantitative and thus more informative. The assessor should ensure that indicators are
measured on a regular basis to monitor progress and advancement.

In addition, the assessor should verify measured indicators are analyzed to identify trends or abnormalities. Justifications for any
identified abnormalities should be provided; when necessary, process optimizations should be introduced to avoid recurrence.
Objective: The objective of this sub‐indicator is to ensure that a system, mechanism, or procedure exists to require the NRA to establish
performance indicators along the entire national lot release chain. Additionally, the objective is to ensure that KPIs are actually
contributing to monitoring of regulatory performance, to measuring effectiveness of national lot release regulatory activities, and
to making any necessary adjustments or optimizations.
Requirement: KPIs for national lot release activities
Page 32 of 33

1. Documents supporting the system, mechanism, or procedure compelling the NRA to establish and implement performance
indicators along the entire national lot release activity chain.
2. Evidence that the performance indicators have been established and implemented, and that the members of staff involved in
the national lot release function are aware of the indicators and the guidelines and SOPs used for monitoring and evaluating their
performance.
3. The current performance indicators for national lot release activities
4. Analyses of the measured indicators along with the investigations done to identify trends or abnormalities.
5. Documentation for follow‐up of any observed abnormalities, including justifications for any identified abnormalities as well as
any process optimizations introduced to avoid recurrence.
References: 1. Quality management systems — Requirements. International Standard ISO 9001:2015. International Organization for
2. World Health Organization. European Observatory on Health Systems and Policies. Performance measurement for health
system improvement: experiences, challenges and Prospects, (125),
(https://www.who.int/management/district/performance/PerformanceMeasurementHealthSystemImprovement2.pdf)
3. World Health Organization. European Observatory on Health Systems and Policies. Health System Performance Comparison: an
agenda for policy, information and research, (126), (http://www.euro.who.int/__data/assets/pdf_file/0009/244836/Health‐
System‐Performance‐Comparison.pdf)
4. Quality management ‐‐ Quality of an organization ‐‐ Guidance to achieve sustained success, ISO 9004:2018, International
Organization for Standardization (ISO), (115), (https://www.iso.org/standard/70397.html )
Framework: Output
Rating Scale:  NOT IMPLEMENTED (NI): There are no KPIs for national lot release activities.

 ONGOING IMPLEMENTATION (OI): The NRA has recently drafted KPIs for national lot release activities but they have not yet
been reported.

 PARTIALLY IMPLEMENTED (PI): The NRA has developed KPIs for national lot release activities and has been applying them for
less than two year or they have not covered all critical steps.

Page 33 of 33

 IMPLEMENTED (I): The NRA has established and implemented KPIs for national lot release activities. The indicators are
reviewed regularly, and appropriate actions are taken, and decisions made.
Limitations and remarks:  When they refer to outcomes, indicators may be ambiguous and difficult to interpret, as outcomes are the result of many
factors that are difficult to disentangle. When they refer to processes, indicators are often too specific, as they may focus on a
particular intervention or condition or they may quickly become outdated as business models develop.
 Different methodologies are used to measure the NRAs performance on national lot release activities. In this case, the
assessor should verify that adequate supporting documents are available. The assessor should consider that developed
performance indicators should be Specific, Measurable, Achievable, Realistic, and Time‐ bound (i.e., “SMART”).
NRAs).

09 GBT LR Ver VI Ver 1nov2018 Final Adjusted

Uploaded by

Copyright:

Available Formats

09 GBT LR Ver VI Ver 1nov2018 Final Adjusted

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

09 GBT LR Ver VI Ver 1nov2018 Final Adjusted

Uploaded by

Copyright:

Available Formats

WHO GLOBAL BENCHMARKING TOOL (GBT) FOR EVALUATION

OF NATIONAL REGULATORY SYSTEM OF MEDICAL PRODUCTS

NRA LOT RELEASE (LR): INDICATORS AND FACT SHEETS

You might also like