29/5/23, 15:56 IS-TST-1101 Performance Testing

Copyright © Guidewire Software, Inc., confidential and proprietary. Latest documentation at

docs.guidewire.com.

IS-TST-1101 Performance Testing

Describes the requirements and necessary metrics when executing a performance
test.

Standard ID
IS-TST-1101
Global or Regional
Global

Standard Category
Testing
Upgrades to Guidewire Cloud Adoption Required
Yes
Usable on Self-Managed Projects
Yes
Assessment Type
Manual
Effective
01-JUL-2020
Expiration
N/A
Products
Guidewire InsuranceSuite

Video

https://docs.guidewire.com/cloud/standards/latest/Testing/Standard-IS-TST-1101-PerformanceTesting.html 1/12
29/5/23, 15:56 IS-TST-1101 Performance Testing

Introduction
The purpose of a performance test is to ensure the system can support the expected
loads while maintaining adequate application response times. Running a successful
performance test requires planning and logistical coordination of tools and personnel.
Environments and monitoring must be set up and scheduled, data prepared, and the
environmental monitoring tools configured and ready. This standard explains
Guidewire's process for executing a successful performance test, as well as the
metrics and tooling involved. It covers:

Types of requirements

Data needs
Scripting needs

Workload model

This standard is intended for project managers, program managers, performance

testers, interested project team members, project sponsors, and anyone who has
responsibility or accountability for performance in the Guidewire project.

Performance Testing
Performance tests must be run before every new implementation release, cloud
update, and major functionality update. The customer is responsible for executing the
performance tests and providing Guidewire with the results. This enables us to
validate the required application response times before deploying changes to
production. Performance tests must be repeatable, measurable, and consistent;
therefore, an automated tool must be used to generate the load. Manual performance
testing, in which a group of users performs synchronized actions, is not acceptable.
https://docs.guidewire.com/cloud/standards/latest/Testing/Standard-IS-TST-1101-PerformanceTesting.html 2/12
29/5/23, 15:56 IS-TST-1101 Performance Testing

Planning
Successful performance testing requires comprehensive test planning, starting with
documenting the requirements in a performance test plan.

Non-Functional Requirements

Requirements for performance testing must contain the following criteria for the
production system:

The number of concurrent users

The data volumes and distribution, and their expected growth

The expected number of transactions performed within a defined time frame

For designated functions, the additional information below is required for each
function:
Estimated peak hourly volume

Expected response time

Breakdown of systems, calls, and integrations involved in the function and their
corresponding service level agreements (SLAs)

Getting the correct information is important. If the number is underestimated, the

performance test will not assure that the production volumes can be met. If the
number is overestimated, the performance test may demand unnecessary changes or
hardware to support production volumes.

The following is an example of the requirements.

Designated Function Estimated Peak Hourly Volume Business Function Component Breakdown

Response Time for 95%

Policy Search 5,840 1.​75 – 3.​25 seconds DMZ (brokers only) 0.​25s

PC (2 sec)

Policy Quote 510 2 – 3.​5 seconds DMZ (brokers only) 0.​25s

PC (1 sec)

ESB (0.​25 sec)

Rating Engine (1 sec)

Form Generation 2,920 2 – 3.​5 seconds DMZ (brokers only) 0.​25s

https://docs.guidewire.com/cloud/standards/latest/Testing/Standard-IS-TST-1101-PerformanceTesting.html 3/12
29/5/23, 15:56 IS-TST-1101 Performance Testing

Designated Function Estimated Peak Hourly Volume Business Function Component Breakdown

Response Time for 95%

PC (1 sec)

ESB 0.​25s

EngageOne (1 sec)

Environment

The test environment should be identical to the production environment. Where this is
not achievable, the impact of the differences, especially system architecture
differences, must be carefully assessed for the test validity. While it is possible to
perform tests on a scaled-down number of servers by adjusting the workload model,
this introduces elements of risk and should be avoided.

Other environmental differences are more difficult to prevent. For example, an

integration to a third party may not provide a full-scale test system. In this case, the
integration may be stubbed and the required response time for that integration can be
assumed to be met.

Information required for environmental decisions needs to be captured. This includes:

Any deviation from the production environment regarding the number and size of
servers

Any stubbing of third-party systems

Any differences in data size, consistency, and distribution

Database

It is important to set up the database with appropriate volume and distribution. IS-
TSD-1181 Test Data Strategy describes what is needed to correctly load a
performance testing database.

The use of production data to populate the performance test database requires
security standards be applied for personally identifiable information stored in non-
production environments. This is outlined in IS-SEC-1008 Personally Identifiable
Information (PII).

Performance Test Scripts

https://docs.guidewire.com/cloud/standards/latest/Testing/Standard-IS-TST-1101-PerformanceTesting.html 4/12
29/5/23, 15:56 IS-TST-1101 Performance Testing

The scripts must simulate processes that users follow when using the system. In
addition to exercising the designated functions, scripts should cover the full extent of
the system being tested, including:

Characteristics to How to Determine if a Flow Has This Action

Consider Characteristic

Critical business These are key business processes that support the Create scripts that cover these
processes business, including those that may occur less often business processes.
but are crucial to the business.

Resource-intensive Create performance models for the key flows using Create scripts to exercise the highly
flows performance modeling tests. utilized resources and expose their
impact on other activities in the
system.

Flows with key Look for flows that exercise one or more synchronous While interfaces are often exercised
synchronous interfaces, as identified by developers when profiling by critical business processes,
interfaces or modeling the flow. Flows that use interfaces may ensure that any interfaces are
be constrained by the limited capacity of the interface. covered in the test scripts.

Areas of major Where significant customizations have been made, Create scripts to include these
customization the potential for experiencing performance issues will customizations.
be higher.

Workload Model

Once the scripts are selected, a workload model is created that describes how many
virtual users will perform each function. This workload model is based on the
expected user volumes and number of transactions. The model must be realistic. If
you assign too many users or too many transactions, you will unnecessarily overload
the system and your results will be incorrect. If you assign too few users or
transactions, you won’t stress the system enough to validate that it can handle the
load.

The workload model tabulates the script to be executed, the associated user type,
and the maximum number of concurrent users. In the following example:
Script Name is the performance test script that will be run.

Concurrent Users reflects the number of virtual users that will execute those
scripts.

Transactions/Hour is the frequency at which the script will be run and repeated,
as well as wait times for each script.

Script Name Concurrent Users Transactions/Hour

https://docs.guidewire.com/cloud/standards/latest/Testing/Standard-IS-TST-1101-PerformanceTesting.html 5/12
29/5/23, 15:56 IS-TST-1101 Performance Testing

Script Name Concurrent Users Transactions/Hour

PC_CancellationAuto 10 50

PC_ChangePolicyAutoPA 80 1300

PC_ChangePolicyAutoTX 80 1300

PC_ChangePolicyAutoWV 80 1300

PC_CreateActivity 34 120

PC_CreateResolveUWIssue 25 7

PC_Inquiry 16 65

PC_QuoteNBindAutoPA1Car 66 1300

PC_QuoteNBindAutoTX1Car 66 1300

PC_QuoteNBindAutoWV1Car 66 1300

PC_QuoteNBindAutoPA5Car 6 130

PC_QuoteNBindAutoTX5Car 6 130

PC_QuoteNBindAutoWV5Car 6 130

PC_RenewPolicyAuto 66 1400

PC_ReviewPolicyChange 25 200

PC_ReviewSubmission 4 40

PC_Search 21 60

PC_TeamScreenTab 1 7

PC_ViewAccount 4 10

PC_ViewPolicy 4 7

Total 600 10156

Writing Scripts
Characteristics of well-written performance test scripts and testing practices include:

They are parameterized to enable different data combinations.

Script data values, such as names, addresses, and coverages must be unique to
each script to ensure the system does not access data cached by the application
or database.

https://docs.guidewire.com/cloud/standards/latest/Testing/Standard-IS-TST-1101-PerformanceTesting.html 6/12
29/5/23, 15:56 IS-TST-1101 Performance Testing

They use multiple user IDs to avoid virtual users from using the same ID.
They configure virtual users with the same roles used in production.

They don't use the superuser account, or any users with the superuser role.
Scripts are not be configured to execute as quickly as possible. Instead, they
incorporate time delays that emulate actual user behavior in production.
The test data is sufficiently varied to avoid two virtual users accessing the same
entity simultaneously.

The script sets transaction start and stop timers or flags to provide accurate
metrics.

Tests include realistic messaging. If the messages need to be mocked, include

realistic delays to simulate end-point processing.

Tools
Load testing requires the use of various tools. Guidewire does not specify which tool
to use, but focuses on the results obtained. The selected tool(s) must be able to
generate the load necessary to ensure consistent and repeatable results and
accurate reporting. Manual performance testing does not meet this standard and must
not be used for load testing

Load Generation

The tool must mimic requests and responses from the Guidewire application
under test.
The load generation tool must generate enough concurrent users to adequately
load the system.
Load generation should capture and report on client-to-server response times.

The tool must provide response times for single steps as well as process flows,
such as the response time for clicking the quote button and total time for the
submission process.
The load generation tool must be able to accept a wide range of data to ensure
multiple users, multiple data points, and multiple process flows can be executed.

Monitoring

The response times metric mentioned above should be the primary metric used to
determine system performance. The following additional metrics are available by
https://docs.guidewire.com/cloud/standards/latest/Testing/Standard-IS-TST-1101-PerformanceTesting.html 7/12
29/5/23, 15:56 IS-TST-1101 Performance Testing

request from Guidewire Cloud Operations:

CPU utilization
System memory for the application servers

Disk space for the load generation machines

Network data/performance

Execution
Pre-work

Before running any performance test, basic maintenance and setup activities must be
completed:
Run all code inspections provided by Guidewire
Execute the database consistency check (DBCC)

Remediate any issues identified in the above inspections

Running Tests

The first step in executing test scripts is to run the script to confirm it performs as
expected and can be run multiple times with a greater diversity of data. The next
execution depends on the test's target. Some tests target very specific transactions
and can be run separately to quickly test that area. Other tests are created to be run
as part of a group and run over a longer time.

It is important to test against the largest expected volume of claims, policies, etc., by
various dimensions, to ensure the application scales adequately. This should be done
in isolation, using the Guidewire Profiler. Determining if a slow load test is due to the
size of the claim/policy versus some other issue is expensive and disruptive.

Load testing involves exercising the test system using production-equivalent loads
while the test transactions are monitored. Use the following practices during load
testing:
Begin with a single test script users and gradually increased the number of users
until the peak load is reached.
Maintain the peak load for at least one hour to verify all elements of the system
can support the load.

https://docs.guidewire.com/cloud/standards/latest/Testing/Standard-IS-TST-1101-PerformanceTesting.html 8/12
29/5/23, 15:56 IS-TST-1101 Performance Testing

During the tail of the testing, gradually reduce the test user count to ensure
resources are being freed up as the load decreases.

Ensure the pace of the execution emulates actual user activity.

Any batch jobs scheduled to run during the testing should be initiated according
to the production schedule.

Batch Jobs

Run batch job tests to establish whether the system capacity supports the expected
volume of work within the batch time window. Batch job testing requires a sufficient
amount of data and should be performed using the following approach:
Run all batch jobs as scheduled, using a fully populated database.

Run batch jobs with the largest expected volume +20%.

Ensure all batch jobs in the schedule are complete before the next job starts.

Verify all batch jobs are completed within the allotted time frame.

Monitoring
Performance testing of the designated functions for the system under test must be
monitored by Guidewire’s Cloud Operations and scheduled ahead of time.

To test any additional functions, the test load generation tool must capture the
response times for metrics that may have been defined in the test scripts.

Reporting
Performance test reports confirm that the performance requirements, as defined in
the SLAs, have been met. For the designated functions, Guidewire’s Cloud
Operations team will provide this report. For other functions, reports should be
extracted from the testing tools and must contain:

The minimum, maximum, average, and 90% times for the transactions
Number of times the transaction passed

Number of times the transaction failed

Overall load on the system at its peak

A copy of the report must be provided to Guidewire Cloud Assurance for baselining
the systems and providing a reference for future assessments.
https://docs.guidewire.com/cloud/standards/latest/Testing/Standard-IS-TST-1101-PerformanceTesting.html 9/12
29/5/23, 15:56 IS-TST-1101 Performance Testing

Project Scenarios
The criteria to determine whether a Guidewire implementation
is required to adopt Guidewire Cloud Standards, and
which standards apply, are defined in Standard-GW-OPR-1235. The following sections
define the adoption, compliance criteria, and applicability of this specific standard in
each of the project scenarios covered in Standard GW-OPR-1235.

Scenario: No Prior Guidewire Implementation Exists

All aspects of this standard are required on a new Guidewire implementation. Quick
Start projects are exempted from needing to meet this standard.

Scenario: Follow-On Release to an Existing Guidewire Implementation

Deployed to Guidewire Cloud

All aspects of this standard are required for follow-on releases.

Scenario: Upgrade of Self-Managed Guidewire Implementation

All aspects of this standard are required for existing functionality being upgraded as
part of the technical upgrade to Guidewire Cloud, as well as for any new features or
functionality being added as part of the upgrade project.

Scenario: Self-Managed Implementations

It is highly recommended that all projects follow these standards but this standard is
not enforced for self-managed implementations.

Successful Standard Adoption Criteria

Through Cloud Assurance, Guidewire will identify any items that do not adhere to this
standard and validate that they are resolved before deployment.

Guidewire Cloud Standard Compliance Criteria

We will assess the following questions as part of Cloud Assurance:

Question ID Question Assessment Guidelines

PAM-200054 What are the non-functional Non-functional requirements must

requirements and how have they include information on data volume,
been documented? user load, and expected application
response times. See the Non-

https://docs.guidewire.com/cloud/standards/latest/Testing/Standard-IS-TST-1101-PerformanceTesting.html 10/12
29/5/23, 15:56 IS-TST-1101 Performance Testing

Question ID Question Assessment Guidelines

Functional Requirements section for

details.

PAM-200908 Does the database have adequate Run a database distribution report to
volume and distribution of data? confirm the expected tables are
populated.

PAM-200909 Do the scripts perform realistic See the Performance Test Scripts
business scenarios? section.

PAM-200913 Have all relevant stakeholders Test and monitor Cloud Operations
committed to their tasks and timings and any third-party system.
in the performance test plan?

PAM-200914 Do the reports provide adequate Reports should include transaction

detail to determine the test results? times, information on user load, and
pass/fail rates.

GW-358378 Does the load generation tool have The tool must be able to support the
all the capabilities necessary to run number of concurrent users, interact
the load test? with Guidewire, and track response
times.

GW-358379 Is the monitoring tool capable of The monitoring tool must be able to
capturing the required information? If monitor CPU, memory, disk space,
not, has Cloud Operations been disk I/O, and response times.
contacted to see if the tool is
monitoring appropriately?

GW-358380 Has all pre-work been completed? Ensure all inspections, DBCCs, and
remediations have been completed.

Impact of Non-Compliance

Issues identified during Cloud Assurance that do not adhere to this standard will be
validated to ensure that they are resolved before deployment.

For More Information

Please contact your Guidewire representative.

Additional Resources

Whitepaper_PerformanceTesting - KCS Article 000028667

Whitepaper_StabilizationTesting - KCS Article 000028679
Whitepaper_OperationalAcceptanceTestingforGuidewireCloud - KCS Article
000028683

https://docs.guidewire.com/cloud/standards/latest/Testing/Standard-IS-TST-1101-PerformanceTesting.html 11/12
29/5/23, 15:56 IS-TST-1101 Performance Testing

Standard_GW-TST-1087-Test Strategy

Standard_IS-SEC-1008-Personally Identifiable Information

Legal information at https://www.guidewire.com/legal-notices/

https://docs.guidewire.com/cloud/standards/latest/Testing/Standard-IS-TST-1101-PerformanceTesting.html 12/12