A Comprehensive

Guide to Building
a Pentest Program
By Ray Espinoza Head of Security at Cobalt.io
Contents
Defining a Pentest …………..…………..…………..…………..…………..…………..…………..………..…………………….. 3

What is a Pentest Program? …………..…………..…………..…………..…………..…………..…………..………….……… 5

Pentest Program Life Cycle …………..…………..…………..…………..…………..…………..…………..……………… 6

Why Use a Programmatic Approach? …………..…………..…………..………….….……..………………………..……… 7

The Problem with One-Off Pentests …………..…………..…………..…………..…………..…………..……….…..… 8

How to Build a Pentest Program …………..…………..…………..…………..…………..……………………………………. 9

Black Box vs Grey Box Pentesting …………..…………..…………..…………..………………………………………..… 12

Who To Involve in Your Pentest Program …………..…………..…………..…………..…………..…………………… 14

Case Study: Building a Pentest Program at Cobalt …………..…………..…………..…………..…………..………….. 16

Cobalt’s Pentest Program Objective …………..…………..…………..…………..………………………………….…… 16

Pentest #1: Q3 2019 …………..…………..…………..…………..…………..…………………………..………………….. 17

Pentest #2: Q4 2019 …………..…………..…………..…………..……………………………………….………………….. 17

Lessons Learned …………..…………..…………..…………..…………..……………………………………………..……… 18

Feedback From Our Pentesters …………..…………..…………..…………..…………..………………………………… 19

Incrementally Improving Security Outcomes …………..…………..…………..…………..…………..……………. 20

Key Takeaways …………..…………..…………..…………..………………………………………..………………………………… 21

About the Author …………..…………..…………..…………..…………..…………………………………….……………………. 22

Defining a Pentest
Right now, there are three primary manual security testing options available to organizations:

Traditional penetration testing. Organizations pay a security service provider to test a

specific asset or set of assets using a clearly defined methodology. Testing is completed by
on-staff security consultants. The customer receives a report of the findings, identifies
genuine issues and opens each one as a ticket for engineering.

Bug bounty. These are open-ended programs where any security professional or hacker can
search for vulnerabilities in an application or asset. The customer pays testers based on the
perceived ‘quality’ of each finding, i.e., the level of risk it poses.

Pentest as a Service (PtaaS). PtaaS is an emerging delivery model that uses a platform-driven
solution and a freelance talent pool to manually search for vulnerabilities. Testers are
background checked and skills assessed before joining the talent pool, and then selected for
each project based on their specific skills and experience, making PtaaS a highly-focused
testing option. The addition of SaaS platform technology to traditional pentest consulting
models drives workflow efficiencies by connecting pentesters directly with engineering teams
to address identified vulnerabilities.

This guide focuses specifically on Pentest as a Service (PtaaS). When we use the term pentest,
it’s in the context of a PtaaS program. A formal testing program is a great way to ensure
continuous security coverage, and PtaaS is the approach we advocate to make security testing
effective, easy, and seamless.

A Comprehensive Guide to Building a Pentest Program 3

 Overview of Security Testing Options

Testing Method Description Deliverables Used for

Traditional Penetration Vendor-provided PDF report Compliance

Testing
One-off Point-in-time security
Checklist-driven assessments

Bug Bounty Crowd-powered Individual vulnerability Ongoing security

reports assessment
Continuous program
Vulnerability disclosure ‘Pay for results’ model Usually combined with
other testing models

Pentest as a Service Platform-driven Individual severity-rated Ongoing security

(PtaaS) vulnerability reports assesment
Regular testing
Selectively-sourced Reporting for compliance Compliance
freelance testers and high level analysis
Insight across tests
Integration with issue
tracking solutions

Security Testing Benefits and Challenges

Testing Method Benefits Challenges

Traditional Penetration Provides a point-in-time assessment Varied tester availability, quality

Testing & expertise
Highlights issues related to common
standards, e.g., OWASP Top 10 Scheduling often a challenge
Slow to provide results

Bug Bounty Provides an ongoing security Customer must triage submissions

assessment
Submissions are often duplicated or
Offers a broad range of tester expertise invalid

Only pay for valid findings Hard to ensure full testing coverage

Pentest as a Service (PtaaS) Testers are chosen for their domain- For the best results, it requires ongoing
specific expertise input from the customer

Runs on a fixed cadence, with on-

demand testing available at a 24 hour
lead time

Ensured coverage via structured

approach and real-time collaboration

Platform connects testers directly to

engineering teams for remediation

A Comprehensive Guide to Building a Pentest Program 4

What is a Pentest Program?
A pentest program is a clearly defined series of pentests designed to systematically identify and
remediate vulnerabilities in one or more assets or asset groups. A program typically follows an
annual, renewable cycle, with testing completed periodically throughout the duration — for
example, on a weekly, monthly, quarterly, bi-annual, or annual basis. By planning pentest
programs annually, security leaders can ensure full coverage of assets and identify the depth of
coverage needed for each asset.

These programs should be structured to ensure the most critical assets — or those that are most
frequently deployed — are tested on a regular basis. This ensures:

1. Continual testing coverage for critical and frequently updated assets; and,

2. A much higher probability that high-risk vulnerabilities will be quickly identified and
remediated.

NOTE
It’s important to realize pentest programs DON’T fulfill the same function as DAST/SAST
solutions OR bug bounty programs. Consistent pentesting of assets by experts with
domain-specific skills is necessary to identify vulnerabilities that require business context
not available from bug bounty programs or automated scanners.

A Comprehensive Guide to Building a Pentest Program 5

 Pentest Program Life Cycle

Plan
Identify the asset(s) that need
to be tested, the cadence of
testing, and time frames.

Repeat Scope
Pentest programs use Determine testing depth
small, frequent tests to and resources.
ensure continual
coverage.
Anatomy of
a Pentest
Program

Retest Test
Testers check to ensure Domain-specific experts
fixes/mitigations have conduct thorough
addressed each automated and manual
vulnerability. testing.

Remediation
Security partners with
engineering to resolve or mitigate
identified vulnerabilities.

A Comprehensive Guide to Building a Pentest Program 6

Why Use a Programmatic Approach?
Pentests help organizations find vulnerabilities in their digital assets. Once found, vulnerabilities
can be patched or otherwise resolved before a threat actor can exploit them.

However, the demands placed on modern organizations ensure that IT infrastructure is never
static. New assets, patches, and lines of code are added all the time, and new vulnerabilities are
inevitably introduced over time. As a result, if the most recent pentest of an asset was six months
ago, there’s a very high chance it has new vulnerabilities that haven’t been identified.

Using a programmatic approach to pentests allows organizations to continuously evaluate the

security of digital assets. This ensures that newly introduced vulnerabilities are consistently
identified, which in turn leads to a reduction in cyber risk and higher overall level of security.

However, the benefits of a pentest program don’t stop there. Maintaining a pentest program also
gives organizations the opportunity to build an ongoing relationship with their testers. Over time,
testers gain a much deeper understanding of the assets they are working with, helping them to
identify deeper issues and vulnerabilities that would otherwise be missed.

Finally, a pentest program can also help organizations identify systemic issues that arise across
multiple assets. These issues often indicate the need for targeted developer training or
implementation of new technologies. Through this insight, pentest programs help organizations
improve their overall security posture by addressing problems at their root cause.

In the Case Study section of this guide, we’ll explore the benefits of a pentest program in more
detail. By analyzing lessons learned from an initial engagement, the Cobalt Core pentest team was
able to get 5X the value during a second pentest.

A Comprehensive Guide to Building a Pentest Program 7

The Problem with One-Off Pentests
There’s nothing inherently wrong with one-off pentests. They can provide a valuable point-in-time
assessment that can be used to find and fix vulnerabilities in an asset. There are many
circumstances when a one-off test makes a lot of sense. For instance:

Ahead of a compliance assessment; or,

During the due diligence phase of a merger

or acquisition.

Even when a pentest program is in place, ad-hoc

testing has value. New features may need to be
rushed into production, or sudden customer requests
accommodated, and in these cases one-off pentests
are appropriate.

However, in most cases, having a structured cadence for coverage is preferable to a point-in-time
assessment.

A pentest program provides added value by

testing an asset continuously at a regular cadence
— e.g., monthly or quarterly — and tracking its
security health over time.

A Comprehensive Guide to Building a Pentest Program 8

How to Build a Pentest Program
Building your first pentest program can be daunting. However, with careful planning, any
organization can get a pentest program up and running within a short timeframe by following five
simple steps:

Step 1: Setting program objective(s)

Step 2: Identify which assets are most critical

and/or at the highest risk of cyberattack.

Step 3: Determine how often assets are

changed or updated.

Step 4: Check whether any major infrastructure

changes are planned for the program period.

Step 5: Plan the logistics of your pentest

program.

Step 6: Define the scope and depth of

your program.

A Comprehensive Guide to Building a Pentest Program 9

STEP 1: SETTING PROGRAM OBJECTIVE(S)

Before you enter into a pentest program, it’s important to determine what it needs to achieve. For
instance, do you need to achieve compliance with one or more frameworks? Do you have specific
assets that need to be tested and hardened? Your objectives will inform the structure of your
pentest program, so be sure to involve all stakeholders in this step.

Common objectives include:

Ensure testing of all assets required for compliance (e.g. PCI-DSS)

Ensure all assets with sensitive data are tested regularly

Ensure business critical assets are tested regularly

Ensure all major critical releases are tested

STEP 2: IDENTIFY WHICH ASSETS ARE MOST CRITICAL AND/OR AT THE HIGHEST RISK
OF CYBERATTACK.

It may be that not all assets are equally important to your organization. Your pentest program
should aim to provide maximum coverage for your most critical assets while testing less critical
assets more infrequently.

STEP 3: DETERMINE HOW OFTEN ASSETS ARE CHANGED OR UPDATED.

A pentest program should provide continuous security coverage for in-scope assets. For that to be
possible, pentests should be scheduled each time a significant codebase change is rolled out.

A Comprehensive Guide to Building a Pentest Program 10

STEP 4: CHECK WHETHER ANY MAJOR INFRASTRUCTURE CHANGES ARE PLANNED FOR THE
PROGRAM PERIOD.

Significant changes to infrastructure — such as a change in cloud providers or a move from on-
premise to the cloud — warrant a higher-than-average frequency and depth of pentesting. When
scheduling pentests, make sure you have a full picture of the organization’s intended digital
strategy during the program period. Equally, plan for some level of ad-hoc testing to account for
any surprises along the way.

STEP 5: PLAN THE LOGISTICS OF YOUR PENTEST PROGRAM.

The precise content of a pentest can vary significantly depending on the asset being tested and
your priorities. For instance, if you want to hunt down business logic flaws or privilege escalation
vulnerabilities, it makes sense to provide testers with login credentials for your asset. If you want
to know what an external threat actor could do without credentials, an uncredentialed test may be
appropriate. Equally, you may decide to include both credentialed and uncredentialed approaches
in a single pentest.

STEP 6: DEFINE THE SCOPE AND DEPTH OF YOUR PROGRAM.

Once planning is complete, it’s time to decide which assets will be included in your program.

A word of caution: a common mistake that security buyers make is including too many assets in a
pentest and not allowing pentesters enough time to thoroughly test for vulnerabilities. If you aren’t
sure, ask your pentest provider for guidance to ensure your assets can be fully tested in the time
allocated.

A Comprehensive Guide to Building a Pentest Program 11

Black Box vs Grey Box Pentesting
One of the most common pentesting conundrums is whether to use a ‘black box’ or ‘grey box’
approach.

Black box pentesting is where testers have no prior knowledge of the assets
being tested and aren’t provided with login credentials. It requires pentesters to
find flaws and vulnerabilities without any knowledge of data flow, design,
architecture, or roles/permissions.

Grey box pentesting equips testers with some level of information

about an asset’s function, business logic, and permissions, and may
also provide them with login credentials. This is intended to make the
pentesting process more efficient, faster, and more productive.

Note that grey box pentesting is not the same as white box
pentesting, where testers usually have access to source code and
conduct targeted testing based on what is found in the code.

So, which is better?

Some argue that black box pentests are more realistic, as real-
world threat actors won’t have access to credentials or guidance. At
Cobalt, we think about it like this.

The time allocated to pentesting is always limited. If testers are forced to spend some of the time
learning about the asset, that diminishes the time they can spend searching for vulnerabilities.
Threat actors, meanwhile, have as much time as they need to find weaknesses in an asset; having
to spend time studying an asset is not a hurdle to them in the same way it is for pentesters.

With a grey box pentest, testers have more time to search for vulnerabilities in an asset, because
they don’t have to waste time trying to reverse engineer an asset’s functionality. As a result, the
chance that testers will find high severity vulnerabilities such as business logic flaws are much
higher.

A Comprehensive Guide to Building a Pentest Program 12

To illustrate this, here’s a simple example. A B2B asset is designed to provide a business service.
There may be support documentation, but it’s unlikely to be publicly available. B2B assets typically
have complex workflows and permissions, which may be unclear to a first-time user. If pentesters
are forced to develop their own understanding of how the asset works, that will significantly
reduce the time they can spend searching for vulnerabilities — and there is a chance they will
overlook something and miss a serious vulnerability.

Once again, if real threat actors were to target the same asset, they would not suffer the same
disadvantages because they have the freedom to spend as long as needed to find a weakness. As a
result, while there are benefits to black box testing, grey box pentesting is a better option for most
organizations.

NOTE
The ‘black box’ vs ‘grey box’ debate is NOT only about credentials. In many cases, the most
important advantage a pentester can gain from grey box testing is a clearer understanding
of an asset’s business logic, permissions, and intended function.

A Comprehensive Guide to Building a Pentest Program 13

Who To Involve in Your Pentest Program
Just like any aspect of security, no pentest program can succeed without support from other areas
of the business. Two stakeholders in particular need to be closely engaged with:

Engineering/Product

Finding vulnerabilities is half the battle. The other half is closing them promptly and successfully.

For this to happen, vulnerability results from your pentest program need to be routed directly into
standard development systems and workflows. In other words, your pentest program should
support the secure development lifecycle (SDLC).

Engage with engineering and/or product leads as early as possible, and seek their input
throughout the planning stage of your pentest program. Developer time will be needed to resolve
identified vulnerabilities, and it’s best if this can be scheduled in advance to ensure sufficient
capacity.

For more on the importance of building relationships between security and engineering, check out
this blog post.

A Comprehensive Guide to Building a Pentest Program 14

 Leadership

Leadership buy-in is critical for any security initiative. Pentest programs work best when they are
run consistently over time, so you’ll want to be sure that you have budgetary support beyond the
current year.

Fortunately, in recent years security has inched closer to the boardroom. In 2020, security is firmly
established as a critical business function, and most boards demand oversight in some capacity.
As a result, gaining approval for security initiatives that have genuine business value is much easier
than it once was.

Providing regular reporting to leadership executives is an excellent way to evidence the ROI of your
pentest program. By highlighting the number and quality of vulnerabilities identified, as well as
their potential impact on operations, you’ll be demonstrating the business value of your program.

However, when seeking buy in, it’s critical to

communicate in a way that executives
understand and are interested in. To achieve
this, you must have an understanding of
what the organization as a whole is trying to
achieve, and how your pentest program can
support those objectives. If you can produce
metrics that clearly tie your program to the
overall success of the organization, gaining
leadership buy-in will be much more
achievable.

For more on creating security metrics for a

board-level audience, read this blog post.

A Comprehensive Guide to Building a Pentest Program 15

Case Study: Building a Pentest Program
at Cobalt
At Cobalt, we feel strongly about the importance of a pentest program. So much so that we have a
pentest program of our own to identify and resolve vulnerabilities and bugs in our platform,
domains, and APIs.

Naturally, our program aims to be the gold standard of pentesting. After all, this is what we do.

COBALT’S PENTEST PROGRAM OBJECTIVE

As mentioned earlier, the first step in developing a pentest program is to set objectives. Cobalt’s
pentest program has the following objectives:

1. Ensure our critical assets stay secure throughout frequent releases.

2. Ensure customers can trust that our platform is secure and regularly tested.

3. Maintain SOC2 and ISO27001 compliance.

The program currently follows a quarterly testing cadence for key assets, which include the Cobalt
web application, API, network, and cloud configuration. In the event of larger releases, we also
perform ad hoc pentests.

Through our program, we’ve learned a lot about how to plan and structure pentests to support
business and security objectives. To highlight some of these learning points, we’ll examine two
recent rounds of testing: Q3 2019 and Q4 2019.

A Comprehensive Guide to Building a Pentest Program 16

PENTEST #1: Q3 2019

Our Q3 pentest targeted several Cobalt domains and subdomains, as well as all of our external IPs.
Our methodology was simple:

1. For web assets, we wanted to cover the OWASP Top 10, ASVS, and test application logic.

2. For our network assets, we wanted to cover OSSTMM and the CIS Security Controls.

3. For our APIs, we wanted to cover the OWASP 2017 Top 10.

The aim of our Q3 pentest was to find out what real-world threat actors might be able to do to our
assets. As a result, we decided to conduct a black box pentest

The results: Our pentesters discovered three total vulnerabilities, one of which was high-risk and
fixed immediately.

What did we learn? Overall the pentest engagement was a success, because it helped us identify
important vulnerabilities in our key assets. However, it didn’t produce as many findings as we had
hoped. We had a hunch that a grey box test would be more effective.

PENTEST #2: Q4 2019

Since we had recently switched cloud providers, in Q4 we wanted to conduct a network pentest.
Our testing scope was similar to that of our Q3 pentest, however, we made several structural
changes based on our lessons learned from Q3:

1. We created test accounts for all of our pentesters.

2. We provided our pentesters with context about the assets, including documentation, use
cases, data flow/network diagrams, information on roles and permissions, etc. Among other
things, we recorded a video that walked testers through critical work flows and the differences
between roles.

3. We had an experienced Cobalt Core member lead the pentest as well as personally test a
critical section of our platform.

A Comprehensive Guide to Building a Pentest Program 17

In simple terms, our Q4 pentest was much more ‘grey box’ than our Q3 pentest had been. When
the results started to come in, the difference was clear.

The results: Our pentesters uncovered a higher number of vulnerabilities during the second
pentest, two of which were high-risk and fixed immediately. On top of this, several of the
vulnerabilities were more complex in nature than those found during the first test — the type of
vulnerabilities that might be missed if testers aren’t provided with thorough context into the assets
they are testing. To illustrate this, many of the findings from the second test were related to roles
and permissions, which can only be thoroughly assessed by a tester who understands the business
logic of each asset.

In addition to the additional findings, comparing the experiences of our Q3 and Q4 pentests
yielded plenty of important lessons.

Lessons Learned

Communication is critical. Communicating early and often with stakeholders — particularly

those you’ll need support from during the test or to drive remediation efforts — is essential
to the success of a pentest program. It’s also important to update all stakeholders on the
results of each pentest.

Pentest programs are great for incremental security improvement. Each round of pentests
within a program provides learning opportunities that can be implemented ahead of the
next test.

Challenge assumptions around your existing pentesting methodology. Following our

lessons learned during Q3, we made significant changes to the structure of our pentests in
Q4. And after discussing the changes with our testers, we found that the new methodology
was far superior.

Grey box testing is the way to go for B2B companies. This may not be true for every
organization, but for our B2B assets, grey box pentesting proved a much more effective tool
for identifying high-risk vulnerabilities.

A Comprehensive Guide to Building a Pentest Program 18

 Pentester engagement is everything. By actively engaging with our testers throughout each
pentest, we learned much more than if we’d sat back and waited for vulnerability reports.
We made ourselves available to answer their questions, and in turn, they came to us with
their concerns or suggestions.

And we weren’t the only ones to learn from our pentest. Our testers also had a few things to say
about the engagement.

Feedback From Our Pentesters

Authenticated testing gives more and better results. This is along the same lines as the ‘grey
box vs. black box’ debate. It was helpful for us to hear that our pentesters also believe the
approach we took in Q4 was superior.

A more engaged customer means more engaged testers which leads to a better pentest.
From our testers’ perspectives, having a direct line of communication via Slack was hugely
valuable. They were able to get answers to questions quickly, and easily discuss their activities
to ensure full testing coverage.

Product walkthroughs and guidance make a huge difference. Again, grey box testing isn’t
just about credentials. It’s about testers understanding exactly how a system is supposed to
work. The better the guidance they receive, the higher quality findings they are able to
produce.

A Comprehensive Guide to Building a Pentest Program 19

 “Having a real time point of contact gave us
a deeper understanding of the environment
while we were testing. This effectively helps
us find more impactful findings.”

Joan Bono, Cobalt Core Pentester

“When working on a pentest time is of the essence.

Offering sufficient product guidance around what
you, as a customer, care about ensures that you are
getting the most out of your test.”

Vinod Tiwari, Cobalt Core Pentester

Incrementally Improving Security Outcomes

In addition to the learning points already mentioned, our experience highlights one of the most
important aspects of a pentest program: Constant incremental improvement.

Perhaps the single greatest benefit of a pentest program over ad-hoc testing is the ability to learn
from each pentest and improve. Each time you conduct a pentest, you’ll have an opportunity to
learn more about how to run an ideal pentest program for your organization.

If you take the opportunity to learn from each engagement and implement changes, the quality of
your findings will improve exponentially over time.

A Comprehensive Guide to Building a Pentest Program 20

Key Takeaways

Pentesting is distinct from both legacy penetration testing and bug

bounties. It provides the benefits of both approaches while compensating for
their deficiencies.

Pentests are smaller, more frequent engagements than legacy

penetration tests. And they are conducted by highly experienced security
professionals with domain-specific expertise.

A pentest program is a series of pentests designed to systematically

identify and remediate vulnerabilities in one or more assets or asset groups.

In general, ‘grey box’ pentesting produces more and higher-quality

findings than ‘black box’. This is especially true for complex B2B assets
where business logic and permissions aren’t obvious, and/or when the time allocated is
not sufficient for testers to conduct a full reconnaissance exercise.

A Comprehensive Guide to Building a Pentest Program 21

 Engaging with key stakeholders is critical to the success of a pentest
program. In particular, engineering leads should be actively involved, while
executives should be given regular updates.

Engagement and communication with testers drastically improves

program results. Quite simply, a more engaged customer = more engaged
testers = more and better findings.

The #1 benefit of a pentest program over ad-hoc is the ability to

constantly improve. Asking for feedback from your pentesters and
identifying your own internal learning points will help you drastically improve the
quantity and quality of findings over time.

To find out more about how a pentest program could benefit your organization, visit our website.

About the Author

Ray Espinoza
Head of Security at Cobalt.io

Ray has over 20 years of technology experience and 12+ years in information security. Prior to
Cobalt.io, Ray drove 3rd party cloud security across Amazon’s retail business. Additionally, he held
VP and CISO roles with Atmosera and Proofpoint as well as various security leadership positions at
Workday, Cisco Systems, and eBay.

A Comprehensive Guide to Building a Pentest Program 22