“How often should we pentest?” is one of the most common questions in security, and one of the least satisfying to answer. Once a year? Every release? After every big change?
Matthew Moog sent us exactly this question for our Scale To Zero episode with Raj Handa, Senior Cybersecurity Engineer in offensive security at Workday, where he leads the pentest program. Matthew asked how often you should pentest, and how to decide the mix between internal teams, external vendors, and continuous automated testing.
Raj’s answer reframed the question.
The Trap: Pentesting as a Calendar Event
“When you operate an offensive security program at enterprise scale, the biggest trap is treating penetration testing or red teaming as a single calendar event.”
It isn’t a checkbox you tick once a year, he said. “It’s actually an offensive portfolio. So you have to match the right adversarial tool to the right risk.” Get the match wrong and the cost is real:
“You’ll end up wasting a lot of dollars having elite hackers run your glorified vulnerability scan.”
Raj decides the mix using three distinct layers: velocity, context, and an unbiased perspective.
Layer 1: Continuous Automation (Velocity)
The first layer runs 24/7. Raj called it “your baseline hygiene.” It includes SAST scanning, external attack surface monitoring, and continuous CI/CD pipeline checks.
This layer answers the “how often” question for known, repeatable issues: always. It is also what frees your humans to do work only humans can do. If you are building out this layer, our post on shift-left security in practice covers where those checks sit in the pipeline.
Layer 2: The Internal Team (Context)
The second layer is your internal offensive security team, working continuously and leaning on threat modeling. Its schedule is not the calendar.
“Our internal engineers don’t really operate on an arbitrary schedule. They actually operate on architectural risk.”
When a product team redesigns a core tenant isolation boundary or rolls out a new AI agent, the internal red team is embedded right there.
“The superpower, I feel, of an internal team is the deep business context.”
Raj pointed out that an outside consultant can spend two weeks just learning how internal microservices talk to each other. The internal team already knows. They know the architectural shortcuts taken years ago, the business logic, and the data flows. That is how they break the features that are unique to your platform.
So for this layer, the trigger is change. A meaningful architectural change is the moment to test.
Layer 3: External Vendors (Unbiased Perspective)
The third layer is third-party penetration testing on a more structured, periodic cadence. At Workday, Raj said, external partners are rotated on a scheduled basis, “typically biannually or annually, depending on what you are testing,” and/or as a requirement before any major flagship release.
He brings in vendors for two reasons:
-
Regulatory. Every enterprise has compliance frameworks, and a third-party report serves as the control artifact.
-
Blind spots. External testers find what the internal team missed, and bring niche tradecraft. His examples: adversarial machine learning, model inversion, and bespoke hypervisor breakout techniques.
Why Rotate Testers?
I asked whether rotating vendors was a regulatory requirement. Raj said not necessarily. Some financial audits require rotating audit firms, but from a security standpoint the point is fresh eyes. That can even mean the same firm rotating its engineers on your project.
If the same person retests your application, they bring the same skill set as last time. A new tester brings a different way of looking at things.
“It’s an art at the end of the day. The more eyes you can get on your attack surface, the more blind spots you’ll be able to discover.”
How the Layers Feed Each Other
The point of a portfolio is that each layer makes the others better. As Raj put it, “each layer basically feeds into each other,” and “it’s not one solution fits all, but a mix and combination of all these various approaches.”
In a separate answer, he described what that looks like over time. In year one, findings might be basic configuration issues or missing headers. By year two or three, your automated pipeline should be catching that noise, and human testers should be finding business logic bypasses, complex tenant boundary hops, or tool-use escalation in AI agents. If they are not, your testing needs a second look. We cover that metric in Never Pay for the Same Bug Twice.
So, How Often?
Putting Raj’s answer into one line each:
| Layer | What it covers | When it runs |
|---|---|---|
| Continuous automation | Baseline hygiene: SAST, external attack surface, CI/CD checks | Always on |
| Internal team | Business logic, architecture, platform-specific features | On architectural change, such as a new tenant boundary or AI agent |
| External vendors | Compliance artifact, blind spots, niche tradecraft | Scheduled (biannually or annually) and before major releases |
Pentest Portfolio Checklist
-
Baseline automation (SAST, attack surface monitoring, pipeline checks) runs continuously.
-
Internal testers are pulled in by architectural change, not a calendar.
-
New AI agents and tenant isolation changes trigger internal testing.
-
External tests are scheduled and also required before flagship releases.
-
Testers rotate, across firms or within one firm.
-
Each year’s findings are compared with last year’s: are basic issues moving to automation?
-
Each external finding is checked against what the internal team and automation should have caught (my addition).
Listen to the Full Episode
Thanks again to Matthew Moog for the question. Raj also covers extension confusion in AI editors, bug bounty in the age of AI-generated reports, and onboarding acquisitions safely. The extension confusion part has its own write-up: Extension Confusion: The IDE Supply Chain Risk Hiding in Your AI Editor.
-
Watch on YouTube: Pentesting, Bug Bounties & The IDE Supply Chain Blind Spot ft. Raj Handa
-
Listen on Apple Podcasts or Spotify
-
Join the conversation on LinkedIn
Scale To Zero is the security podcast we run at Cloudanix. If you have a question you want a guest to answer, send it our way. Matthew’s turned into this whole post.
People Also Read
- Extension Confusion: The IDE Supply Chain Risk Hiding in Your AI Editor
- AI Slop in Bug Bounty: How to Keep Your Program From Becoming a Spam Folder
- Never Pay for the Same Bug Twice: Vulnerability Prioritization Beyond CVSS
- Bug Bounty and Pentesting Through M&A: How to Bring an Acquisition Into Scope
- Mastering Cloud Penetration Testing