What Automated CTEM Tools Miss, and Why Human Attackers Still Win

Picture the report. A security team runs automated testing and validation tools against a checkout flow overnight, hundreds of attack scenarios, executed automatically. Morning comes and the dashboard is green. Every control held.

Three days later, a researcher sits down with the same checkout flow and starts asking questions instead of running scenarios. Twenty minutes in, they find a way to check out for a fraction of the listed price, on a path the scanner never touched.

That's the structure of automated validation in a continuous threat exposure management (CTEM) program. Breach and Attack Simulation (BAS) platforms and automation confirm whether controls hold against known attack patterns.

But chained exploits, business logic flaws, novel attack paths, and AI-specific weaknesses like prompt injection don't live in any library. The data makes that gap visible, and the gap is expensive.

The checkout flow is a small version of a much bigger pattern. Prompt injection reports grew 540% in a single year. The number of HackerOne customer programs bringing AI into scope, or reporting a valid AI finding, grew 270% over the same stretch.¹

That growth demonstrates what an exposure class looks like when only a human, asking the kind of question no scenario script would think to ask, can find it.

Why the Ceiling Is Structural, Not a Gap You Can Patch

The category was built to replace BAS and automated penetration testing because both were limited to known scenarios executed at scale. AEV does more of that, faster and at greater breadth. But it is still a machine confirming known attack patterns. That is a precise and valuable thing, but not the same thing as finding what nobody thought to test for.

It also does nothing to close the gap between discovery and remediation. AEV confirms a finding exists. It does not ensure the right team sees it, understands it, or fixes it. When vulnerability submissions are up year over year and remediation throughput hasn't kept pace, a tool that finds more things faster compounds the backlog rather than resolving it.

That distinction matters because the ceiling isn't a product limitation a vendor can ship around. It's structural. Automated validation executes scenarios from a library that only holds what someone already thought to put in it. Three finding types live entirely outside that boundary:

1. Chained exploits. A single low-severity finding rarely triggers a review. Three of them, combined in a sequence an attacker discovers by exploring rather than running a script, can produce a critical compromise. AEV tests scenarios, but does not improvise a four-step chain across systems that were never designed to be tested together. HackerOne's researcher community ranked multi-step, chainable vulnerabilities second among the nine categories AI tools handle worst, named by 39% of researchers.¹
2. Business logic flaws. The checkout flow from the opening is one of these. Instead of code, the vulnerability lived in a design assumption nobody wrote down and nobody tested. There's no signature to match because the flaw is unique to how that specific application was built. In the same researcher survey, business logic ranked first, named by 58% as the category AI tools handle worst.¹
3. Novel attack paths. Every environment has a configuration that exists nowhere else, like a specific stack of cloud services, internal tooling, legacy systems, or custom integrations. An attack path through it has never been scripted because it has never existed anywhere else to script. Where AEV scenarios generalize, real attackers can specialize.

The Cost of Skipping the Human Layer

The vast majority (94%) of organizations expanded their AI footprint in the past year, yet only 66% formally test more than 60% of what they deployed. HackerOne's research shows what sits on each side of that line: organizations testing 91% or more of their AI systems are 16% less likely to report an AI-related attack or vulnerability than the ones testing less, and the gap in expected annual impact comes out to roughly $730,000 a year.²

That number is not the cost of a single incident. It is the annualized difference in expected impact between organizations with strong AI testing coverage and those without. For a CFO evaluating whether to fund a second layer of human-led testing, it is the right denominator: not the cost of the program, but the cost of the gap the program closes.

Image

Coverage doesn't make a single incident cheaper to clean up. It makes the incident less likely to happen at all.

Part of what drives that difference is a dynamic no scenario library can replicate. Using the H1 Platform, security researchers have logged more than 580,000 valid findings across its history, the share of researchers focused on AI and ML systems more than doubled in a single year, from 9% to 19%, and rewards paid for valid AI findings grew 339% over the same stretch.¹ 

A scenario library updates on a quarterly release schedule, but the researcher community updates the moment someone finds something new, and that finding immediately becomes part of what the next researcher tries on the next target. The collective intelligence compounds in a way no vendor roadmap can match.

That compounding happens because researchers are not just a supply channel, but the source of the intelligence itself. Every novel technique in that collective pool was first found by a person who was curious enough to ask a question no script had thought to ask. The 580,000+ valid findings on the H1 Platform are not a metric HackerOne owns; they are work the researcher community produced, and the speed at which that knowledge compounds is inseparable from the incentives that keep skilled researchers engaged.

That gap in adoption is where it shows up. Seven testing methods make up a mature AI security program. Bug bounty and crowdsourced AI testing, the only method on the list that is continuous, human-led, and adversarial by design, is the least adopted of all seven, used by just 29% of organizations. AI red teaming sits at 55%.²

The security leaders most confident in detecting AI-specific attacks in real time, and the ones most prepared for emerging AI governance requirements, both lean disproportionately on those human-led methods as part of a full seven-method stack. The methods organizations trust most under pressure are not the methods most organizations have actually deployed.

Build a CTEM Program Combining Automated Validation and Researcher-Led Continuous Offensive Testing

HackerOne tracked 68 programs that cut bounty payouts by 20% or more between 2018 and 2025. Valid submissions fell by an average of 22%, and critical-severity submissions fell by half.¹

Automated tooling doesn't disengage when a budget tightens, but a researcher community does, because the work only happens where the incentive exists. The tradeoff a CTEM program makes when it underfunds the human side is a measurable drop in exactly the findings that represent the highest risk.

The solution is recognizing that AEV and human-led testing aren't answering the same question. Ask AEV whether your defenses hold against everything you already know to test for, and it will tell you, reliably and at scale. Ask it what you haven't thought to test for yet, and it has nothing to say. That's the harder question, and for most organizations, the bigger threat.

In practice, run both layers in parallel with AEV aimed at your known control set continuously and bug bounty and pentesting seeking potential issues AEV can't see: AI deployments, business logic-heavy applications, anything recently shipped, anything where no scenario library has caught up yet.

Image

Guide

The Beginner's Guide to Bug Bounty Programs

Download the guide

Skip the human layer and your program will pass every test it designed for itself and stay exposed to every test it never thought to design.

What a Complete CTEM Program Looks Like in Practice

Automated validation confirms what you know. It cannot find what nobody thought to test for. That gap is structural, and it is where the most consequential findings live.

Three steps can help close it:

• Audit what your scenario library actually covers and name what falls outside it.
• Add AI red teaming and bug bounty if they are not already in your stack.
• Run both layers in parallel, continuously, with automated validation on known controls and offensive testing on everything else.

A complete program tests what it designed for and what it didn't. Right now, most programs only do one of those, and the gap shows up not in what gets found, but in what stays unresolved. An AI-only CTEM program that surfaces vulnerabilities faster than teams can validate, prioritize, and fix them does not reduce risk. It relocates it from the attacker's side of the ledger to the remediation backlog. The goal is not more findings, but fewer unresolved ones.

See our Complete Guide to CTEM to structure a program that covers both layers

^{1. Hacker-Powered Security Report 2025: The Rise of the Bionic Hacker}

^{Survey methodology: HackerOne and UserEvidence surveyed 99 HackerOne customer representatives between June and August 2025. Respondents represented organizations across industries and maturity levels, including 6% from Fortune 500 companies, 43% from large enterprises, and 31% in executive or senior management roles. In parallel, HackerOne conducted a researcher survey of 1,825 active HackerOne researchers, fielded between July and August 2025. Findings were supplemented with HackerOne platform data from July 1, 2024 to June 30, 2025, covering all active customer programs. Payload analysis: HackerOne also analyzed over 45,000 payload signatures from 23,579 redacted vulnerability reports submitted during the same period.}

^{2. Closing the AI Security Gap: Containing Risk Before It Scales}

^{Survey methodology: HackerOne surveyed 303 security leaders between January and February 2026. Respondents were screened to ensure they oversee or contribute to tracking, managing, or testing their organization’s AI/ML systems, and represent a range of senior security and offensive security roles within organizations reporting $250 million or more in revenue across the United States, Canada, the United Kingdom, Australia, Singapore, and Germany. Respondents represented multiple industries, led by Technology Hardware/Software (37%) and Banking/Financial Services/Insurance (16%), with additional representation across manufacturing, healthcare, retail/e-commerce, and other sectors.}