Project Glasswing: What We Learned Running a Frontier Model on Our Own Codebase

HackerOne joined Project Glasswing to advance cyber defense with frontier AI, applying Claude Mythos 5 to our own production environment. In July, we stood up a sandboxed pipeline against our own code and ran experiments over 30 days.

Here's what we found.

A Critical RCE Almost Went Undetected

On its first run against our code base, Mythos surfaced a critical, no production exposure Remote Code Execution (RCE) vulnerability. We remediated it in under 48 hours. Though deployment conditions prevented it from being exploitable, a change made on any given day could have made it reachable.

The interesting part wasn't the finding itself. It was how the vulnerability got there.

The RCE came from three individually safe code changes:

• A dynamic method dispatch, protected by a strongly typed GraphQL schema enum. Safe in the context of the enum.
• A new analytics version introduced an untyped filter argument. Safe in the context of the new analytics filter.
• A refactor consolidated the two. Safe in the narrow context of the refactored components—but not across the systems it connected. 

The type system assumption dissolved. It routed the untyped input into the first query builder, which had never needed sink-level allowlisting. The type system was supposed to handle that upstream. The RCE only existed when all three changes were in production at the same time.

We think about this as compositional risk: vulnerabilities that don't live in any single commit, but in how safe-in-isolation changes interact over time. Traditional code review evaluates pull requests in isolation; it's structurally blind to this class of problem. Whole-program static analysis had a better chance of finding something like this, since it reasons over the entire tree rather than one diff at a time. But these tools struggle to model custom sinks, and often cannot resolve dynamic dispatch.

What Mythos did was follow the thread across commits and authors until it found where the logic had come apart. It traversed repository history, reconstructed developer intent across authors, and applied offensive security reasoning to the full call graph. Not just the diff.

Three Things We Learned Running a Frontier Model on Our Own Code

After the RCE finding, we ran a series of structured experiments. Three things stood out.

1. How Models are Evolving

Mythos not only found more vulnerabilities than earlier models, but also more sophisticated ones. We ran this across two independent harnesses to rule out a setup artifact. The gain is in recall. It catches more of what is actually there with comparable precision.

2. Remediation Is Where the Real Work Is

A few weeks of scanning produced hundreds of findings. This is where validation, confirming a vulnerability is exploitable and assigning a severity to it, can become a bottleneck for organizations.  The model generates findings faster than teams have historically been resourced to triage, route, and patch them. Innovation and acceleration in validation and remediation are essential to keep up with frontier model discovery capabilities. 

3. Evolving Our Harness and Guardrails

Running Mythos confirmed what we keep relearning: as models advance, the work shifts. As models have advanced, the work that used to go into telling it how to think now goes into reviewing what it finds and governing how it presents those findings. 

Today we separately track findings the model flagged and then talked itself out of. That lets us distinguish a validation layer that's filtering noise from one that's suppressing real issues. We also added a coverage gate: a run that examined only part of a codebase can produce findings that look just as complete as those from a run that covered all of it, so the harness now fails runs that didn't review enough of the surface before reporting.

A final learning is about memory. As models can now follow years of commits across multi-repository systems, we’ve had to build guardrails around how they interpret contribution history. An example of this is what we call blameless root-cause-analysis (RCA): a principle that security vulnerabilities are introduced by the system and the work, not a person. That holds even when an AI agent writes the code. The agent is an extension of the developer's toolkit, and the developer's name stays on the commit. The harness enforces that framing.

Compositional Risk Scales With Your Codebase, and Many Teams Aren't Ready

Compositional risk is not a HackerOne-specific problem. Any organization shipping code at speed is accumulating implicit assumptions across subsystems, authors, and time. Most hold. Some don't, and the ones that don't won't be visible until the right conditions align.

What Mythos represents is a tool that can reason across that accumulation. It doesn't just flag a bad line of code. It reconstructs the full context of why something is dangerous, across multiple authors and commits over time.

Building the Infrastructure to Act on What Frontier Models Find

Running Mythos against our own platform was the start. The larger work is building the infrastructure to act on what it finds: validation at scale, routing that gets findings to the right engineering teams without friction, and fix verification that closes the loop rather than just closing tickets.

Our participation in Project Glasswing is scoped exclusively to our own infrastructure. Researcher trust is important to us, so we want to be clear about how confidential data is handled across the platform: HackerOne does not use confidential researcher submissions or customer vulnerability data to train, fine-tune, or otherwise improve generative AI models.

You can find more information in our Hai Security and Trust documentation and our Responsible AI at HackerOne blog.