Measuring The Fix: How We Evaluate Frontier Models on Remediation Quality
We evaluate each new frontier model against the hardest thing our remediation product does: write a security fix an engineer will trust and merge. Our main observation is that the choice of judge moved our scores more than the choice of model, and that the leading models were separated by less than our measurement noise. This post is about how we built an evaluation we can trust, and how a trustworthy measurement changed what we could improve.
Critical vulnerabilities are being resolved faster than ever. On the H1 Platform, average resolution time for critical reports fell 61% between June 2024 and June 2026. Over the same period, open valid reports across all severities grew 131%, and open critical reports grew 123%. Speed gains are real, but volume is outpacing them. The value now concentrates on what happens next: turning a validated finding into a verified fix that an engineer reviews, accepts, and merges.
That is why organizations are focusing more on remediation as part of continuous threat exposure management (CTEM): find the vulnerability, validate it, find the root cause, remediate it, confirm the fix held, repeat. Remediation is the step where risk actually comes down, so it is the step we hold to the highest bar.
We evaluate frontier models against that bar. This post covers what a controlled comparison of those models tells us about remediation, and how those findings advance our harnesses and evaluation frameworks.
The questions that drive model evaluation:
1. Which model writes the best fixes?
2. Which harness around the model performs best with it?
3. And can the difference be measured consistently and reliably enough to trust the answer?
The path from the first question to the third is where we’ve learned the most and what we’ll cover here.
HackerOne does not use confidential researcher submissions or customer vulnerability data to train, fine-tune, or otherwise improve generative AI models. You can find more information in our Hai Security and Trust documentation and our Responsible AI at HackerOne blog.
It starts with the harness
The model is one component, and the harness acts as an operational layer. The H1 Platform's harness sits between the frontier cyber AI model and the security team: it scopes what the model can touch, validates what it finds against noise, and traces every confirmed finding to a fix a developer can act on. In other words, the scaffolding around it decides most of what you observe: how you present the finding, how many steps the agent gets, whether analysis or the fix comes first, and what happens when it returns nothing. As the rest of this post shows, we run our harnesses across different models and then improve the harnesses themselves to get the most from each.
Figure 1. The harness is the scaffolding around the model: it takes a finding, runs root-cause analysis, and produces a fix we can grade. We run this same pipeline across different models on the same cases with the same scoring, which lets us attribute a change in results to a specific change in the model or the harness with more confidence.
The test cases determine what specific comparison is being made. Models tend to score higher on publicly available benchmarks and disclosed open-source vulnerabilities, in part because that material is well represented in their training data. We built our set with this in mind, drawing on vulnerabilities our own engineers fixed rather than on public benchmarks. The set spans more than ten weakness classes, including access-control flaws, IDOR, XSS, CSRF, code injection, HTTP request smuggling, resource exhaustion, information disclosure, and business-logic errors. Severities range from low to critical, and no single class makes up more than about a quarter of the set. Each real vulnerability is paired with the patch our HackerOne engineers actually shipped. Roughly 40% of cases contain no vulnerability at all because every false alarm costs a security team time: the agent has to decide whether a finding is real before deciding how to fix it. Every model is graded on the same shared set for more than 1,000 tests in total.
Reading the patches ourselves
Our first move with each frontier model was to read the work rather than blindly trust a score. We pulled a 20% sample, weighted toward outliers and the lowest-scoring cases, and read each model's patch side by side with the fix our engineers shipped. The cases spanned generic classes, such as HTML injection, as well as findings that require real domain knowledge, such as an authorization check in a GraphQL layer.
What we read did not match what the scores said. In many of the cases we reviewed, our engineers judged the newer frontier models’ fixes comparable to the patches we shipped, and some were more comprehensive. Yet the judges were scoring them lower. The reason was in how the scoring was weighted. The judge already considered several dimensions, but localization signals carried too much of the score. A patch was rewarded for landing on the same files and lines as the human-written fix, and a correct fix that solved the problem at a different location was penalized. Our measurement was biased toward patches that resembled one reference fix, rather than weighting most heavily whether the vulnerability was actually fixed.
If a genuinely better fix could score poorly for the wrong reason, then comparisons built on that score were suspect, and the work to do was on the evaluation rather than the model. This signaled to us that refinements were needed for the judge.
Refining how we evaluate
Grading a security patch well is a research problem in its own right. A vulnerability often has more than one correct fix, and the most convenient ways to measure quality capture less of what matters than they appear to. Grading ‘it was fixed’ is far easier than ‘it was fixed the right way’. Improving the judge meant working on three fronts at once, and reading the patches had pointed at all three.
First, what the judge measures. We set out to replicate, inside the judge, the reasoning our engineers used while reading patches by hand, and to do it without over-tuning to any one test set. Our scoring advanced through three approaches, each a response to a measured limitation of the last.
Figure 2. Three approaches to scoring, each one a deliberate response to a measured limitation of the last. Overlap is exact, but credits only fix that land where our engineers did, undercounting a correct fix placed elsewhere. A single judge that asks, “would we ship this?” recognizes quality that overlap misses, but its score varies from run to run about as much as the models differ from each other. Decomposing the question into nine independent checks, each run multiple times, gives a stable measurement precise enough to detect the differences we report.
The advance was to decompose one broad question into nine specific ones. Two are hard gates: if the patch does not apply, or the vulnerability still reproduces, the score is zero, regardless of how clean the code looks. The other seven mirror the questions a reviewer actually works through:
- Does it fix the underlying cause, or only block the exact example in the report?
- Does it also cover the other places in the code with the same flaw?
- Is the fix robust, or can an attacker work around it?
- Does it avoid blocking normal, legitimate use?
- Does it follow the conventions of the codebase it lives in?
- Can it be peer-reviewed by another engineer?
- Is it safe to use without causing a regression?
None of these match the human patch's location. They ask whether the vulnerability is fixed and whether the fix is sound, which is exactly what the spot checks had shown the old overlap score was missing. As a bonus, the decomposition is diagnostic: a single number tells you a model scored poorly, while nine separate answers tell you how, and the disagreement rate across the nine is itself informative.
Figure 3. Judge disagreement varies sharply by question. “Did the vulnerability stop reproducing?” is close to deterministic, while “is this cleanly reviewable?” carries the most variance. Knowing which checks are stable and which are noisy tells us where the repeated runs earn their keep, and where a single-judge score would be least reliable.
Second, who is doing the judging. Reading the spot checks surfaced a bias that lived in the judge models themselves. When a model does the grading, who grades the grader? A single judge introduces a conflict of interest, because some of the models under test share a family with the judge scoring them, and we saw it directly in our own data: a judge graded a same-family patch more generously than an equivalent patch from another family. So we grade with a panel of six judges from different families, no model ever grades a member of its own family, and we take the median rather than the mean so the panel is never more confident than the judges within it. We also check whether the judges are genuinely independent, because adding judges only helps if they make different mistakes; the most valuable pairs are the ones that agree least.
Third, how stable a single reading is. Rerunning the same judge on the same patch did not always return the same score. So we run each check three times per patch and resolve it by majority vote, a standard way to reduce the run-to-run variance inherent in model-based grading. It is not a formality: across thousands of sampled checks, roughly one in eight returned a split vote across its three runs. Scoring a check only once would leave a meaningful share of results decided by that variance rather than by the patch.
Figure 4. The same set of patches, graded by six judges. The spread between the most demanding and most lenient judge, roughly a factor of two on the same work, is wider than the spread between the best and worst model in the comparison. In other words, the choice of judge influences the reported score more than the choice of model does. This is the central argument for a panel with median consensus: any single-judge leaderboard reflects that judge's calibration as much as the models it ranks.
On identical patches, the choice of judge moved the score more than the choice of model. That is why we grade with a panel, run in triplicate, and prohibit a model from grading its own family.
What a trustworthy judge let us catch
With a judge we could finally trust, we re-analyzed the results, and a new opportunity surfaced. In one comparison, a strong frontier model matched an earlier model only on writing patches, despite being clearly better at root-cause analysis, on the same cases and setup. Understanding the vulnerabilities better while producing no better patches is a signal worth chasing down, and the decomposed score let us do exactly that: because the nine checks separate “produced a weak patch” from “produced no patch at all,” the cause was quick to isolate.
In cases where the model had pre-confirmed the vulnerability, it was returning an empty patch a meaningful fraction of the time. Those were not failures of understanding: each carried a complete written explanation of the fix, the approach, the steps, and the verification checks. What was missing was the code itself, in an applicable form. Two causes, neither a limitation of the model. The step budget was allocated unevenly, generous for analysis and tight for writing the patch, so a model that explores thoroughly could exhaust its budget locating the right code before writing anything. And the handoff between stages was lossy, passing along the prose while dropping the code references, forcing the patch stage to re-locate code its own pipeline had already found.
We addressed both and shipped the result as a model-tuned harness, keeping the earlier one intact so prior comparisons stayed valid. The lesson generalized: to get the most from a model, you adjust the harness and the resources around it, and a precise evaluation is what turns an odd-looking number into a specific, fixable cause. Different models call for different machinery.
The self-improving harness
Adjusting a harness by hand for each model does not scale, so we built a harness that automatically improves its own prompts. It seeds from published harnesses for these models, uses our existing scores as the baseline, and proposes and tests many prompt variants, accepting a change only when the data proves the improvement holds at scale rather than on a favorable slice.
The order of operations matters. We had tried a version of this loop earlier, against the original overlap-weighted scoring, and it produced little of value, because the signal it was optimizing was shaped by the same measurement issues the patch review had surfaced. Once the refined judge was in place, decomposed, family-aware, and run in triplicate, automated prompt improvement produced real, repeatable gains.
The optimizer also never saw the findings it is scored on. We split the cases into a tuning set the loop iterates against and a held-out set it never touches, and a prompt change is accepted only if the gain appears on the held-out set. Without that split, a loop like this learns the test set rather than the task.
Figure 5. Automated prompt improvement moved every graded dimension in the right direction on held-out findings, while the gated boundary check held throughout, so no gain came at the cost of safety. The point worth underlining is not the size of the gains but their dependence on the evaluator: they became measurable only once the scoring was stable enough to see them. Against noisier scoring, the same loop optimized against its own noise.
The clearest gain was in recall. Scored by the new judge, the self-improving harness delivered a usable fix on far more findings than before, including the cases where it had previously returned nothing, while precision held steady with little to no cost. That is a result that the first, weaker judge could never have produced. It would have chased its own noise and reported improvements that were not there.
Figure 6. Coverage and quality are distinct measures, and our scoring tracks them separately. Iterating on the harness produced a sharp gain in coverage, the share of findings for which the agent delivers a patch at all. Quality, whether that patch is ready to ship, improves on its own, on a slower track. Measuring both is what lets us direct effort precisely rather than optimize one at the other's expense.
A good judge does more than rank models honestly and separate “produced a patch” from “fixed the vulnerability,” so a rise in recall cannot hide a fall in precision. It is the foundation that the harness stands on to keep improving. Without a measurement more precise than the effect you are chasing, a self-improving system optimizes against its own noise; with one, it compounds.
Putting frontier AI to work on remediation
H1 Remediation is developed under continuous, data-backed research. Evaluating frontier models is what forced us to revise the evaluation itself. The models were good enough that our old scoring penalized their best work, which is why we rebuilt the judge before trusting any comparison. This is how the loop closes in practice: a validated finding in, a verified fix out, confirmed rather than assumed. Every model is graded on the same set of findings by the same panel and rubric, so the ranking reflects the fixes, to within our measurement noise.
The leading frontier models cluster tightly on remediation quality, separated by less than measurement noise, so the large generational gaps visible on public coding leaderboards do not appear on the harder, more level problem of writing a patch a security reviewer will ship. And across the board, evaluation design moved the reported result more than model choice did, which is why the judge, not any model, is the instrument we tend most carefully.
Evaluating each new frontier release as it arrives is a standing part of how we work, and it keeps our harnesses and our scoring ahead of models that improve faster than the tests built to measure them. What all of this adds up to is product quality. Our evaluation asks the questions an engineer asks: does the patch apply cleanly, does the vulnerability stop reproducing, and is the fix likely to avoid regressions. H1 Remediation is built and tested against that standard. And because we run this evaluation on every new frontier release, H1 Remediation is designed to improve as the models do, so customers do not need to run their own comparisons.
Learn more about H1 Remediation
Limitations. These results come from an internal benchmark and may not carry over to other codebases, harnesses, or model versions.They are shared for information only. They are not a warranty about how any model or HackerOne product will perform, and they do not replace engineer review of a fix before it is merged. Product and company names are trademarks of their owners, are used only to identify the products tested, and do not imply endorsement or affiliation.
About the Authors
