Skip to main content

The HackerOne Success Index - Vulnerabilities Fixed

  • November 23rd , 2015

In case you missed it, check out the first installment of our blogpost series on the HackerOne Success Index.

We recently introduced the HackerOne Success Index, a method to measure the effectiveness of HackerOne-powered vulnerability disclosure programs. The index calculates values from 1 to 10 across six dimensions by which programs can benchmark their success each month. In the first of our deep-dives into the dimensions of the HackerOne Success Index, "Vulnerabilities Fixed" describes the quality and frequency of security improvements from a vulnerability disclosure program over time. Vulnerabilities Fixed is a strong indicator of both the maturity of the overall program and security of the application, since all other index measurements will affect it to varying degrees. The number of vulnerability reports and the breadth of vulnerability types fixed make up this dimension, and are weighted for recency, giving newer reports a higher impact on the index. We take a deeper look at these two factors below.

Number of Vulnerabilities Fixed

Number of Vulnerabilities FixedNumber of Vulnerabilities Fixed

In the chart above you see the average number of resolved reports in the last six months for HackerOne programs within two Vulnerabilities Fixed index bands, high performers between 7-9 and mid-level between 4-6.

We see companies constantly ship new products, features, and updates which can include new vulnerabilities; these two groupings both contain large and small companies from a variety of industries that incentivize persistent examination of continuously changing code. The upper group is averaging a little over 20 vulnerabilities fixed each month, while the middle tier resolves about 6 reports per month.

A long-term commitment to your program encourages researchers to stay involved and surface harder-to-find vulnerabilities. The Vulnerabilities Fixed dimension of this index favors a steady and continuous volume of high quality reports. A program's month-to-month count of resolved vulnerability reports is the most heavily weighted input because this most directly translates to enhanced security as issues are surfaced and fixed.

How do other dimensions affect Vulnerabilities Fixed?

Metric Correlation strength
Researcher Breadth +++
Researcher Depth +++
Reward Competitiveness ++
Response Efficiency +
Signal Ratio +

**Pearson Correlation table representing level of positive correlation between Vulnerabilities Fixed and other HackerOne Success Index dimensions. Correlation does not imply causation, only that some positive relationship exists between dimensions.

As we mentioned earlier, the Vulnerabilities Fixed dimension is directly affected by other dimensions, which we will explore in future blog posts. We don't have causal proof in the data yet, but we can point to very strong positive correlations in the table above. You can see that improving your performance in any of the other HackerOne Success Index dimensions, but especially Researcher Breadth and Depth and Reward Competitiveness, is generally associated with increases in your Vulnerabilities Fixed dimension. Some common tactics include: inviting more researchers periodically (if your program is invitation-only); broadening your program's scope so that researchers have new challenges to focus on; and increasing your rewards over time to match researchers' greater time investments.

What variety of vulnerabilities are being found and fixed?

Vulnerability Type DiversityVulnerability Type Diversity

The chart above shows the average number of unique vulnerability types that are resolved each month by HackerOne programs in the same two Vulnerabilities Fixed index bands, 7-9 and 4-6. Teams that fix a greater variety of vulnerabilities at volume will also improve their performance in the Vulnerabilities Fixed dimension, reflecting enhanced security for their products and properties.

The HackerOne platform currently offers 15 vulnerability types (as well as a None Applicable catch-all that we won't be examining here) for reporters to choose from. Nearly 10% of all fixed vulnerabilities represent rare but severe issues like Remote Code Execution, SQL Injection, or Privilege Escalation, along with a large number of more common bug types. Our data show that our most successful programs address on average about 13 different types of vulnerabilities each month, while mid-range programs average 8 per month.

As always, we welcome any questions and feedback, and we'll be back in a couple weeks with a deep-dive into the next HackerOne Success Index dimension, Reward Competitiveness.

– HackerOne Customer Success and Data Science team

Recent articles

Announcing The Largest DoD bug bounty challenge ever: Hack The Air Force

The Air Force is asking hackers to take their best shot following the success of Hack the Pentagon and Hack the…

Zero Daily Newsletter: Fun, yet informative, AppSec, bug bounty, and hacker news

Read the news every day, and check the usual websites? Want to get your industry news and have a little humor…

More Hardware, More Problems

Bounties are for hardware, too. Microwaves notwithstanding, there is an increasing amount of connected…