Skip to main content

Improving Signal Over 10,000 Bugs

  • July 6th , 2015

On July 2nd, 2015, HackerOne reached 10,000 valid security vulnerabilities resolved on our platform - that's 10,000 fewer ways for attackers to exploit users, steal data, and cause damage. The first bug was filed on the HackerOne platform 20 months ago, and today over 250 organizations resolve hundreds of bugs each week thanks to the help of thousands of researchers around the world.

Reflecting on the journey to 10,000 bugs, we thought it would be interesting to share some of the ways we've tackled the challenge of increasing signal on a fast growing platform. As a team, we spend a great deal of time improving automation and refining our platform algorithms to increase signal and enhance the customer experience.

Defining Signal

To help clarify our process, we can start by looking at how we categorize signal. At the highest level, we categorized vulnerability reports into three groups as response teams close them:

  • Clear Signal - Vulnerability reports closed as "resolved". This means the issue was a valid security bug that was fixed by the vulnerability response team.
  • Nominal Signal - These reports are closed as marked "won't- fix" or duplicates of resolved issues. While not contributing to clear signal, many of these reports were technically accurate based on the best information available to the researcher.
  • Invalid - These reports are closed as "Not Applicable," "Spam" or duplicates of these types. This represents the noise in the signal to noise ratio.

HackerOne Platform Signal

The chart below shows HackerOne's clear signal compared to that of other programs sharing data publicly. From left to right, we see that the programs from GitHub, Facebook and Google document signal rates of 4%, 5%, and 7%, respectively. HackerOne's baseline clear signal percentage was twice that of Google's and roughly 3 times that of GitHub and Facebook's.

HackerOne vs. other public programs

Improving Signal with Reputation

After benchmarking the platform, we introduced Reputation, a system that scores researchers based on the validity of submitted reports. As expected, it had an immediate impact on improving signal. Three months after we introduced Reputation, we added basic Rate Limiting, which works by capping the number of reports that a low reputation researcher can submit in a given time frame. Our current algorithm is extremely lenient toward researchers who submit invalid reports. In other words, we can still make significant improvements to our signal ratio by further highlighting high-quality researchers and guiding new and low-signal researchers towards better behavior.

HackerOne quality with Reputation and Rate limiting

The chart above shows the progression of signal improvement we achieved by first adding Reputation and later Rate Limiting. The left-hand columns show a baseline of 14% clear and 33% nominal signal before adding Reputation. The middle columns show that adding Reputation raised clear signal to 20% and nominal signal to 41%. The right-hand columns show the impact of next adding Rate Limiting onto the platform, which improved clear signal to 23%, while nominal signal held steady at 41%.

Distribution of Reputation Percentile across Report Quality

The chart above shows how higher-reputation researchers drive the majority of clear signal on HackerOne. Inspecting the left-hand column, you'll note that top 10% of researchers (top three sections in column) find 80% of valid vulnerabilities. By contrast, the right-hand column shows that the bottom half of researchers (lower section in column) account for nearly 60% of all invalid reports. With these new insights into our platform, we can better inform our future endeavors to refine the experience for both researchers and organizations.

Invitation-only Versus Public Program Signal

In contrast to limiting the number of invalid reports in our publicly-listed programs, our invitation-only programs draw participation only from our top reputation hackers, thus creating a high-signal environment for organizations. The chart below separates public and invitation-only programs on HackerOne, listing their respective signals. In public programs, the sum of nominal and clear signal is 53% compared to 85% in invitation-only programs.

HackerOne Report Quality: Public vs. Invitation-only

10,000 Bugs and Beyond

HackerOne is committed to applying what we've learned from the first 10,000 bugs resolved to improving the platform for the next 10,000 bugs found and beyond. We look forward to helping organizations become more secure, creating opportunities for hackers, and making the Internet safer for all. From all of us at HackerOne, thank you 10,000 times.

Recent articles

Bug Bounty Field Manual: The Definitive Guide for Planning, Launching, and Operating a Successful Bug Bounty Program

H1-415 Hackathon Delivers to Customers, Community, and Hackers

Just a few short weeks ago, an elite group of hackers huddled in conference rooms in a San Francisco high-rise…

Introducing CWE-based Weaknesses

HackerOne updated their vulnerability taxonomy to include a more complete weakness suite based on the industry-…