At HackerOne’s 2021 Security@ conference, two experienced HackerOne program managers, Allie Lugton and Denzel Duncan held a session on tracking and interpreting data from bug bounty programs.
Allie and Denzel explained how organizations can leverage data to maximize their programs’ security and development value.
The Three Phases of a Bug Bounty Program
Bug bounty programs have three distinct phases.
Phase 1: Preparation
During the first phase, your organization’s team will develop the overall program, including the following:
- Security page—sets out the rules of engagement and participation criteria for ethical hackers.
- Program scope—tells hackers which assets they can work on and the techniques they can (and can’t) use.
- Rewards—provide the return on investment (ROI) for hackers.
During this phase, your organization will also set up integrations with existing vulnerability management and developer tools such as JIRA or ServiceNow and establish responsiveness targets for your security and development teams. These targets will allow you to track program performance over time. Ideally, make them challenging but realistic, and consider tightening them over time.
Phase 2: Launch
Launching a bug bounty program is a big step, and it’s important to avoid overwhelming security and development teams. To do this, take a step-by-step approach, starting with a small private program and gradually inviting more hackers to participate, giving your teams time to identify gaps in existing vulnerability management processes without being inundated with reports.
During this phase, common metrics include:
- Report volume
- Valid report volume
- Report volume by severity
- Number of hackers invited
- Number of hackers accepted
If you spot a problem, work with your program manager to identify adjustments to help you get back on track. For example, if you’re receiving too many low severity reports, you might consider adjusting the program scope to exclude less critical assets.
Phase 3: Growth
As your program settles in and you establish KPIs, you’ll naturally shift into the Growth phase. During this phase, you’ll conduct ongoing reviews to ensure your program remains active and effective.
Track metrics that will help you ensure your program is achieving its overarching purpose. For example, if your objective is to harden web-facing assets against common threats, track the categories of vulnerabilities reported and compare them to OWASP's Top 10 web application security risks.
Common KPIs to track include:
- Report volume
- Valid report volume
- Report volume by severity
- Vulnerabilities by category
It’s also important to track how quickly you’re acknowledging, validating, and fixing reported vulnerabilities and how long it takes you to pay bounties. These metrics play a crucial role in keeping hackers engaged with your program, so you should aim to maintain consistently high responsiveness.
Spotting Patterns and Trends
Like all HackerOne program managers, Allie and Denzel are experienced at uncovering the meaning behind program data and helping organizations take appropriate actions. Some of the common trends (and causes) they encounter include:
A large volume of duplicate reports could indicate an issue in the remediation cycle. If it took a team a long time to verify and remediate vulnerabilities, the chance of receiving duplicate reports would increase, causing internal teams to spend time triaging duplicates. Security teams and hackers would be frustrated. Since hackers aren’t paid for duplicate findings, they might feel they had wasted their time finding and reporting those issues.
Trending vulnerability categories across multiple assets suggest a root cause that needs to be investigated, e.g., there is a need to train developers on a particular issue.
Trends in the severity of reported vulnerabilities for an asset can suggest several things, including:
- A high volume of simple vulnerabilities suggests a program may have a low security maturity.
- A steady volume of critical findings may indicate an asset maturing and requiring more complex approaches and techniques.
While these are common findings in bug bounty program data, they are just examples. To maximize the value of your bug bounty program, work with your program manager consistently to uncover and respond to issues.
Using Data to Connect Security and Development Teams
The relationship between security and development teams is critical to a program’s success. Program data can help you understand where there may be breakdowns in collaboration and implement solutions to help the teams work together effectively to support vulnerability management objectives.
Important metrics to track include:
Time-to-resolve (TTR) vulnerabilities. As a program becomes more established, TTR should naturally fall. A high TTR suggests a breakdown somewhere in the remediation process. Tracking this metric can help program leaders recognize when issues arise and investigate to ensure the appropriate processes, collaboration, and SLAs are in place.
Types of valid vulnerabilities reported. If a particular category of vulnerabilities is reported regularly, this suggests there may be holes in existing vulnerability management processes. Typically, organizations can solve this either by training developers to avoid producing similar vulnerabilities in the future or by tightening code review processes to catch them earlier in the development cycle.
Training and empowering developers to write secure code is crucial. Allie explained during the session:
“We are always looking at data trends that come out of a program. This data is imperative to the maturation of any bug bounty program. Look at remediation times for valid vulnerabilities and see how long it takes development teams to address tickets and use the data to push where needed. Bring back trends on most commonly introduced vulnerabilities and train development teams to develop code without introducing these whenever possible.”
HackerOne integrates with companies like HackEDU, which offer secure code training for developers based on trends uncovered in program data to make this process easy.
Bolster Security with Data
The ultimate goal of most bug bounty programs is to improve the organization’s security profile. To ensure your program achieves this consistently, you should pay close attention to data trends and take prompt corrective actions where necessary.
We’ve explored several ways to use program data. Still, the core message from the session with Allie and Denzel was clear: your program manager is ideally positioned to help you use data to improve your bug bounty program.
They know how to track, interpret, and act on bug bounty program data and have helped many organizations build lasting, effective relationships with hackers that support critical security objectives.
Discover the Best Kept Secret in Cybersecurity
Register here to watch the entire training session on using bug bounty program data to improve security. Also, explore other on-demand sessions from Security@ 2021, our 5th annual global cybersecurity conference, including presentations, roundtable discussions, and training sessions focused on how your organization can work with the best-kept secret of the cybersecurity industry: ethical hackers.
