(Best) Practice Makes Perfect

Everyone at HackerOne has the goal of making sure that hackers and enterprises are partnering together with excellence. The role of the Chief Hacking Officer at HackerOne is to assist with this goal, serve as a point of escalation, and make sure we learn from unusual edge cases. And we certainly have seen some interesting cases! We’ll look at six real situations that arose and how we handled them to get the best outcome for both the hacker and customer.

An important part of this work is ensuring that programs across the HackerOne platform operate consistently, giving both hackers and customers predictability and fairness, based on transparent principles and rules. To support this, we have published our Best Practices For Programs. In this post, we’ll recap what best practices are and then look at six unique situations that helped us to refine our documentation.

HackerOne documents our best practices and reviews and updates them regularly, based on hacker and customer feedback, as well as drawing on the experience of leading bug bounty programs across the industry. These best practices include practices that we expect all programs to adhere to. There are also some other practices that are not universal, but the best programs tend to follow them. The primary goal of having published best practices is to ensure excellent outcomes for hackers and customers alike. Treat hackers fairly and accurately, and you’ll get more engagement, remediate more vulnerabilities, lower your risk of breach, and attain a better security reputation. Best practices give our Mediation team the tools to resolve common situations quickly. When a program deviates from a baseline expectation, the mediation becomes an educational note to the program. Or, when a program faces a best practice that is not mandatory, but strongly recommended, HackerOne can advocate for a decision that we know leads to a win-win result.

HackerOne runs the largest vulnerability disclosure and bug bounty platform in the cybersecurity industry. We operate at considerable scale, having dealt with millions of reports and hundreds of thousands of impactful vulnerabilities. Given this, new edge cases are inevitable, but we train our Customer Success, Community, Mediation, and the office of the Chief Hacking Officer to be ready to handle them. As the Chief Hacking Officer, I get personal satisfaction from assisting in resolving individual cases – after all, this often means a hardworking hacker is being accurately rewarded for their efforts. However, the most important part of the job is to make sure we learn from the patterns in the individual cases. The only way to scale improvements is to make changes to policies, processes, and documentation that are usable across all programs.

Hackers and customers alike may be interested in some specific cases that the Office of the Chief Hacking Officer has ruled on over the past year. Most of these cases represent great outcomes – which is probably why you won’t have heard of these. After all, it’s more common for people to write about experiences when they’re unhappy. So we want to bring these positive examples to the light of day.

Case 1: Not Having Fully Inclusive Bounty Tables Means A Critical Vulnerability Could Go Unreported

A hacker reported a vulnerability in a domain that was not part of a bounty table – such as docs.customer.com. The vulnerability, however, was clearly leaking internal database credentials for the core product. The customer was unsure whether to reward or not because the domain exhibiting the problem was not in a bounty table.

Resolution:

HackerOne Customer Success worked with the customer to explain that, while bounty tables are a useful starting point, they sometimes are not sufficient for nuanced situations. HackerOne advocates for the industry best practice of “pay for value” when it comes to determining rewards. This concept immediately resonated with the customer, who realized that since the vulnerability had an impact relating to the core product, they should reward based on the impact, and that the initial domain was a red herring. The customer promptly rewarded according to their main www.customer.com bounty table, to the satisfaction of all involved.

Note:

Sometimes you will hear the phrase “out of scope” to describe domains that are not part of a bounty table. Customers should be extremely wary of declaring anything out of scope. It’s often a dangerous idea. Remember, everything is in scope to a cybercriminal. To lower your chances of a breach, we need to let the ethical hackers go toe to toe with the criminals, and report anything that has a security impact.

Case 2: Failure To Reward For Third-Party Issues May Expose You To A Breach

A hacker reported a serious vulnerability where a retail giant was at risk due to a critical vulnerability in a third-party database component. The third party had released a bulletin and patch a couple of days previously. In a happy turn of events, the customer reached out for HackerOne expert advice before making a determination. This is the best kind of mediation: one that is solved before it exists because of positive, proactive behavior!

Resolution: 

HackerOne Customer Success again introduced the “pay for value” mindset. When analyzing the value inherent in the report, the customer realized that their standard tracking process for applying third party patches would take weeks. Given that the vulnerability was the sort of critical issue that could lead to a breach, the customer patched within a day (instead of weeks), which was an outcome only made possible by the hacker’s report. The report was rewarded $3,000.

Note:

There’s a lot of variance in how different customers handle reports for third party components. Customers running a top-tier security program and adhering to industry best practices will always reward in cases where a report causes them to take any action, or accelerate any action.

Case 3: HackerOne Mis-characterized a Certain Type of Denial-of-Service Report

A member of HackerOne’s Community team escalated a case for Chief Hacking Officer consideration. A hacker had reported a Denial-of-Service class vulnerability and received a warning for “unsafe testing.”

Resolution:

The HackerOne teams (Community, Triage, Chief Hacking Officer) collaborated to understand the full technicalities of the report. We found that it is possible to safely test the specific vulnerability type (a type of cache poisoning), and the hacker had indeed tested safely. We performed a suite of reparations, including the removal of the “unsafe testing” note, and the correction of the bug’s status to correct the hacker’s reputation. We assessed the report against the customer’s configured bounty table at the time of the report. We awarded $1,500 from HackerOne’s Make It Right fund. I also personally reached out to the hacker to apologize. Learning from mistakes is important, so we improved our triage documentation and runbooks to make sure this vulnerability type is not miscategorized in the future.

Case 4: Utilize Industry Standards To Make Final Decisions On Severity

A hacker raised multiple mediation requests relating to a program that was aggressively downgrading the severity and bounty amounts on multiple reports.

Resolution:

Upon investigation, we did find a pattern of the customer deviating from industry standards around severity assignments. When this happens, our first port of call is to engage with customers to understand the situation and engage in education as appropriate. Education takes time, but if we can effect change in customer behavior, it’s better for everyone. Future hackers will be rewarded correctly, and the customer will gain the benefits of stronger hacker engagement, therefore lowering their chances of experiencing a breach and avoiding getting a reputation for dubious security. After an extended period of engagement and education, the customer corrected the reports and the hacker was rewarded in the order of $10,000 extra in bounties.

Case 5: Transparency and Integrity Builds Trust With Hackers

A hacker raised a general mediation for a series of reports getting downgraded, and / or closed with surprising statuses. Upon investigation, part of the issue was that the customer was surprised that a low-priority asset was in fact mapped as a subdomain of their main scope and bounty table.

Resolution:

The customer immediately changed their main scope to exclude the subdomain. However, it’s important to note that such changes cannot be made retroactively. To protect the platform's integrity, customers must honor any commitments made in their bounty tables. Everyone agreed that aside from being required, it is also common decency. However, disagreements still arose regarding the severity and duplicate status of some of the reports. Normally, such disagreements would be resolved with abundant transparency and detailed technical reasoning. Unfortunately, the discussions were inconclusive. In order to avoid the hacker being out of pocket, HackerOne awarded over $15,000 from our Make It Right fund. While it would be easy to get excited about such a resolution, we always see the use of Make It Right as a failure condition for all involved. We always recommend that customers reward generously and magnanimously, as this leads to better customer outcomes, more engagement, more security, a better reputation, and happier hackers.

Case 6: Accept Industry Standards, such as Coordinated Vulnerability Disclosure

In a few instances, hackers have reported vulnerabilities to public programs where the initial vulnerability submission is accompanied by an industry-standard coordinated vulnerability disclosure (CVD) requirement. Questions have arisen as to whether this is a Code of Conduction violation.

Resolution:

Reporting vulnerabilities where the initial vulnerability submission is accompanied by an industry-standard coordinated vulnerability disclosure (CVD) requirement is not a Code of Conduct violation on a public program. Exercising industry-standard CVD on a public program is reasonable. In the event that this industry standard runs counter to a customer policy, it is also reasonable for a customer to decline to reward a bounty, but unreasonable to engage in attempts at recourse with a good faith hacker. That said – we encourage customers to focus rewards solely on the impact of the information in the report. One of HackerOne’s responsibilities is to ensure that our platform avoids absurd outcomes. In one instance, a hacker observed a customer policy that contained overly onerous and aggressive language. Accordingly, the hacker emailed the customer instead, to avoid being tangled in any policy or terms and conditions. However, the security email address was not properly monitored, leading to a languishing report, and thereby increasing the customer’s risk of compromise. We aim to reduce lengthy delays by writing good policies and supporting reports that decline custom policies in favor of well-established industry standards such as Coordinated Vulnerability Disclosure.

In closing 

So what can hackers and customers expect next? We will continue to learn from every unusual mediation case and fold what we learn into policies and externally facing documentation such as our Best Practices page. For hackers, please continue to impart your trust in our mediation process and team as we will give every situation an unbiased and detailed review, with our primary goal to always be a mutually beneficial outcome. Continuing to use this process will help us to continue to track themes and expand our learnings. 

As covered above, we split best practices into either baseline requirements, or strong recommendations to exhibit top-tier maturity. Hackers do prefer mature programs, so we’re working on ways for programs to robustly signal to hackers (and customers, regulators, and insurance providers!) that they commit to mature practices. For example, see our Program Levels initiative.

Overall, HackerOne’s culture for running our platform is one of continuous improvement. We’re grateful for all of the feedback we receive from customers and hackers. Thanks to everyone who has collectively helped improve our processes as we drive towards a safer internet. We’ve enjoyed documenting some of the things we’ve learned, and how we’re applying that knowledge to scaleable improvements going forward.

Chris Evans

Chief Hacking Officer and CISO, HackerOne