AISA 2017 - Bug Bounties: Traitorous cooperation with an enemy

During AISA's 2017 annual conference, Atlassian's Head of Security Daniel Grzelak and his colleague, System Engineer Matthew Hart, looked into the question of ethical hackers and using bug bounties to find the gaps in security posture.

Every non-security stakeholder has asked "Should we be allowing hackers to hack us?” said Grzelak. And while he says the answer is yes, we need to frame the question differently. He believes we should use incentives to categorise the types of hackers out there.

Some, he says are bad actors who will hack you no matter what. Then, there are neutral actors who can be incentivised to act in the third group; the good actors.

Hart says that traitorous cooperation is treacherous. He outlined Atlassian's approach.

Rather than talking about hackers, Hart talks about "researchers". He has worked closely with researchers in educating them into how they should conduct research in a way that doesn't impact customers. In some cases, as the researchers actually breached customers, they weren't paid as Atlassian did not want to incentivise the behaviour. Instead, the offered to provide the bounty hunter with an environment they could test their attack without harming customers.

Hart says some of the best value he gets from researchers comes from well documented reports. When great reports come in, he pays more than the promised bounty.

"Rewarding this hard work is important, otherwise that researcher might not put in the hard work again later," he said.

When dealing with researchers, he advocates a consistent, "no bullshit" approach. Getting on the front foot with communication, such as when a bounty value is made higher or lower is important.

He also says it is important to see the researchers as part of the team.

However, things don't always go smoothly. It's important to remain focused and not distracted when multiple reports come in. The researcher community is quite small so the experiences of one can be shared quickly.

Grzelak says trying to estimate who the actual enemy is hard. We have limited information and many of the approaches we currently use are flawed.

For example, if you use two or three pen testers for two or three weeks, you are time-boxing the work and using people with a limited set of skills and toolkit. In contrast, threat actors are more numerous and have mire time.

This is where a bug bounty program works. The program runs for as long as you are paying bounties and appeals to thousands of bounty hunters. And, hackers only attack when they are incentivised. Instead of being paid to attack you, they are paid to tell you how you could be attacked.

The researchers out there, said Grzelak, fall into four groups. Some focus on specific bug classes across all bounties. Others, which are the ones you really want said Grzelak, focus on vertical attacks. They are highly focused on specific platforms or applications.

As researchers become more specialised, they develop their own automation and tooling, to focus on specific bug types they are most interested in. And they develop specific skills as a result of the competition they are facing in order to be first to find a vulnerability, so they receive the maximum return for their effort.

The goal, said Grzelak, is to learn about your systems so you can increase the cost of finding and exploiting a vulnerability. He said the bounties paid are a reflection of the effort required to execute an attack.

Hart says all this information can then be used to create better application security.

In the early days, Hart says they were too open in receiving vulnerability reports. He said 85% of the reports received were invalid. This lead to Atlassian putting together a checklist that researchers could use to validate their own reports.

Also, initially, reports were rewarded with a t-shirt. But by changing this to a financial investment, and using the checklist, the number of invalid reports fell to 25%.

In a 40-hour work week, this meant they moved from spending six hours on valid reports to investing 30 hours on valid reports.

He also said increasing the incentives, using a set of metrics to validate the reports, allows researchers to be better paid as they deliver more value.

The goal, said Hart, is to raise the bar. The bug bounty program has provided a metric to determine whether their systems are becoming more secure.