What's the Difference between Penetration Tests and Bug Bounties?

Every day you hear about another breach or a vulnerability you should worry about. At the same time, you are told from different people how you should test. Some people say to stick with the traditional penetration testing and others are encouraging the latest hotness: bug bounties. So which is better and what should you pick?

Defining the context:

To begin, we need to make sure that we are talking about the same thing and understand what these terms mean. To begin, let’s look at how NIST defines penetration testing in 800-53:

“Penetration testing is a specialized type of assessment conducted on information systems or individual system components to identify vulnerabilities that could be exploited by adversaries. Such testing can be used to either validate vulnerabilities or determine the degree of resistance organizational information systems have to adversaries within a set of specified constraints (e.g., time, resources, and/or skills). Penetration testing attempts to duplicate the actions of adversaries in carrying out hostile cyber attacks against organizations and provides a more in-depth analysis of security-related weaknesses/deficiencies.”

That is a mouth full, so let's try and simplify it. A penetration test is simply a test evaluating how an attacker would view and exploit your organization. The purpose is totally dependant on your goals, but they tend to be a way to prove out and prioritize security issues. You work with the penetration tester to create a scope and focus for the testing.

There is a second part to the definition of a penetration test, and that is who does it. You can use either a third-party (such as Secure Ideas) or you can leverage your internal staff. Either way, you have to ensure that the people performing the pentest have the skills necessary to effectively test the target. These skills, as outlined in other articles on this site, aren’t just elite hacking capabilities. The testers need to be able to both handle the security assessment and exploitation needs but also understand the target; be it a specific type of organization or type of IT infrastructure or application.

Bug bounties are different yet the same. According to the Oxford Dictionary, a bug bounty is “A reward offered to a person who identifies an error or vulnerability in a computer program or system.” This leaves out the program or process to run or process the submitted bugs, which is often part of the definition. There are three main types of bug bounties seen today:

• Internal
• Self Hosted
• Third-Party

Many companies start with the internal bug bounty. This is where they build out a reward system for employees who find and potentially fix bugs. If you set up something like this, quite often you will run a hack-a-thon or such to kick it off.

After running the internal program for a while, you may move to an external-facing bug bounty program. If so, you can either run it yourself or work with a third-party such as BugCrowd or HackerOne.

So let’s discuss the good, the bad, and the ugly of each of these options. And keep in mind that doing both is an option.

Bug Bounties

For many organizations, an internal bug bounty program is a good idea. Promoting bug reporting and awareness is really no different than reporting phishing or other issues. You just need to do a few things to ensure that you are ready for it and efficiently handle the items being reported.

First, you need to determine who should manage the reports. This is often the development or IT staff responsible for issues already. They already deal with problems and bugs, so it makes sense to include them as the responsible party for this. If you don’t, then you will be increasing the steps in the process, which is the opposite of efficient.

After determining the who, you need to look at the how. You probably have a bug reporting or ticketing system in place already. For example, most organizations looking into bug bounty programs have a help desk ticketing or a development bug reporting system. Look into if this system and process works for your bug bounty. Is it as simple as using the current help desk as an entry point for the bug reports? Can the bug tracker work as the place to report bounties? In most cases, the answer is yes.

After working on the how you need to then look at the what. What is the reward for reporting a bug to the bounty program? Basically, you need a bounty! This option is entirely up to you. It can range from time off or company swag all the way to cold hard cash.

After working internally, your organization can then look at moving this to an external-facing bug bounty. The only choice there is to determine if you want to run it yourself or use a third-party service. Mainly the question is do you have the resources to run it or do you save money by paying a third-party.

So why is it worth doing? Mostly as a way to continually evaluate your security. It is the theory of many hands make light work. If we have multiple people of various skills and experience looking for issues, there is a good possibility they will find the issues that exist. So instead of the traditional penetration testing model of a point in time check, your organization is almost constantly looking for flaws.

So why would it not be a good idea? It depends on where you are in your security process and what your goals are for the program.

One of the main reasons to not run a bug bounty program is that your organization is not mature enough in its security growth to use the results. For example, we often test organizations that don’t have a vulnerability scanning process or application security testing during development. These organizations aren’t ready to have the world start testing their stuff. A way to determine if you are mature enough is to see how well you handle the findings you already have. Do you remediate things well? Do you have a documented process for security exceptions? If so, then you may be mature enough.

Another negative in running a bug bounty program is when people believe it replaces a compliance requirement. While it may, often auditors or partners won’t accept general bug bounty programs as a replacement for penetration testing. Talk to your auditors and partners before doing this, or (as we mention above) do both.

Penetration Testing

So if bug bounties are so great why would we get a penetration test instead? Like with all the other questions here, it depends. There are a couple of ways that penetration tests are better for most organizations.

One main reason a penetration test would be better is if your organization is not mature enough yet for a bug bounty. As discussed above, if you are having trouble fixing the issues you know about, then having the world start picking at your infrastructure and applications. Working with your penetration testing team should help you build out a remediation process.

Another reason, as implied before, is that your compliance requirements mandate a penetration test. While the main bug bounty programs have options that attempt to fulfill the penetration testing requirements, not all auditors accept them yet. So having a pentest may be required.

So do you need to hire some team to come hack you? Actually, in many cases, no. If your internal staff can show that they have the skills required, they can perform the testing. We would recommend talking to your auditors and partners if they are why you are doing the testing, but this can be an inexpensive way to fulfill those requirements.

So bug bounties and penetration tests can both be a foundational part of your security program. And no matter what you do, just ensure that you are supporting the people and efforts to evaluate your organization!