Checking Under the Bed

A lot of businesses lean on managed security service providers (MSSPs) for help with various controls. There are a lot of reasons for that, some better than others, but ultimately the question arises, are you really getting what you pay for? Particularly when it comes to monitoring and alerting of potential issues, is your MSSP going above and beyond, or are they just throwing stuff under the bed? I’m fairly confident that you’re paying them more than I pay my little bedroom-cleaning service providers, so let’s consider how to make sure they’re doing the job.

There are a wide range of security services out there, and I don’t think it’s necessary to make a list here, but for the sake of this article I’m going to speak to services focused on monitoring/alerting. This includes services to monitor your firewalls, IDS/IPS, endpoints, security logs, etc. But in general, the principles outline here can be applied to any MSSP. And even to your own internal resources if you want to gauge their effectiveness.

Know What to Expect.
The first step in figuring out how your MSSP is performing is to make sure you understand exactly what they’re supposed to be doing. Pull that contract out and read through the finer points to ensure that your expectations are realistic. Here are a few key things to consider:

• What services are they providing? What exactly are they supposed to be monitoring? Are they responsible for configuration or does that depend on your internal resources?
• How are alerts being generated? In other words, what exactly are they monitoring? Make sure that you know exactly what logs they should be receiving and from what devices.
• What is their expected response time? Is there an SLA that outlines how quickly they’ll respond for certain types of events?
• How are notifications sent and who do they go to?
• Is there an escalation process if you don’t acknowledge alerts within a certain time frame?

Before you move on to the next step, make sure you completely understand what you’re paying for and what you should be expecting.

Examples of Testing
So the short version here is, whatever you expect your MSSP to alert you on, make that thing happen and then see if you get an alert. Do you want to know if someone port scans your internal network, run a port scan and see how long it takes to get notified? If you don’t get an alert, then you have the opportunity to follow up and find out why not.

What exactly you test is going to depend greatly on the above questions. But here are some ideas to get you started

• Network
  • Port scans, vulnerability scans, etc
  • Password spraying attacks and login attempts from tools such as CrackMapExec.
  • Man-in-the-middle attacks (Gratuitous ARP packets)
  • Data exfiltration (Hashes, SSNs, etc)
  • Command & control tunnels (Metasploit, PowerShell Empire)
  • Potentially malicious traffic (Tor, Spam, Torrents)

• Endpoint
  • Adding users to privileged groups
  • Disabling AV
  • Modifying configuration files
  • Unusual use of system commands (PowerShell, net, wmic, etc)

• Logs
  • All of the above
  • Logs no longer being received
  • Tampering with dates/times
  • Denial of service (lots more logs generated than normal)

For each of those issues, simply generate the event and see what happens. Be sure to keep track of exactly what you did, and when you did it.

Testing Considerations
Here are a few things to consider as you begin the testing process. Remember, our goal here is to improve your controls, so spend some time planning before you jump in.

Be Creative. Think like an attacker and try some attacks that aren’t so common. For example, I’ve personally found that many SIEM tools by default alert when a user is added to the Domain Admins group from the GUI, but not when it’s added using the net command.

Be Realistic. Don’t go overboard. You don’t need to chastise your MSSP because they didn’t catch a port scan that you slowed down to one packet per minute. It’s easy to get excited, but try to keep it practical.

Be Careful. In some cases, you may be using actual attack tools, or even actual credentials. Think through those situations and consider the ramifications. You don’t want to introduce new vulnerabilities to the network in the process.

Be Smart. Do Not Test Without Permission. No one looks good in an orange jumpsuit (almost no one). If you’re not at the top of the org chart, be sure and get permission before you start this process.

Be Gracious. While your goal is to improve security, you may end up catching someone doing a bad job. Keep in mind that they may just be having a bad week. And even if they are completely incompetent, you can still be respectful. Your testing may just be the impetus they need to improve their processes and become a more valuable business partner.

When to Test
Once you’ve outlined your simulated attacks, consider the optimal time for testing. This will vary greatly by organization, security maturity, and the goals of the testing. For more critical systems, you may want to schedule these simulations during specific off-hours, especially at first. But as you refine the process, you may generate the events during random times to more thoroughly test the MSSP’s availability.

If your goal is to see what the MSSP is capable of testing, you can create a list of events and generate them all at once. Then compare notes with what they alert you on, or follow up with them directly. But if you want a more real-world simulation to see how consistent they are, consider spreading out the tests. For example, you could generate several events per week at random times throughout the week.

When you’re just starting out, I recommend starting slow and building from there, especially if you don’t have the bandwidth to devote a lot of time. Make a list of 3 things you expect your MSSP to catch, and then spend 15 minutes thinking through how to generate those 3 things. The next week, add 3 more to the list.

Third Party Testing
During penetration test we often find customers who are using an MSSP for various monitoring, but have no idea how effective they are. During the test we find that some are better than others. Some generate a large number of alerts throughout the testing period, while others generate only a few alerts, or none at all. While this can be a helpful gauge, it’s not effective as intentional, methodical simulations as I’ve outlined.

If you need help generating simulations, or have questions about how to better assess your security vendors, let me know at nathan@secureideas.com. This article is a summary of a talk I gave recently at ShowMeCon. The talk contains more detail and specific examples and is available at https://www.youtube.com/watch?v=_SF4vw_mVnY.