How to Measure the Effectiveness of Information Security Controls

If you’re in charge of any aspect of information security for your organization, you operate in an area of much risk and great uncertainty. You’re facing a lot of questions, particularly about your controls, such as:

• How do I know my controls are effective?
• If my controls are too restrictive, how much should I dial them back?
• If they aren’t working well enough, how far off are they?
• If I want to make improvements, how much budget will I need?

To manage the effectiveness of information security controls, we need a way to measure them. Measurement will give us peace of mind if our controls are doing what they’re supposed to do. And, if necessary, measurement will help us prioritize and take corrective action.

Ideally, we want to avoid imprecise measures like high, medium and low. At the other extreme, most of us never learned quantitative methods (let alone how to explain probability theory to executives).

So, is that all we need? Nope!

We also need to measure controls within the context of the true nature of security, which, like most things in life, is a careful balancing act. That is: We need a way to measure when we have too little security, too much security, and just the right amount. You can see this visually in Figure 1.

Figure 1

As we go from left to right along the x-axis, we’re spending more and more resources trying to reduce risk. Notice that risk goes down rather quickly as we begin to manage it.

As you move to the right and enter the green zone, the curve goes lower and risk levels drop to an acceptable level. However, as you continue to spend money and add more controls, the risk increases again as you move further to the right and out of the green zone.

Why is that?

Well, past a certain point, security gets to be so difficult that people begin to look for ways to go around the controls, which can create a false sense of security for you, the person responsible for managing risk. In other words, you may be using more resources than are required and getting a risk level that’s much worse than you need.

I’m sure you’ve experienced a situation where there was too much security. I’ve seen remote access systems that were so secure it required four separate, two-factor authentications before I could do real work! It was so complicated and time-consuming, most people didn’t use it, which reduced that organization’s productivity. And it caused them to spend a lot of money on a system that was operating far under capacity.

As I said before, the challenge with security, as with most things in life, is to find a good balance; in this case, between good protection and ease of use. Using this curve, let’s create a score key that captures these three security states and encourage us to find balance. See Figure 2.

(Figure 2)

Let’s explore the numbers, starting on the left:

• The scores zero through four, colored in yellow, represent various levels of insecurity. From no security at all, to some security.
• The scores from five through eight, colored in green, represent a range from minimally acceptable security to fully optimized.
• And scores nine and ten represent too much security, which is wasteful of time, money, and morale, just like the remote access solution I described above.

You’re probably wondering how to choose a number. Since every control should produce a desired outcome, that’s a great way to measure its effectiveness. See Figure 3 for some definitions.

(Figure 3)

Here’s a quick example: If your ability to detect a data breach “rarely or never happens when needed” then your score for that control is a zero. Once you know that, you can set a target score and drive for improvement. You could choose a 5 (“happens consistently with minor flaws”) or even an 8 (“happens consistently with high quality”). You might set a goal of being a “5” next year, and an “8” the year after.

Now that you have a defined gap, you can do some analysis to figure out what it will take to close the gap. Your analysis will form the basis for a spending proposal to your management. To increase your chances for approval, be sure to see my previous blog post on how to present your proposal in terms of the business value it will create.

Kip Boyle has spent 20 years working in information security and is the founder and CEO of Cyber Risk Opportunities. To learn more from Kip on information security, view his course Implementing an Information Security Program.