Communications of the ACM

Communications of the ACM

Necessary Measures: Metric-Driven Information Security Risk Assessment and Decision Making

By Wade H. Baker, Loren Paul Rees, Peter S. Tippett
Communications of the ACM, October 2007, Vol. 50 No. 10, Pages 101-106
10.1145/1290958.1290969
Comments

Managers increasingly aim to leverage the full potential of IT while protecting corporate assets made more vulnerable because of it. Following this trend, security incidents have resulted in significant financial losses for many organizations—even as they invest in security technologies and their staff and policy documents, practices, training, and certifications pile up. Moreover, evidence exists that there is often no correlation between increased spending on such initiatives and a better overall security record [1]. Our personal experience with business leaders around the world suggests that many are frustrated by their inability to identify the most effective strategies for limiting organizational loss. Though there is no shortage of security standards and research, these leaders often admit they have no reliable methodology for measuring the effectiveness of their security initiatives or collecting the data needed to make strategic decisions and determine the financial value of their efforts. Fittingly, the U.S. Department of Homeland Security in 2004 named a lack of real-world data on risk factors as one of the most pressing information security research problems [10].

Although managers frequently cite confusion about how to measure the effectiveness of their security efforts, such confusion is also the result of confusion about what to measure—an issue of equivocality. In organizational theory, equivocality stems from the existence of ambiguity and conflicting interpretations, whereas uncertainty is the result of a lack of information [4]. In terms of organizational security programs, equivocality results from ambiguity about what to measure, while uncertainty results from managers being unsure how to measure it. This combination results in a debilitating lack of information about the factors driving information security risk. In order to generate data and lower uncertainty for improved decision making, equivocality must be reduced by defining, expanding, and clarifying risk factors, so metrics, the "necessary measures" of management, can be applied unambiguously.

Addressing equivocality, we start with the premise that an organization's goal is to minimize loss expectancy, or risk, by maximizing the efficiency of its mitigation efforts. There is substantial research on this topic, including systems risk [8], economic models [5], game theory [3], and value at risk [7]. Though varied in form and terminology, models sharing this purpose generally define risk as the product of three main factors: frequency of threats/attacks; the likelihood of their success; and their impact on the organization. We have used such models while implementing information risk management programs and attest to their effectiveness. However, we have also found that although this approach provides an adequate superstructure for decision making, a high degree of confusion remains across, and even within, organizations about how threat, vulnerability, and result are measured and modeled.

Current approaches recommend and employ estimates for such critical factors as attack frequency and countermeasure effectiveness, because no clear and accepted framework exists for what real-world measurements are needed to drive risk models or how they are to be collected. With the goal of minimizing equivocality in risk-based decision models, we aim to formulate a system that does three main things: allow accurate measurement and tracking of threats; enable determination of the impact and loss of successful threats; and aid in evaluating the effectiveness and return on investment of countermeasures. With reduced equivocality, uncertainty can then be addressed and risk models made useable, as we illustrate here through a managerial decision scenario. We begin by introducing a system for threat classification that represents a foundation for risk modeling and the collection of reliable metrics.

Back to Top

Threat Classification

To be able to take systematic action to reduce risk [8], management must identify and understand the threats facing the organization [11]. Numerous taxonomies using a variety of techniques have been proposed to classify and systematize threats to information security [6]. However, they tend to be geared toward security professionals or to such purposes as trending and surveys and are not suitable for managerial decision making. In order to maximize its value for decision making, a threat taxonomy must focus on two main purposes: use metrics to quantify risk factors and identify efficient risk-reduction strategies.

We find the Computer Security Incident Information Taxonomy in [6] to be sound and agree that threat taxonomies must be mutually exclusive, exhaustive, unambiguous, repeatable, acceptable, and useful. However, to optimally satisfy the two purposes identified in the last paragraph, we extend their overall taxonomy requirements in the following way: In our system, all possible threats would be categorized into a manageable number of threat scenarios in which individual threats are grouped together if they share six main characteristics: source; channel or mode of attack; target; frequency of occurrence; severity; and set of countermeasures with similar mitigation value. Several previous taxonomies included viruses and hacking but made no allowance for the fact that each comes in multiple varieties, thus requiring that different metrics be utilized in decision models. For instance, malicious code (malcode) can infect an organization through email, Usenets, P2P applications, and Web surfing. Each enters the organization through a different channel at highly varied frequencies, produces diverse effects on the organization, and requires different mitigation countermeasures. These threats clearly violate our six requirements and cannot be classified as a threat scenario, even though other taxonomies have conflated them into a single threat.

Additionally, and most important, threat scenarios are described within the context of our system as a chain of events, from conception to resultant financial losses to the organization. In so doing, we emphasize that security threats are not simple and one-dimensional (like viruses) but complex, often combining data, devices, and people to achieve success. Failing to identify, measure, and incorporate them greatly reduces management's ability to quantify risk factors for decision models. Constructing the entire event chain for threat scenarios also allows threat variations (such as those for malcode) to be represented accurately. Figure 1 models common malcode variations according to our taxonomy, and Table 1 outlines descriptions of events and results. Though we use malcode to highlight the characteristics of our system, all information-security threats, from hacking to employee theft, can be modeled as well. Classifying threat scenarios in this manner is not just an exercise in theory but has practical implications for data-collection and decision models. Because every link represents a distinct, therefore measurable, stage of progression, the application of metrics to threat scenarios becomes a much less equivocal endeavor.

---

Security initiatives are more commonly driven by compliance mandates than by the principles of risk management.

---

Back to Top

Linking Threats to Financial Loss

Quantifying the frequency and impact of threats, as well as determining the mitigation power of various security countermeasures, is essential in building decision models for reducing risk. However, this process is greatly misapplied in many security-management programs, and confusion exists concerning what it is about these factors that should be measured and quantified. Organizations can accurately accomplish this through consistently obtained and rationally applied metrics. However, additional distinctions must be defined beyond those described earlier for threats if one expects to remove equivocality from security-related decisions.

Threats alone do not directly cause financial loss. Rather, it is more precise and helpful to recognize that successful threats cause any number of undesirable business-level effects. For example, in Figure 1, the event chain TSE1b-2a-3a causes the organization to become infected, as well as several other adverse results, including those in TSR1a, 1b, and 1c. Within the organization, each of these outcomes has a distinct effect that must be assessed and quantified separately.

Many who attempt to quantify the impact from various risk models mistakenly ask: How much does a virus incident cost? when it would be more appropriate to ask: What adverse impact does a virus incident impose on the organization and its assets, and how much does it cost? The primary task is to identify and model all possible outcomes caused by a threat scenario and later to assess the associated financial losses.

Care must also be exercised when translating threats to impacts and then to costs to keep units consistent. Threats must be measured as, say, rates per year and then into outcomes by specifying the number or extent per threat. For example, we conducted a study in 2004 of 1,500 organizations to quantify specific outcomes resulting from the MyDoom worm of January 2004 and found they spent an average of 44 person-hours on clean-up and recovery following infection. Another study [2] found that compromised firms lost an average 2.1% of their market value in the days surrounding an incident. When the likely results of an incident are thus factored into the equation, an organization can more accurately calculate its exposure, or expected loss for a threat scenario, which is essential for prioritizing mitigation efforts.

Besides enabling the quantification of threat frequency and impacts, modeling threat scenario chains has powerful implications for evaluating countermeasures intended to reduce organizational risk. A threat scenario must succeed at each stage of the event chain. If the chain is broken, the threat is effectively mitigated. Management's goal then becomes to identify and evaluate the many possible countermeasures it could deploy to impede threat scenarios along the event chain. A countermeasure matrix proposed in [8] expressed the effectiveness of a countermeasure to mitigate threats as a fraction between zero and one (inclusive); this value is typically an estimate, the accuracy of which is often diminished due to poorly defined threats and other complications involving implementation quality. We adopt this concept to countermeasures applied at any stage of a threat scenario in order to more accurately calculate potential risk reduction.

The precision we insist on significantly eliminates equivocality. Because each link in our threat scenario is a distinct channel and target, it is far less ambiguous to evaluate the effectiveness of countermeasures independently at specific links rather than for broad categories like viruses. For example, we could test the probability that antivirus software will stop malcode entering TSE2a in Figure 1 rather than estimate this value for the threat as a whole. (Antivirus software is typically tested at 100% effectiveness against recognized signatures but is ineffective against new variations appearing since the last update; the implication is that the ratio of "new" to "old" malcode and antivirus update frequency are important metrics.) Training users about the risk of opening strange email attachments could interrupt the threat scenario if the scenario's success depends on segment TSE2a, but this countermeasure would be ineffective mitigating variations TSE2b and TSE2c. The synergistic value of "defense-in-depth" approaches to security management can be visualized and modeled in this way.

Threat scenarios may be broken along the event chain both before and after they affect the organization. If the former, deterrent and protective countermeasures help minimize the probability of a threat's occurrence or success. When the extent of the resulting effects is decreased, and thus incident costs lowered, countermeasures assume a detection-and- recovery role. Figure 2 outlines the TSE1b-2a-3a->TSR1b-1c chain of threats from Figure 1, applying four deter-and-protect countermeasures and two detect-and-recover countermeasures.

Back to Top

Implications for Risk Assessment and Decision Making

We now expand the example of email-based malcode in Figure 1 for two purposes:

• Demonstrate how measures of frequency, cost, and countermeasure effectiveness are derived through metrics to lower an organization's uncertainty while building actionable intelligence; and
• Help make clear what should be measured at each step, since event chains for each threat scenario are modeled from conception through financial loss.

In 2004, we monitored traffic from 40 organizations, collecting metrics necessary to demonstrate the risk-calculation process for the threat scenario in the figures. Averages for relevant metrics (and their sources) collected in the study are listed in Tables 2 and 3.

Note that by listing the complete chain of events (in Figure 2), our system makes it possible to derive an initial threat frequency, then revise it through the use of metrics at each successive intersection of the event chain. The bottom portion of Figure 2 demonstrates how organizational risk for the threat scenario is calculated using the data in Tables 2 and 3. The first calculation depicts the loss expectancy of an organization having no security measures, though the calculation is absurd, as such an organization could not maintain its operations while connected to the Internet. However, the calculation is pedagogically valuable, as it shows the proper translation of threat rates into financial losses. Since all organizations have at least some security controls, we provide a more realistic example in the second calculation in Figure 2 where six common countermeasures are applied, along with their effectiveness, at each stage of the threat scenario. Using the data from Tables 2 and 3, the example shows the organization can reduce its risk from the threat scenario to approximately $44,000 annually.

What should an organization wishing to mitigate its IT security risk do? Collect four categories of data: expected threat frequency; countermeasure-effectiveness data; threat-to-impact transitions; and data concerning the financial loss from the impacts. During the risk-assessment process, the organization has many options relating to metric sources. For example, Tables 2 and 3 include metrics collected from internal logs, public sources and databases, and internal/expert analysis. Deciding which ones to use depends on availability, internal resources, and the maturity of the risk-assessment program. (For a more detailed discussion of security metrics, see [9].) Organizations generally find that the further to the right in the figures they collect metrics, the more important it is that they develop organization-specific numbers rather than rely on national averages. For example, costs to the organization for loss of productivity vary widely, as opposed to the effectiveness of, say, antivirus software as a countermeasure or the threat rate of malcode (both further left in the figures).

---

Current risk models, and the decisions being made through them, are only as good as the data on which they are based.

---

The ability to calculate risk through real-world metrics has far-reaching implications for managerial decision making. A promising extension is in evaluating the potential return on investment of proposed security initiatives. To illustrate, data from the study we described earlier revealed a nearly threefold increase in malcode activity during the first half of 2004. Noting this increase, a manager might decide to patch systems monthly instead of quarterly in order to reduce the system vulnerabilities commonly exploited by malcode. This reasoning is flawed for several reasons, all of which would be clear if the proper metrics were available:

• We estimate that testing and deploying patches in a 1,000-system organization can easily reach $73,000/push, or approximately $584,000 annually, for the proposed eight additional patch deployments. It would be impossible for the organization to regain its investment regardless of the level of benefit realized from additional patching;
• Our metrics showed that only a third of email-based malcode exploited vulnerabilities, suggesting that patching would be ineffective anyway; and
• Malcode that could exploit vulnerabilities targeted a vulnerability for which a patch had been released nearly three years prior; in other words, the quarterly patching cycle would have been more than sufficient.

Increased patching is often an inefficient mitigation strategy against malcode; the example shows how organizations with inadequate data are prone to wasting security dollars.

Our research also produces a second pragmatic implication. For the past seven years, we have used these methods to collect thousands of metrics for global threats, vulnerabilities, and impact to drive Verizon Business's Quantitative Risk Model (QRM). We've used QRM to analyze more than $1 billion of security spending across major organizations worldwide. In each case, we typically find that roughly a third of information security spending does not reduce risk in any meaningful way. This failure stems from the fact that security initiatives are more commonly driven by compliance mandates than by the principles of risk management. While compliance and risk-driven decisions should complement one another, they often do not. Organizations lacking a systematic, metric-driven approach to security spending quickly find themselves in the audit crosshairs without an adequate defense.

Back to Top

Conclusion

The old axiom "you can't manage what you can't measure" describes how business leaders confront the security of their organizations. Current risk models, and the decisions being made through them, are only as good as the data on which they are based. We therefore call for a more reasoned approach to measuring and modeling security factors that include:

• Carefully delineating threats;
• Collecting threat-to-asset impact data;
• Gathering impact and financial loss data; and
• Calculating reduced vulnerability through countermeasures.

If these approaches are not pursued, managerial decision making will continue to be plagued by uncertainty, resulting in wasted security investment, loss of boardroom credibility, poor security configurations, and costly regulatory penalties. If, however, the "necessary measures" are taken, improved analysis and decision making should be expected to follow.

Back to Top

Back to Top

Back to Top

Back to Top

Back to top

The Digital Library is published by the Association for Computing Machinery. Copyright © 2007 ACM, Inc.

---

No entries found