Towards a Comparable Cross-Sector Risk Analysis: RAMCAP Revisited

The search for a uniform risk analysis approach for critical infrastructures has prompted a reexamination of the Risk Analysis and Management for Critical Asset Protection (RAMCAP) methodology to see if it can accommodate emerging threats from climate change, aging infrastructure and cyber attacks. This chapter examines the challenges involved in taking a site-speciﬁc formulation and turning it into a general model capable of analyzing performance under a full range of simulated conditions. The AWWA J100-10 standard provides the blueprint for a basic RAMCAP model that calculates risk as an attenuation of consequences via probability estimates of vulnerability, threat, resilience and countermeasures. The RAMCAP model was subjected to varying scenario loads in deterministic simulations that examined all hypothetical conditions and probabilistic simulations that examined likely conditions. RAMCAP performance was measured by the average net beneﬁt and represented by the distribution of component values. Contrary to expectations, RAMCAP performance did not improve as the number of scenarios increased in the simulations. The methods and results of this study may hold implications for other critical infrastructure risk methodologies that are based on consequence, threat and vulnerability.


Introduction
Concerns about the threats to the water and wastewater infrastructure posed by climate change, aging infrastructure and cyber attacks prompted the Science and Technology Directorate of the Department of Homeland Security to undertake the development of a new risk analysis standard that uniformly measures risk across all the lifeline infrastructures.According to the 2013 National Infrastructure Protection Plan (NIPP) [23], the lifeline infrastructures include water, energy, transportation and communications, four of the sixteen sectors identified in Presidential Policy Directive 21 (PPD-21).Uniform risk analysis, the ability to compare risk analysis results across infrastructure assets and sectors, facilitates cost-benefit analysis and strategic planning that are critical to optimizing homeland security investments and safeguarding the nation from catastrophic incidents, both natural and human initiated.A uniform risk analysis can also help the Department of Homeland Security achieve its goal of measuring resilience and quantifying the efficacy of countermeasures -in other words, inform the President and Congress where we are, where we are going and at what cost.
The importance of a uniform risk analysis approach for critical infrastructures was recognized by the White House when it requested the American Society of Mechanical Engineers (ASME) to develop a methodology shortly after the terrorist attacks of September 11, 2001 [2].In 2006, ASME released the final specifications of the Risk Analysis and Management for Critical Asset Protection (RAMCAP) methodology.RAMCAP is a seven-step process that assesses the risk to an asset as a product of threat, vulnerability, consequence, resilience and applied countermeasures.RAMCAP incorporates a reference set of 41 threat and hazard scenarios to guide estimates of its terms and render the methodology uniformly applicable across the infrastructure sectors, The 2006 National Infrastructure Protection Plan [22] recommended RAMCAP for conducting risk analyses, but the methodology was not mentioned in the 2009 and 2013 revisions of the national plan.No RAMCAP implementations are known to be employed today.However, RAMCAP continues to serve as the basis for the American Water Works Association (AWWA) J100-10 standard for Risk and Resilience Management of Water and Wastewater Systems [2].
In October 2014, the University of Colorado at Colorado Springs launched a RAMCAP needs assessment to develop the requirements for uniform risk analyses of lifeline infrastructures.The project involved three tasks: (i) analysis of emerging threat and hazard scenarios; (ii) analysis of RAMCAP performance; and (iii) analysis of RAMCAP requirements.Task 1, completed on December 31, 2014, identified 38 candidate scenarios with catastrophic potential for the water, electricity, aviation and Internet subsectors due to emerging threats from climate change, aging infrastructure and cyber attacks.After combining the similarities and eliminating redundancies, the candidate scenarios were reduced to thirteen nominee scenarios.Based on the Task 1 results, the question put before Task 2 was whether or not RAMCAP would perform better with 54 reference scenarios instead of its current 41 scenarios.This project was the first attempt to evaluate RAMCAP performance in general terms.This chapter discusses the RAMCAP methodology and its characteristics, and the unexpected results that ensued.

Background
The Homeland Security Act of 2002 prescribed a risk management approach for protecting the national critical infrastructure.Accordingly, the 2013 National Infrastructure Protection Plan specifies a risk management framework (RMF) to: (i) set goals; (ii) identify assets; (iii) prioritize risk; (iv) implement countermeasures; and (v) measure results [23].The risk management framework is implemented with voluntary participation by industry through Sector Coordinating Councils that represent the sixteen infrastructure sectors identified in PPD-21.Every four years, the Sector Specific Agency (SSA) federal representative compiles a Sector Specific Plan (SSP) that summarizes the risk management efforts for the assigned sectors [15].In 2010, the water, electricity, aviation and Internet subsectors identified the risk analysis methodologies used in support of Step 3 of the risk management framework (Table 1).For example, the VSAT methodology, which is employed by the water subsector, has been certified by the American Water Works Association to be RAMCAP compliant [2].
By one estimate, there are more than 250 critical infrastructure risk analysis methods [12].The question is why are there so many methods?One answer may be that critical infrastructure risk analysis is beset by tradeoffs.Each method represents a different set of tradeoffs, determining both the type and the terms of the analysis, as shown in Figure 1.
Tradeoffs that determine the type of critical infrastructure risk analysis begin with the question of completeness: Does one analyze the asset or the network?Some researchers [3,17] argue that an analysis is incomplete without considering interdependencies.Pederson et al. [18] identify 30 models that specialize in interdependency analysis.According to Creese et al. [7], interdependency models must be highly detailed to yield reasonable results.Since assets are part of the network detail, they must be assessed individually at some level.Thus, it is reasonable to begin risk analysis with an asset, but understand that the analysis is incomplete without including the network.
The next tradeoff involves qualitative risk analysis versus quantitative risk analysis.Qualitative risk analysis simplifies risk assessments by reducing inputs to a manageable set of judgments [6].The risk and vulnerability analysis (RVA) methodology is one example of a qualitative approach [10].A general criticism of qualitative methods, however, is that poor resolution of input data can lead to erroneous or misleading results [5,6].
In contrast, quantitative methods promote confidence in results by reducing subjectivity [25].However, quantitative risk analyses of critical infrastructures are tempered by precision.Unlike a disease, whose risk can be assessed in absolute terms [20], safety is not absolute and can only be assessed in a relative manner.This does not preclude the accuracy of results.Bayesian networks, conditional linear Gaussian networks, stochastic models and other formal quantitative methods have proven records of performance in diverse fields such as engineering, finance, healthcare and meteorology [1,9,11,19].What trips them up is the paucity of data for statistical analysis of catastrophic incidents, par-ticularly those involving malicious human intent [16].The attempts to work around this obstacle often lead to formulations that are neither transparent nor repeatable [16], rendering their consistent application problematic.The need for consistency has fueled the development of informal quantitative methods, many of which formulate risk as a function of threat, vulnerability and consequence.What distinguishes similar formulations [8] and even the U.S. Department of Homeland Security formulation itself [23] are the tradeoffs that are made in developing their terms.
Tradeoffs in developing terms for critical infrastructure risk analysis can take similar formulations and render them completely different from each other.Whereas the U.S. Department of Homeland Security has applied specific forms of the threat, vulnerability, consequence formulation [13], the National Infrastructure Protection Plan [23] does not specify any particular application.Therefore, it is not unreasonable for asset owners and operators to employ ad hoc methods for internal risk analyses and to assign threat, vulnerability and consequence values based on the best estimates of onsite personnel.To be sure, expert elicitation is an acceptable form of value estimation [16], but the consistency required for effective comparisons will not be realized without employing a formal system.
Rigorous systems for estimating threat, vulnerability and consequence values encompass various means of elicitation and modeling.The Delphi method is, perhaps, the best known rigorous system among the various elicitation methods [4].Fault trees, event trees, reliability block diagrams and other causal analysis methods are well respected in the reliability and safety engineering discipline [16,21,24].However, rigorous methods require substantial investments in time and resources that make them impractical for large-scale applications.Alternatively, a bounded system could elicit threat, vulnerability and consequence values with respect to a set number of scenarios as proposed in [14].The approach may be less rigorous, but it is also less resource intensive and, thus, practical for large-scale applications.
RAMCAP takes the bounding process one step further by stipulating a homogenous set of reference scenarios.The same reference scenarios are used in each analysis to facilitate comparisons across assets.An important component of Task 2 of the RAMCAP needs assessment project was to determine if additional reference scenarios would improve RAMCAP performance.

Problem Formulation
The chosen method of analysis for Task 2 was to develop a basic RAMCAP model and evaluate its performance under varying load scenarios.According to the AWWA J100-10 specification, RAMCAP calculates a net benefit value and a benefit-cost ratio [2].
The gross benefit Gb is the amount of risk-resilience gained by implementing a particular countermeasure for a given threat-asset pair: Table 2. Seven-step RAMCAP process.
Step Description Step where Rs and Rs ′ correspond to the risk-resilience values with and without a countermeasure, respectively.The net benefit N b is the sum of gross benefits for all the threat-asset pairs associated with a particular countermeasure: Note that the higher the net benefit N b, the greater the reduction in risk.
The selection of the countermeasure to be implemented is determined by the benefit-cost ratio BCR, which is computed by dividing the net benefit N b by the cost of the countermeasure.The higher the benefit-cost ratio BCR, the greater the return on investment.Because the task was to compare performance, the benefit-cost ratio BCR was not computed.Thus, RAMCAP model performance was based on the computed net benefit N b.
RAMCAP calculates the net benefit N b using the seven-step process shown in Table 2.The process is very site-specific, especially when characterizing assets and countermeasures, and assessing consequences, vulnerabilities, threats and resilience.About the only independent component is Step 2 (threat characterization), which is based on the 41 RAMCAP reference scenarios.The challenge was to apply this specific approach to a general situation and evaluate RAMCAP performance against every conceivable threat-asset pairing under a number of scenarios for every possible combination of consequence, vulnerability, threat, resilience and countermeasure.
A basic RAMCAP model was developed by following the AWWA J100-10 seven-step process beginning with Step 1 (asset characterization).In RAM-CAP, an asset is part of a system comprising some component of a critical infrastructure [2].For example, a generator is part of a power plant, which is a component of the electricity infrastructure.RAMCAP Step 1 identifies assets whose disruption or destruction could result in "worst reasonable consequences" [2].RAMCAP leaves it to the user to decide what constitutes the worst reasonable consequences.
RAMCAP Step 1 is conceptualized by assuming some combination of assets A whose disruption or destruction have consequences C that are some fraction of the worst reasonable consequences WRC : where A takes values from the unit interval [0,1].RAMCAP Step 2 (threat characterization) aligns the identified assets with the 41 reference scenarios to form threat-asset pairs [2].The purpose of this step is to set up subsequent value estimations and risk calculations for each threat-asset pair.Because A represents a combination of assets, not individual assets, the effect of this step is to control the number of value estimations and risk calculations based on the number of reference scenarios.Accordingly, this is the step in the model where the number of reference scenarios is varied to evaluate RAMCAP performance under different loads.
RAMCAP Step 3 (consequence analysis) assigns a magnitude to the worst reasonable consequences WRC .The magnitude is the sum of the individual estimates of fatalities, injuries and financial and economic losses converted to point values using the provided charts [2].RAMCAP calculates the worst reasonable consequences WRC for each threat-asset pair.Again, because A represents a combination of assets, WRC can be assigned a value of 1.0 and Equation (3) reduces to: Note that this assignment avoids controversial conversions between personal injury and property damage.Also, it implicitly accounts for disruptive incidents as well as destructive incidents.RAMCAP Steps 4 through 6 address the risk and resilience calculations.RAMCAP calculates the risk for each threat-asset pair as the product of consequence, threat and vulnerability.Step 4 estimates the probability that a given asset will be disrupted or destroyed by a given scenario for each threat-asset pair [2].Step 5 estimates the probability that a given asset will be subjected to a given scenario for each threat-asset pair [2].
In Step 6.1, RAMCAP calculates the risk as [2]: Upon substituting Equation (4) into Equation ( 5), the following risk formulation R is obtained: RAMCAP Step 6.2 calculates the resilience [2].According to the AWWA J100-10 standard, perfect resilience is the ability to withstand a threatened incident [2].Since vulnerability accounts for the inherent ability of a system to withstand a threatened incident, resilience presumably accounts for external mitigating factors such as first responders, National Guard and other such capabilities from outside the fence.
Accordingly, RAMCAP calculates resilience as a fraction of risk by attenuating the duration D and severity S as follows [2]: This computation is generalized by combining D and S into a single mitigating factor F (F ∈ [0, 1]) representing a percent reduction in expected consequences brought about by external agents.
A risk reduction multiplier M is then computed as: and the resilience Rs is computed as: Substituting the risk R in Equation ( 6) into Equation ( 9) yields: RAMCAP Step 7 adds the effects of countermeasures to the risk and resilience calculations, and takes the difference before and after countermeasures are applied to calculate the gross benefit Gb, net benefit N b and benefit-cost ratio BCR.RAMCAP Step 7.2 calculates the effect of countermeasures on risk and resilience [2].Because countermeasures attenuate risk similar to resilience, a mitigating factor Next, a reduced risk profile after applying a given countermeasure R ′ is computed as: Applying the same countermeasure to resilience yields the increased resilience after applying a given countermeasure Rs ′ : RAMCAP Step 7.6 calculates Gb, N b and BCR [2].For the purposes of this model, it is assumed that all evaluated countermeasures result in risk reduction, therefore R > R ′ and Rs > Rs ′ .Gb is computed according to Equation ( 1) for all threat-asset pairs.N b is computed according to Equation (2) as the sum of Gb for all threat-asset pairs.
The preceding formulations make it possible to construct a general model for evaluating RAMCAP performance across every conceivable threat-asset pairing under a varying number of scenarios for every possible combination of consequence, vulnerability, threat, resilience and countermeasure.for V = r to 1 step r do 7: for T = r to 1 step r do for F = r to 1 step r do for F ′ = r to 1 step r do 14: end for (various resilience)

24:
end for (various threat probabilities)  also summed to compute an overall average net benefit N bavg for the entire simulation.Thus, RAMCAP performance for the n scenarios is characterized by the computed average net benefit N bavg and the corresponding distribution curve.The average net benefit N bavg was expected to increase as the number of scenarios increases; interestingly, this did not occur.

Results
The basic RAMCAP model specified in Algorithm 1 is designated as SIM1.SIM1 was almost immediately upgraded to SIM1b, which replaces the average net benefit N bavg calculation in Line 28 with the running average calculation: in Line 21. SIM1b was executed ten times while varying the number of scenarios n from 10 to 100 in increments of 10.For each execution, the simulation resolution r was set to 0.01 and the number of bins n was set to 20. Figure 2 presents the results.As expected, the distributions are proportional; the curves have the same shape, but the magnitudes are larger as the number of scenarios increases.
However, the SIM1b results in Figure 3 are surprising -N bavg decreases as the number of scenarios increases.The results may be explained by observing that, in Figure 2, the smaller N b values on the left-hand side of the graph increase more than the larger N b values on the right-hand side of the graph The curious results produced by SIM1b necessitated another look.SIM1b is a deterministic model that helps examine the hypothetical limits of RAM-CAP performance.On the other hand, RAMCAP was specifically developed to analyze low-frequency, high-consequence events that are homeland security concerns.SIM2 was developed as a probabilistic model that generates random Gaussian values between 0 and 1 for A, T , V , M and M ′ .The calculated average net benefit N bavg would be different for each execution.Consequently, SIM2 required multiple executions to calculate the average net benefit N bavg.
Figure 4 plots the average net benefit N bavg for 1,000 SIM2 executions for 40 scenarios at 0.05 simulation resolution with 20 bins.As shown in Figure 4, the average net benefit computed by SIM2 for 1,000 executions appears to be well behaved, clustering fairly tightly in the range between 0.2904 and 0.6954.The average of all these averages is 0.485125, which is considerably lower than the N bavg value of 0.7199 computed by SIM1c.In fact, none of the SIM2 results approach the N bavg values computed by SIM1c.5, the curve representing the SIM2 N bavg distributions rises much higher in Bin 1 than that for SIM1c, pulling down the SIM2 N bavg results as seen previously.SIM1c has an additional advantage because it has a larger number of higher values than SIM2; this is because the SIM1c curve cuts across and rises above the SIM2 curve in Figure 5.Given these observations, it is expected that the SIM2 N bavg values would be generally lower, albeit no less valid, than the SIM1c N bavg calculations.
Hence, the SIM1b experiment was repeated using SIM2.In particular, SIM2 was executed 100 times each for 10 to 100 scenarios.More iterations were possible, but 100 iterations required 29 hours even at a reduced simulation resolution of 0.05. Figure 6 presents the SIM2 results.
The SIM2 results in Figure 6 are somewhat misleading, suggesting a large fluctuation with no clear trend.In reality, a 0.024556 difference exists between the largest value of 0.495424 and the smallest value of 0.470868.A trend is apparent in Figure 7, when the SIM2 results are compared against the results obtained for the same execution using SIM1c.
Note that the simulation resolution was reduced from 0.01 for SIM1b to 0.05 for both SIM1c and SIM2, so the same downward trend seen in Figure 3 is not seen in Figure 7.However, the expected clear upward trend is also not seen in

Conclusions
The search for a uniform risk analysis methodology for critical infrastructures prompted a re-evaluation of RAMCAP to determine if it can accommodate emerging threats posed by climate change, aging infrastructure and cyber attacks.This research has examined the challenges of taking a site-specific formulation and turning it into a general model capable of analyzing performance under a full range of simulated conditions.
A basic RAMCAP model with a systematic attenuation of consequences based on estimations of probabilities for threat, vulnerability, resilience and countermeasures was developed.The model was made possible by formalizing the relationship between consequences, worst reasonable consequences and assets, and then normalizing the worst reasonable consequences.This insight eliminates the problems associated with defining consequences in terms of injury, death and damage; additionally, it implicitly covers disruptive as well as destructive catastrophes.
The model also considers resilience and countermeasures as risk mitigating factors.This insight simplifies the estimation of both terms and enables them to be incorporated as individual risk reduction multipliers.Simulations involving 10 to 100 scenarios quantified RAMCAP performance in terms of an average net benefit and net benefit distribution -the higher the net benefit, the better the performance.Since the scenario parameter only controls the number of internal iterations, it was expected that more scenarios would result in a higher average net benefit.However, this did not occur -as the number of scenarios increases, RAMCAP performance and the net benefit metric decrease.
The counterintuitive results prompted a second experiment using a probabilistic model.Instead of calculating parameter values, parameters were assigned random Gaussian values.The second experiment resulted in no improvement in RAMCAP performance.An examination of the corresponding distribution curves reveals that the magnitudes of the curves increase as the number of scenarios increases.However, as the number of scenarios increases, the number of lower net benefit values is proportionally higher than the number of larger net benefit values; the smaller values tend to outstrip the higher values, contributing to the decrease in the average net benefit value as the number of scenarios increases.The immediate implication is that, contrary to intuition, adding more reference scenarios does not improve RAMCAP performance.These counterintuitive results may also apply to other critical infrastructure risk methodologies that, like RAMCAP, are based on consequence, threat and vulnerability formulations.

25 :
end for (various vulnerability probabilities) 26: end for (various scenarios) 27: end for (various asset-WRC combinations) 28: Nbavg = Nbsum/x Algorithm 1 specifies the basic RAMCAP model computations.The RAM-CAP model has three inputs: (i) number of scenarios n; (ii) resolution of the simulation r; and (iii) number of bins b for tabulating the calculated net benefits.The reference scenarios are reduced to a set of threat and vulnerability values as determined by the simulation resolution r.Because the generated threat and vulnerability values span the entire range of combinations, they can represent any type of natural hazard or human initiated threat at any geographical location.The simulation resolution r directly determines the magnitude of the generated values and number of loop iterations.By the same token, the resolution r indirectly determines the time required to execute the simulation.For each combination of assets A, the RAMCAP model computes the net benefit N b for varying combinations of vulnerability, threat, resilience and countermeasure for the given number of scenarios.The calculated N b values are tabulated in b bins in order to graph the resulting distribution.The calculated N b values are

Figure 5
Figure5may help explain the difference between the SIM2 and SIM1c results.As seen in Figure5, the curve representing the SIM2 N bavg distributions rises much higher in Bin 1 than that for SIM1c, pulling down the SIM2 N bavg results as seen previously.SIM1c has an additional advantage because it has a larger number of higher values than SIM2; this is because the SIM1c curve cuts across and rises above the SIM2 curve in Figure5.Given these observations, it is expected that the SIM2 N bavg values would be generally lower, albeit no less valid, than the SIM1c N bavg calculations.Hence, the SIM1b experiment was repeated using SIM2.In particular, SIM2 was executed 100 times each for 10 to 100 scenarios.More iterations were possible, but 100 iterations required 29 hours even at a reduced simulation resolution of 0.05.Figure6presents the SIM2 results.The SIM2 results in Figure6are somewhat misleading, suggesting a large fluctuation with no clear trend.In reality, a 0.024556 difference exists between the largest value of 0.495424 and the smallest value of 0.470868.A trend is apparent in Figure7, when the SIM2 results are compared against the results obtained for the same execution using SIM1c.Note that the simulation resolution was reduced from 0.01 for SIM1b to 0.05 for both SIM1c and SIM2, so the same downward trend seen in Figure3is not seen in Figure7.However, the expected clear upward trend is also not seen in

Figure 7 .
Figure 7.The results in Figure 7 support the previous observation that adding more reference scenarios does not improve RAMCAP performance.

Table 1 .
Lifeline infrastructure risk analysis methodologies.