Multi-stakeholder Preference Analysis in Ex-ante Evaluation of Policy Options - Use Case: Ultra Low Emission Vehicles in UK

. While the simulation-based impact assessment of public policy proposals allows policy makers to identify the feasible policy options and verify their economic, social and environmental impacts, it does not provide the explicit evaluation of policy options. Multi-criteria decision analysis (MCDA) techniques can support an in-depth performance evaluation of policy options taking into acount the preferences of decision makers and stakeholders. These preferences reflect acceptable trade-offs of performance among objectives. This study re-views multi-attribute decision-making (MADM) technique and presents a common policy appraisal format using main evaluation criteria linked to a set of measurable, context dependent attributes. We argue for a rank-based approach for eliciting preferences, select a novel method for attribute weight elicitation, and show how it can be integrated within a public policy multi-criteria evaluation framework. A use case for policymaking, ‘Ultra-Low Emission Vehicles (ULEV) Uptake in UK’, is used for demonstration of the proposed approach for policy decision analysis. This approach seeks to couple systems modelling and simulation of policy scenarios with MCDA, stakeholder analysis and preference elicitation. The outputs can further provide analytical insights in controversy/accept-ability of policy options, and consequently guide further policy formulation and the design of better options.


Introduction
Public policy decision processes are often characterized by the presence of multiple and conflicting objectives and multiple stakeholders or decision makers who may have differing point-of-views. Using decision analysis practices to provide informed public policy decisions is not a novel approach per se. What might differ public policy decision analysis from a traditional decision analysis endeavor is that the former does not explicitly aim to reveal a clear recommendation of choice, but also to explore the problem and reveal conflicts, cf. (Quade, 1982).
Prescriptive approaches for the study of such decision processes have been suggested within the field of systems thinking and in the field of multi-criteria decision analysis (MCDA). In (Larsson and Ibrahim, 2015), we presented a work process with associated operational research (OR) modeling and analysis tools that supports a prescriptive analysis for policy that includes: the problem definition, ex-ante impact assessment and evaluation activities carried out at the policy formulation stage of the policymaking process. The proposed approach exploited the use of causal maps for problem structuring and scenario-based simulation for the design of policy options, together with decision analysis for evaluating generated scenarios. Having a set of feasible policy options identified, there are two main tasks remaining in structuring MCDA evaluation models; (i) representation of objectives in a structure, commonly a value tree, and (ii) the definition of attributes to measure the achievement of objectives (Franco and Montibeller 2010).
Multi-criteria evaluation can be organised with an objective to produce a single synthetic conclusion or to produce conclusions adapted to the preferences and priorities of several actors. Thus, applying MCDA should provide the relative global performance of each alternative taking preferences of decision makers and stakeholders into account. This is particularly useful when selecting one out of a finite set of feasible alternatives.
Since the rise of the graphical user interface, a flora of computer based decision analysis tools have emerged, aiming to exploit the strengths of graphic interaction with users and thereby enabling for users with less know-how of decision analysis to conduct decision modelling and evaluation, see, e.g., (Riabacke et al. 2014). A prerequisite to increase the practical aspects of decision analysis is to offer accessible tools, and of greatest importance for practical applicability of decision analysis methods seems to be how "easy" they are to use (Stewart 1992, Hülle et al. 2011).
This work is carried out within the contexts of the research project Sense4us 1 , which is developing a web portal that integrates public policy decision support tools to linked open data search tools and social media analytics, in order to enable easy access to information sources and knowledge creation. The aim of this paper is the design of a preference elicitation method based on a multi-criteria evaluation model for policy decision analysis. The method should: (i) not require substantial formal decision analysis knowledge; (ii) not be too cognitively demanding by forcing people to express unrealistic precision or to state more than they are able to; (iii) not require too much time; (iv) make use of the information the decision-maker is actually able to supply, and (v) be supported through a graphical user interface (GUI) accessible from different client operating systems.
This paper is structured as follows. Section 1, introduces the research problem, context, challenges and tasks. Section 2 provides a review of the state-of-the-art research related to concept and application of decision analysis and preference elicitation for policy analysis. Section 3 presents the proposed criteria model for ex-ante evaluation of policy options. Section 4 presents the proposed rank-based preference elicitation method. Section 5, describes the public policy use case, as real example for demonstration of the use of the proposed policy decision analysis process. Finally, Section 6, provides the conclusions and possibilities for future work.

Related Work
Elicitation of preferences in policy making requires that there exists a basis for decision, typically in the form of more or less objective so-called "impact assessment reports" describing the impact of different alternatives on objectives but strive to be neutral with respect to decision maker preferences. The impact assessment approaches can be more or less . One important feature of problem structuring approaches is to facilitate the context-setting activity and defining the environment of a decision problem to be modelled and subsequently evaluated, (see, e.g. Belton and Stewart (2002) for an introduction to how problem structuring relates to the decision analysis process).
Further, the use of decision analytic support in participatory/group decision processes involving several decision makers and/or stakeholders has received increasing attention during the last few decades. The decision makers can actually disagree on what the best alternative is but they need to select a common alternative having sufficient support from the group (Kilgour et al. 2010). Within such a context, a number of proposed decision analytic approaches and cases of its use have been published in the literature. For instance, Danielson et al. (2007) propose the analytical decision layer process, incorporating the stakeholders and decision maker views in an interaction layer and aggregating these preferences in a decision layer exploiting methods from interval decision analysis to accommodate for differing preferences such that all decision maker values are captured by each interval. Hansson et al. (2008) presents an MCDA approach for evaluating flood management strategies, putting weights on the stakeholders when aggregating their preferences. Schroeder and Lambert (2011) outlines a decision analysis process for comparison of transportation policies in the case of an emergency exploiting scenario planning in order to set the context.
Multi-Attribute Decision Making (MADM) techniques can be applied when the decision consists of selecting one out of a finite set of feasible alternatives, (Belton and Stewart, 2002). The process of MADM is an analytical process that brings together three components; the decision objectives as measured by their associated attributes, the decision options or alternatives and the decision maker's preferences toward the importance of the attributes as reflected by the attribute weights.
The determination of attribute weights is an important and time consuming process. Attribute weight elicitation methods, given the decision problem's structural elements, include: the Balance Beam (BB), Lottery Technique (LT), Paired Comparison (PC), Point Allocation (PA), Pricing Out (PO), Ratio Weighting (RW) and Trade-off Weighting (TW), (for a comparative review see (Crain, 2003)). These methods can be distinguished by the specific technique employed to compute the attribute weight and whether the technique is designed to determine an attribute importance weight or an attribute swing weight. Importance weights reflect the relative preference without regard to the decision problem at hand, whereas swing weights are sensitive to the values that the attributes possess for the particular decision problem. Crain (2003) emphasized the greater utility and correctness of eliciting swing weights in a MADM problem versus importance weights because of the sensitivity of the swing weight to the range of values taken on by the associated attribute.
A common practicability issue of applying decision analysis methods is that often too hard requirements are put on the users. It can be argued that much of the applicability issues in employing formal decision analysis in practical settings roots in the need to elicit beliefs and preferences from decision makers and stakeholders. One recurrent practicability issue reported in the literature is the need to obtain the decision-maker's preferences in such a way that they can be formally represented. That leads to cognitively demanding methods less encouraging for decision makers to use. See, for instance (

A Common Policy Appraisal Format
In MADM techniques, the decision process starts by structuring the problem as an attribute tree hierarchically ordering the decision makers' aims at different abstraction levels. It is generally assumes that each criterion can be operationalised by a set of measurable attributes allowing for assessing the consequences arising from the implementation of any particular alternative. In the next step preferential information is elicited. The relative importance of criteria is captured in weights for each criterion at each abstraction level . At the lowest level of the value tree these objectives are translated into attributes, with each one of them evaluating a given characteristic of the decision options (for example, an objective 'efficiency' may be measured by the attribute 'operating cost'). The performance of each decision option against each attribute is determined and weights are elicited from the decision-makers (Comes T., et al. 2011). Figure 1, illustrates a multi-attribute value tree, with measurable attributes in gradient boxes, criteria and overall goal in white boxes. Dependence of attributes on the alternatives (represented as diamonds) is shown by dashed lines.
Given a set of decision options, = { 1 , … , } and evaluation criteria, we can represent a decision option with a vector of performance levels ( 1 , 2 , … , ).
Conforming to multi-attribute value theory (MAVT), the global value of a decision option is given according to the additive value function: (1) where ( ) is a value function representing the value of alternative under criterion , is the weight of criterion , 0 ≤ ≤ 1 and = 1. This additive model cannot capture the dependencies between criteria, (i.e., being substitutes or complements of each other). We refrain from eliciting such dependencies from decision makers, to achieve the use of use of the elicitation method. We propose a criteria model for ex-ante evaluation of EU policy interventions, based on a set of main evaluation criteria 2 Effectiveness, Efficiency, Coherence, Relevance and Added value and the underlying measurable attributes. For each of the policy options, the observed changes on the targeted policy impact variables and the policy financial impacts are provided by the impact assessment results. Figure 2, illustrates the structure of the proposed common policy appraisal format as a multi-attribute value tree. The figure shows policy options to be evaluated.
Each of the attributes underlying the "effectiveness" evaluation criterion reflects how successful a policy option is in achieving the policy goals by comparing the observed change (OC) to the targeted change (TC) for each impact variable. The attributes underlying the "efficiency" criterion are concerned with the financial impacts of a policy option, whether financial costs or benefits (cost savings). The evaluation of the impact of the action is expressed as the costs involved on governmental departments given the achieved changes. In addition costs on businesses (including small businesses), or different stakeholder groups are expressed in terms of different attributes. The efficiency evaluations can be made in a quantitative or a qualitative way. The attributes underlying the "relevance" criterion are using qualitative descriptors (or value scales) to assess the relevance of a policy option to the policy objectives, to the problems/needs, to the technological advances and to the citizens. The attributes underlying the "coherence" criterion are using qualitative descriptors or value scales to assess the coherence of a policy option: (i) internally if it includes multiple policy instruments, (ii) with other similar interventions, (iii) with EU policy interventions and (iv) with international obligations. Finally the attributes underlying the "added-value" criterion are using qualitative descriptors or value scales to assess the added value of the policy option in terms of legal impacts, increased effectiveness, complementarities and the need for continuing this intervention. if the preference direction is negative. This is often labelled as "proportional scores" and is an intuitive way to generate a value function.
2) Use surrogate values derived from ranking statements according to the novel CAR method (Danielson and Ekenberg, 2015b). For the CAR method, we use > ( ) to denote the strength of the rankings between criteria or alternatives, where = 0 means that they are equally ranked, > 1 means that option is better important than option , and , > 2 means that option is "much better" than option and so forth. This can be represented such that each option is assigned a preference position ( ) ∈ { 1, . . . , } where lower position indicate stronger preference such that for two options whenever , For example, Figure 3 shows four criteria and a scale with eight steps where scale position 1 is at the right end of the slider and scale 8 at the left end of the slider. The ranking visualized is 4 > 2 3 , 3 > 1 1 and 1 > 4 2 , yielding surrogate weights 1 = 0.21, 2 = 0.06, 3 = 0.26, and 4 = 0.47.

Figure 3 Ranking of criteria
Further, following proportional scores value assignments for all three criteria but the Social criterion, for which we have the following ranking for options 1, 2, and 3, a 1 > 3 a 3 , a 3 > 2 a 2 yielding values v 3 (x 13 ) = 1, v 3 (x 33 ) = 0.4, and v 3 (x 23 ) = 0.
This can be illustrated using a multi-criteria evaluation matrix, constructed with as many columns as there are criteria and as many rows as there are decision options to be compared. Table 1 shows exemplary impact assessment results, each cell represents the performance of option under criterion in the performance indicator used for that particular criterion. Note that: MCDA requires an evaluation of all the decision options for all the criteria, but does not require that all the evaluations take the same form. The technique can support a mix of quantitative criteria expressed by indicators, qualitative criteria expressed by descriptors, and intermediate criteria expressed by scores (e.g., a scale 0-10). x 31 = 10% x 32 = 615 x 33 =Neutral x 34 = 6 Table 1 : Impact assessment matrix In Table 2 below, we denote positive preference direction with '↗' and negative direction with '↘' together with S or R representing "proportional scores" or "ranking" respectively for the value function model. Unequal weights are assigned to criteria (note that the sum of the weights 1 , … , equals 1).  Thus, the preference elicitation methods proposed for enabling decision evaluation in the policy formulation process are:  For decision option/alternative values: Cardinal ranking or proportional scores, complemented with direct rating.  For criteria weights: Cardinal ranking, complemented with direct assessment. The reason for complementing the ranking approach with direct rating is that the latter provides flexibility for exploiting other preference elicitation schemes for advanced users. this way, it was relatively straightforward to define preference directions or neutrality (no interest) with respect to the policy evaluation criteria. The selection of the concepts as evaluation criteria and/or attributes for each stakeholder group, has been based on the following principles: (1) the criteria set should represent balance and diversity of aspects for a given stakeholder group, (2) the criteria in the set should be relevant and representative, and (3) preferentially independent for a focal stakeholder group.
For instance, as a representative of the "Government" stakeholder group the Office for Low Emission Vehicles (OLEV) of the British government has been selected. The following attributes were considered for evaluation of the effectiveness of policy options by the government stakeholder group: ULEV industry economic value added, new jobs created, green-house gas (GHG) / CO2 emissions, urban air pollution and local electricity demand.
The proposed policy decision analysis process supports a visual hierarchy of criteria. Figure 4, shows an example for the resulting criteria hierarchy tree with exemplary criteria and attributes weights.

Figure 4
Visual hierarchy of the evaluation criteria for the use case A relative global performance of the policy options for stakeholders can be calculated through a step-wise decision evaluation process to define: (i) Stakeholder groups and/or individual stakeholders; (ii) Evaluation criteria and/or the underlying attributes, each stakeholder stipulates its own set of criteria and attributes; (iii) Scenarios/Alternative policy options are common for all stakeholders; (iv) Scenario values: using cardinal ranking statements or surrogate value statements (a point, an interval or both) by each stakeholder; (v) Criteria weights: using cardinal ranking statements. Figure 5 and Figure 6 provide examples for ranking of criteria or scenario values utilizing a friendly GUI.

Figure 5
Example for criteria ranking using a GUI Figure 6 Example for scenario values ranking using a GUI

Conclusions
The proposed common policy appraisal format provides a basis for computational decision analysis for evaluation of policy options. Further, identifying the stakeholders groups enables appraising a policy from the perspective of multiple stakeholders with different priorities and preferences. By allowing for preference modelling, we can gain insights into the level of disagreement on policy proposals as well as how a policy could be efficiently refined for the mutual benefits of two or more stakeholder groups or arriving at more than one competing policy options for further decision analysis. The preference elicitation method introduced in this study, uses rank-based elicitation complemented with proportional scores or direct rating in order to provide flexibility for advanced users. Future work aims to design an integrated decision support tool that couples systems modelling and simulation of public policy problems with MCDA techniques and preference elicitation methods. The implementation of the proposed decision analysis process using an online GUI, enables policy makers and stakeholders to provide preferential statements over their respective goals in a simple way.