Developing Criteria for Evaluating a Multi-Channel Digitally Enabled Participatory Budgeting Platform

. “Enabling Multichannel Participation through ICT Adaptations for Participatory Budgeting ICT-enabled platform” (EMPATIA) 1 is a multi-channel participatory budgeting (PB) platform that represents a significant social innovation process of democratic deliberation and decision-making, involving citizens within complex public-institution structures. EMPATIA was targeted to deliver socio-economic and political benefits, such as enhancing citizen-government engagement, increasing public value through PB process, promoting ‘inclusiveness’ among the marginalized groups of citizens, and impeding political discontent that underpins distrust and skepticism towards the government. The attainment of these benefits will be driven by the EMPATIA's performance. Hence, a performance measurement tools is needed to enable assessment of EMPATIA, empirically. With an aim to propose an integrated performance evaluation metrics, this study presents a set of assessment criteria for multi-channel digitally enabled PB service platforms – especially EMPATIA. Findings from a qualitative, multi-strategies research approach suggest that the metrics should include five key technical and non-technical performance indicators, to be used as basis for the development of future evaluation instruments. Of major signposts, the metrics would inform key performance aspects to be considered during the PB platform development, and evaluated to indicate the PB platform performance.


Introduction
The emergence of Participatory Budgeting (PB) programs in the public institutions were often linked to the increasing pressures imposed by the stakeholders including the governments, citizens and non-governmental organizations (NGOs). The central target is to provide opportunities for citizens to deliberately negotiate over the utilization and allocation of public money [1]. Besides fostering citizens'-government engagement and increasing transparency in public service delivery, subliminally PB has potentials to educate and empower the citizens, especially those niches. Traditionally, low income and low-level of formal education citizens refrained themselves from involving in the government-related decision-making activities. By empowering them, the government could expand outreach and enhance the quality of democracy [1]. Acknowledging these potentials, PB could be a tool for more inclusive and accountable governance, contributing towards a higher public value [2,3]. Many public institutions use social media (e.g. Facebook, Twitter) to engage with their citizens [4,5]. Nevertheless, such alternatives constraint the citizens' empowerment process [6,7] due to the absence of real time process and the "virtuous circles" [8,9]. Hence, the need for a digitally-enabled PB platform triggers, leading to the development of EMPATIA (Enabling Multichannel Participation through ICT Adaptations for Participatory Budgeting ICT-enabled platform). EMPATIA was developed under the framework of European Union's innovation and research programme i.e. Horizon 2020. Its ultimate aim is to benefit global society and democracy [10]. The platform integrates the existing e-government platforms, and adds new features such as auto generation of data and visualization, voting systems and opinion mining to accommodate the desired PB function and enhance performance. It was hoped that these features would entice its adoptions, thus help to achieve the policy objective. However, the failure of past e-government platforms due to scarcity of adoptions had triggers a risk to EMPATIA. To mitigate such risk, the critical performance indicators need to be identified. Hence, an objective and cohesive evaluation metrics is needed to facilitate such action -which is what this study attempts to propose. The rest of this paper is organised as follows. Following this introduction, the paper presents the conceptual background discussing on the significance of digitally enabled platform and participatory budgeting. Then, a description of the methodological approach was outlined, entailed by discussion on the EMPATIA's evaluation metrics. Finally, the paper concludes by highlighting the expected impact of the EMPTIA platform and way to move forward.

Conceptual Background
Digitally enabled service transformations offers vast potentials to create selfsustaining change in a broad range of connected technology, universal culture, and closely-linked communities [11,12]. Explicit growth of digitally enabled services in public institutions were flourished by the expectancies to enhance governance process, increase citizens' participations, and break the siloes between public institutions [12]. Such trend is also affiliated to the impact of communication technology advancements [3]. Many do not realized that the phenomenon partly caused by the New Public Management (NPM) movement [12]. NPM had redefined public services and government-citizens' relationship, raising expectations on service standard and quality [13]. The scenario increases adoption of private sectors' practices, including computer-aided services that subliminally deter democratic values by treating a citizen as customer and imposing charges for rendered services. Next, the focus was shifted towards the quality of governance, where the concept of "participation" was introduced [14]. Such concept urged revitalization of the public sector's roles through "partnerships" between government-citizens to improve social well-being and the quality of public administration, leading to the emergence of the participatory budgeting (PB) concept [13]. Meanwhile, the evolution of technology, increasing demands / expectations of the civil societies and complex political inferences prospers the growth of digitally-enabled services in public institutions [15]. The growth was also due to the over-arching economy pressures, which has channeled the focus of service delivery from public value (PV) creation (evaluated against services, outcome and trust) to cost feasibility [16]. Along the same line, the PV concept was argued as mimicking the definition of 'perceived value' in marketing discipline. As such, it was claimed that the PV theory was unfit to evaluate the actual 'value' created by the digitally-enabled services -as the value was partly attributed to the information quality, system quality and service quality. In this perspective, value is an antecedent determining satisfaction and trust among the users that will lead to re-use intention and increases usage level that characterized the IS success [16]. The e-voting feature in EMPATIA is claim as the main PV determinant. Besides promoting citizens' empowerment, such feature facilitates political inclusiveness, while reducing contentment. Scholars have different views on how the introduction of online platform for participation will affects the citizens, politically. Evidence shows that the introduction of online platform for participation will encourage citizens' participations [17]. Nevertheless, the digital divide advocators suggest that due to unequal access to the internet, the online PB platform will increase the involvement number of "economically advantaged groups" (i.e. people from the middle class to high income groups) -who are already politically active, hence has null effect on the politically-excluded group [18]. Instead, it will further empower the groups who are already resourceful and determined to participate. As the effect varies, this aspect worth an assessment. To enable the strategy development for the 'politically excluded' groups, citizens' demographics information (e.g. income group, age, education level) and their motivation underpinning the system usage need to be retrieved. On top of the "inclusion" issue, the rising tide of discontentment in the global political landscape has aroused anti-politics orientation among the citizens [19]. This scenario creates the feeling of unrepresented in politic, unheard by the politicians, excluded due to social class and constrained opportunity to influence political decisions [20]. These feelings are often expressed in negative sentiments towards politicians, political institutions or politic per se, creating a gap between citizens and politically linked process, institutions or individuals [21]. Noting its ability to facilitate transparency and effi-cient execution of institutional roles, hypothetically, such issue can be reduced by PB implementation [22] -hence, should also be included in the metrics for evaluation.

Methodology
The overall methodological approach adopted to develop the EMPATIA's evaluation metric was based on qualitative approach. It was stated the use of a single strategy limits data richness about certain phenomenon, and therefore the combination of strategies in a single research allows the weaknesses of certain strategy to be complemented by others [23]. Thus, a combined strategy of state-of-the-art literature review (secondary research), expert views mining and focus groups were used to collect data on the potential key performance indicators that will allow objective assessment of the online PB platform from the users' point of view. Besides rich in data, such combination also facilitates the data triangulation process, which is underpins the rigorousness of research [24]. The process started with a comprehensive state-of-the-art literature review (SOTA LR) and archival research on the relevant public sector -technology adoption and diffusion reports and publications. Using the thematic analysis approach, the investigation's results were classified into two types of indicators -i.e. technical and non-technical, followed by an evaluation on its context-suitability against PB platform (i.e. EMPATIA). Next, a group of 20 experts dominating various roles across public and private sectors in several European countries were identified and invited to present their views towards PB platform in the two series of online 'expert mining sessions'. They include the renown IS/IT/e-Government/Public Administration scholars, public sector employees (i.e. council staff and policy makers) and private sector practitioners (e.g. IS/IT consultants / contractors / consortiums) who have vast experience and specialties related to the PB platform. The findings from SOTA LR were used to stimulate the experts' discussions and help to clarify certain issues. The outputs from the two sessions were used to structure an agenda for the subsequent focus group. Although it is not necessarily required [25], it was acknowledged that the agenda had elucidated themes for probing during the focus group sessions, which were held in three countries. Each session involves ten participants consisting the municipality staff, consortium members and public representatives (i.e. citizens). Finally, the findings were analyzed and triangulated to form a list of KPIs (and sub-indicators) that could facilitate an objective assessment of the PB platform's performance

Proposed Evaluation Metric
The process of making decisions pertaining budget allocations is part of the government's primary role. Hence, the implementation of a multi-channel digitally enabled PB platform, or in this context EMPATIA -implied that such role is about to be heavily determined by the citizens. In this case, the proposed evaluation metric for the platform should be more incline towards the citizens' interests. As stated in the earlier section, the platform was aimed to define new form of democracy for the 21st century nations and will be available freely for everyone to use. For such purpose, EMPATIA will be piloted in three different countries -each with unique requirements, scenarios, and users. Ideally, the platform should be assessed differently (i.e. according to their specific context). Nevertheless, the development of a generic evaluation metric is required as a performance baseline. Hence, this section presents the evaluation metric for EMPATIA's performance assessment, developed based on the synthesis of findings derived from data collection strategies that were described in section 2. The metric is displayed in Table 1. While the technical indicator focuses on the operation and performance of the platform's "network" and "architecture", the non-technical indicator evaluates the social-economy, behavioral, political and process aspects of the platform. To measure the citizens' acceptance and satisfactions derived from the platform usage.

Socio-Economic
To identify the socio-economy factors influencing citizens' decision to use the platform. Political To indicate the level of 'inclusiveness' (who participate in the PB process) and political alienation (unhappy or dissatisfy with certain aspects of society) from the platform use.

Process
To identify the 'process' factors influencing citizens' decisions to use the platform

Technical KPIs
Data syntheses suggest that 'technical indicator' is highly important for the measurement of the platform's technical-related performance. It is particularly critical, since the platform will be implemented across various contexts. Besides enabling the troubleshooting process, the indicator helps to determine the root of discrepancies in four dimensions: performance, usability, maintenance, and monitoring -adapted from the "Systems and software engineering -Systems and software Quality Requirements and Evaluation (SQuaRE) -System and software quality models", as outlined in ISO/IEC 25010:2011. Although SQuaRE proposes the assessment on all characteristics its two models of "quality in use" (focus on the outcome of interaction) and "product quality" (focus on the software's static properties and system's dynamic properties), findings suggests that only five characteristics (from the "product quality" model) are relevant to the research context [26]. The technical KPIs for EMPATIA are listed in Table 2.

Non-Technical Indicator: Behavioral
The behavioral indicator is one of the non-technical KPIs selected to indicate the PB platform's performance. The sub-indicators under the behavioral indicator category are proposed based on the findings of the literature review. It was discovered that various models / theories were introduced to facilitate investigations on the reasons underpinning technology acceptance among the users. Nevertheless, user satisfaction remains as a central focus in all approaches. Against this backdrop, a theory known as 'Unified Theory of Acceptance and Use of Technology (UTAUT) was developed to integrate eight 'technology acceptance and use' theories such as Theory of Reasoned Action (TRA) and Technology Acceptance Model (TAM), and used to explain the relationship between users' intentions and their subsequent usage behavior [27]. Although many argue that the IS design, implementation strategy, and usage level are the important determinants for IS success, Bailey and Pearson [28] suggest that 'user satisfaction' is the key driver for the IS usage, which leads to its success. Next, Technology Acceptance Model (TAM) that encapsulates 'user involvement concept' was introduced [29]. TAM demonstrates how perceived usefulness and ease of use; as well as attitudes and behavioral intentions resulted into IS adoption. This indicates that user involvement is key to IS success. Such insight was incorporated in the Information System Success Model (ISSM), where success was attributable to the information quality, system quality, use, user satisfaction, individual impact and organizational impact [30]. According to 'perceptions regarding information privacy' theory, system security and information privacy are two other critical factors affecting users satisfaction on IS usage, thus worth evaluation [31]. Hence, it is proposed that the PB platform behavioral model to include concepts from UTAUT, ISSM and 'perceptions regarding information privacy' theory, and used these concepts as subindicators for evaluation purpose.

Non-Technical Indicator: Socio-Economy
In a study that investigates citizens' perspective toward digitally enabled public service, three indicators i.e. technical, social and economy were combined to assess the socio-economy impact [see 32]. As this evaluation metric will be used against the same context of digitally-enabled service, the same indicators will be used to evaluate the PB platform performance from the socio-economy perspective. However, since the technical evaluation will be conducted separately, the metric for socio-economy impact will be limited to social and economy indicators, as outline in Table 3.

Non-Technical Indicator: Political
Political indicator is important to enable the evaluation of the PB platform's impact on the evolution of internal and external efficacy of users and overall trust in democratic institutions. Such evaluation can be performed by assessing the "inclusiveness" (i.e. to investigate participants' profiles), and "Political Alienation" (i.e. to investigate participants' "incapability" and "discontentment") [32]. Two types of political alienation are identified as incapability and discontentment [33]. Incapability refers to efficacy (either internal efficacy i.e. citizens' self-assessments of their own political judgments, or external efficacy i.e. citizens' perceptions on how they influence the political decisions), and discontentment (i.e. negative affectation towards political objects, explaining why people believed that certain government policies are biased, or why they distrust political authorities). Restoration of trust towards government's integrity in the decision-making processes is vital to remedy these situations [33]. As a platform that encourages transparency in public-budgeting processes, PB is advocated as a tool to fix the integrity issues, and subsequently enhance trust towards government. Thus, a metric to evaluate such political aspects of PB platform performance, should consist of two main types of indicators -(1) internal and external efficacy and (2) anti-politics (measures of trust), where comparison of the users' attitudes post PB platform usage with the baseline attitude of the random population in the same region is required to allow an objective assessment.

Non-Technical Indicator: Process
Process indicator is a required to assess the level of process standardization during the pilot implementation. It was proposed that the process indicator should outlines the basic requirements for the pilot implementation (i.e. to start the pilot), and criteria to exit the pilot (i.e. to stop the pilot) against user and process perspectives. Hence, the aspects to be considered in the evaluation metric under this indicator are listed in Table 4.

Conclusions
The 'participatory budgeting' (PB) concept was introduced against the backdrop of a contemporary public administration to provide a platform for citizens to involve in the public budgeting process. In general, PB was targeted to improve the quality of public services and social well-being. With the emergence of new technology, the digitallyenabled PB platform has sheds new lights in fostering better citizens-government engagement and broadening social-political impacts (e.g. improve economy conditions, enhance education level and flourish democracies). Nevertheless, the attainment of these desires is conditioned by the platform's success -measured by its level of usage. Usage is determined by series of cause-effect relationship. The process started with gaining value from the services, which raises satisfaction level among the users, producing the 're-use intentions'. Hence, value creation (often linked to the service performance) is essential to ensure the PB platform's success. To do so, the KPIs representing the PB platform's performance need to be identified, followed by the metric for evaluation. Since the existing models or theories offer limited scope of evaluation, this study proposes a new evaluation metric for the PB platform, outlining the technical and non-technical KPIs, developed against the context of EMPATIA. The metric served as a basis for the evaluation's instruments development in a near future, where the platform's performance result will be obtained. Besides signposting numerous potential practical and theoretical insights, the evaluation's outcomes will assist interventions, thus help to promote success of the digitally-enabled PB platform.