Benefits and Trade-Offs of Different Model Representations in Decision Support Systems for Non-expert Users

. Researchers have reported a lack of experience and low graph literacy as signiﬁcant problems when making visual analytics applications available to a general audience. Therefore, it is fundamental to understand the strengths and weaknesses of diﬀerent visualizations in the decision-making process. This paper explores the beneﬁts and challenges of an intuitive, a compact, and a detailed visualization for supporting non-expert users in two diﬀerent decision-making contexts. Using objec-tive and subjective means proposed by earlier work, we determine the beneﬁts and trade-oﬀs of these visualizations for diﬀerent task complexity levels. We found that while an intuitive visualization can be a good choice for easy level and medium level tasks, hard level tasks are best supported with a richer, yet visually more demanding visualization.


Introduction
In our daily lives, we may consider multiple risks when planning a financial investment, a project strategy, or when finding a clean and safe place to live.Visual analytics offers a promising approach to facilitate such decision-making tasks.Thomas and Cook [60] define visual analytics (VA) as the science of analytical reasoning facilitated by interactive visual interfaces.In its core, VA supports this analytic reasoning with automated visual data analysis [32].VA tools can provide control over a variety of settings, interactive discovery, exploration, and understanding of real-world complex systems [5].Moreover, VA gradually supports users in their sense-making process, allowing them to gauge the effects of different parameters and to gain more insight with each step of the humancomputer dialogue [66].
While the interactive steering of algorithms allows insight into massive data sets, such as social data streams [18] and geospatial data [4], there is a need for research concerning an adequate presentation of such analysis with regards to users inexperienced with visual analysis [14] or unfamiliar with the given visual representation [37].In the remainder of this paper, we refer to this group of users as non-expert users.
Despite the growing needs of non-expert users to control complex analytic processes, little work has been done to support them.Examples include researchers in the humanities who want to apply analysis techniques to large text corpora [26] and non-expert users making decisions about storm forecasts [45].Different visual representations might have different levels of "visual literacy" that impact the cognitive activities of users [9] [38].For instance, simple and intuitive visualizations may be easier to understand, but general ones might offer more detail and insight into the underlying data and processes at the cost of "representational compatibility" [55].Although much effort has been geared towards finding the most suitable representation of different types of data [64], researchers report challenges that non-expert users face when using these interactively in VA [23].To address this issue, we investigate the strengths and weaknesses of different representations to support non-expert users in decisionmaking.Our research questions are as follows: 1. Regarding objective factors such as accuracy and efficiency, how do the intuitive, detailed, and compact visualizations relate to the task difficulty? 2. Regarding subjective factors such as usability and uncertainty, what are the benefits and challenges of intuitive, detailed, and compact visualizations in an interactive decision-making process?
To answer these questions, we designed and evaluated an intuitive, a detailed and a compact visualization to support non-expert users with their decisionmaking under uncertainty.Using subjective and objective means, we provided a set of tasks to the participants and evaluated the three visualizations in two contexts in two separated studies.The study focuses on stock prices investment, using market indicators to support non-expert users.

Decision Support and Visualization
Decision support systems literature has modelled decision-making as an iterative process of problem recognition, perspective development, perspective synthesis, actions, and results [42].Visualization plays a crucial role to support the development of these multiple perspectives.
More specifically, visualization facilitates human decision-making by providing structured views about information pertaining to the decision-making problem space [51].Visualization is a well-established method to support decisionmaking in a wide variety of domains.Verbert et al. [62] showed that accuracy of recommender systems that suggest relevant items to a user increases with a set-based cluster map representation compared to a traditional ranked list of recommendations.Speier and Morris [57] showed that particularly when the solution set is large, visual query interfaces help users with greater effectiveness than text-based interfaces.Recent studies show how visualization affects and supports user decision-making in the context of financial services [6].
Decision-making usually relies on perception-based information [3].However, adequate interpretations for perceptions may not always be easy to find under uncertain situations, and thus defining models of this outstanding capability is a difficult, yet a highly promising research area [3].More specifically, visualization facilitates human decision-making by providing structured views about models related to the decision-making problem space [51].When making decisions in the real world, information visualization not only helps decision-making, but also offers a means of knowledge creation, as well as an appropriate communication channel [2].Several application areas have been using visual techniques, including health-care [1], supply chain management [43] and financial services [6].
However, there is a lack of knowledge about which visualization and interaction techniques work best for particular settings and particular users [20].Also, research has shown that the choice of representation of data influences decision-making outcomes [56].For instance, Gettinger et al. [13] found that different visual representations have an impact on the decision-making process when giving simple and complex tasks to the participants.We are interested in researching further the factors that influence decision-making, such as the effects of the visualization context, uncertainty representation and trust [27].

Visual Analytics
In recent years, the importance of Visual Analytics to support decision-making has grown [32].Visual analytics extends interaction of traditional information visualization techniques with facilities for updating, steering and improving the analytic processes.The key objective is to incorporate feedback from end-users to improve an automatic analysis process.
Examples of VA tools in decision making include the work of Senaratne et al. [49] that explores the use of a VA approach to tackle the inherent uncertainty in urban mobility patterns.Data collected from citizens' mobile communication is used to help to determine the urban dynamics of a city.Afterwards, city planners can use these tools to improve decision-making processes.Moreover, Höferlin et al. [22] introduced a VA approach to analyze video content supported by the communication of uncertainty generated from computer vision feature extraction to end-users.Goda and Song [15] presented a framework for visualizing Tsunami risks using VA techniques to support decision-making in emergency response capability.Their work emphasizes the importance of uncertainty modeling and visualization.The framework was developed through a case study for the 2011 Tohoku tsunami.Other examples include non-expert interpretations of hurracaine prediction [45], climate change data [19], and uncertainty in climate prediction [59].Visual analytics is also a useful tool to improve understanding of data in predictions [34].For instance, Ming et al. [41] introduced an interac-tive visualization technique to support non-expert users in machine-learning to explore and understand predictive models.
Several researchers have compared different visualizations of analysis outcomes and uncertainty associated to these outcomes to support decision-making for non-experts.Ibrekk and Morgan [25] compared nine different visualizations, including error bars, pie chart and density functions, to communicate uncertainty to users with no experience in statistical analysis.The authors suggest to use a combination of cumulative and probability distribution functions (PDF) to avoid misinterpretation.More recently, Greis et al. [16] compared different representations of uncertainty for non-expert users.Results indicate that even though a PDF plot is the best way to communicate uncertainty information, qualitative factors are also important to consider in representations for decision-making.Dynamic visualizations have been researched as well.Hullman et al. [24] compared the effects of an animated visualization against two static visualizations, and measured levels of difficulty by considering one and two variables in the model.Even though accuracy was similar in all representations for single-variable distributions, participants tended to be more accurate with their judgments when using the animated visualization with two variables.Tak et al. [58] compared seven visual representations of uncertainty for non-expert users, including solid and dashed borders, colour-band, gradient, thinning and random lines, and error bars.Results indicate that the visual representation of uncertainty affects the perceived certainty of participants.However, further exploration is required towards the use of other visual variables such as the use of colour and uncertainty range.
In this paper, we build on this earlier research and focus specifically on the design and evaluation of different types of visualizations (intuitive, compact, detailed ) for non-expert users.We extend the work of [58] by including the uncertainty range in different representations.As in work of [13], we consider different levels of problem complexity.We evaluate different aspects of the utility and effectiveness of different designs, including ease of use and visual appeal as identified by [16].We discuss these evaluation metrics in the next section.

Evaluation Metrics
Metrics to evaluate VA applications comprise a variety of criteria.However, definitive, generalizable guidelines do not appear to exist.Therefore, we analyzed previous work to define a set of objective and subjective metrics that may be considered useful in measuring an application's adequacy for supporting the decision-making process.Yang et al. [65] evaluated information graphics based on user preference and their capability to provide different levels of insight.While they could establish a link between information type and presentation, similar to Wongsuphasawat et al. [64], they did not investigate how well the representations would perform as part of an interactive analysis in a VA application.Mckenzie et al. [40] considered objective measures such as time spent and accuracy, whereas Bertini et al. [47] measured the actions in an uncertainty-aware interactive interface.Is the use of the visualization resulting in longer or shorter decision times?[40] Actions How many interactions are required to solve the tasks?[47] Subjective measures, usability-related Visual appeal The visualization was visually appealing.
[16] Ease of use The visualization was easy to use.
[16] Suitability The visualization was suitable for solving the tasks.
[50] Likability The preference towards the uncertainty visualization method. [50] Task difficulty Solving the tasks using the visualization was easy.

Trust
The visualization was trustworthy.
[30] Confidence I felt confident using the visualization.
[39] Understanding The visualization was easy to understand.
[16] Accuracy I find the accuracy of this visualization acceptable.[16] When comparing visual representations, Senaratne et al. [50] ranked visual representations towards their suitability according to the user domain.They explored metrics such as performance and likability, defined as the preference towards the uncertainty visualization method.Using a set of tasks, Lee et al. [36] used objective measures such as time spent and, as the work of Greis et al. [16], they used subjective measures such as ease of use, visual appeal and easiness to understand.To evaluate uncertainty factors, Kay et al. [31] proposed a questionnaire to evaluate acceptance of accuracy in visual analysis, considering ease of use and intent to use.[46] defined guidelines towards uncertainty propagation in VA systems, indicating that trustworthiness is a critical factor towards uncertainty awareness.Deitrick et al. [11] evaluated different uncertainty representations and their influence on the confidence of participants decisions.To conceptualize a framework for uncertainty, MacEachren et al. [39] defined credibility as an important subjective measure that has an impact on the user's decision-making and analysis.We compiled a list of objective and subjective measures in Table 1.

Study Design
In this study, we collected financial data and implemented intuitive, compact and detailed model representations to support financial decisions for non-expert users3 .

Design Motivation
As indicated above, there is a lack of knowledge about which visualization and interaction techniques work best for particular settings and particular users [20].Research has also shown that the choice of representation of data influences decision-making outcomes [56].We are interested in researching further the factors that influence decision-making, such as the effect of the complexity of the visualization, uncertainty representation and trust.Overall, our decision motivations follow Shneiderman's visual information-seeking mantra [52]: -With a detailed representation, we aim to provide a general context for understanding the predicted data.A detailed visualization addresses the representation of uncertainty considering the data used in the model.-A compact representation provides a close-up, allowing to examine the distribution of data in limited screen spaces and high data complexity situations, as viewing every detail might be impractical.This visualization provides focused insight into a particular risk, clearly showing its spread and uncertainty.Yet, providing detail for a single factor comes at the price of depicting relationships between multiple factors.-An intuitive representation is a visualization that is commonly used and familiar by the end-user to resolve tasks.These representations might include a minimal depiction of uncertainty and might not explain a model in detail.
A typical scenario for decision-making is the investment in stocks.Such a task is subject to uncertainty due to many variables that are involved.We used data related to stock prices to compare different model representations that support users in their decision-making while investing in stocks.
What and When: intuitive representation (time series).Investing capital in stocks requires to see the behavior of the company stock price in the past.Typically, stock prices fluctuate a lot over time, partly depending on different external factors, such as worldwide news, oil price, and world economy.For the intuitive representation, we asked a financial expert about typical time series visualization.Based on this expert feedback, we designed a time series visualization showing the last year historic fluctuation of the stock value and a next year forecast.The time-series visualization is suited for answering simple questions concerning aspects of what and when.
Why: detailed representation (scatter plot).A detailed visualization of the prediction model allows users to investigate why and how a prediction is made.By inspecting the variables in the model, users can identify possible sources of uncertainty.Differences between the distributions of observations can be depicted and users can see with high precision how data is being correlated [28].As the approach requires a certain degree in graph literacy, it is important to investigate when and how such a representation is appropriate to support decision-making for non-expert users.
How: compact representation (dot plots).The compact representation used probability distribution functions (PDF), as also used by [25] and [29].The visualization focuses on showing a clear representation of future events to happen showing discrete outcomes.The dot plot is less detailed than the scatter plot representation, as it does not represent the observations and model.However, it does show a clear representation of probability outcomes.The most likely predicted value was shown above the mean of the distribution, while the predicted value was displayed below the visualization.
The visualization of uncertainty is important to correctly interpret an algorithm's prediction [53].In our earlier work [48], we investigated quantitative and qualitative performance of different visual variables, such as opacity, blur, contour lines, grid size, and texture, for representing uncertainty.We found that overall opacity performed best, followed by texture.Therefore, we chose a change in opacity in the three visualizations for depicting uncertainty.

Application Design
We created a set of visualizations to represent the predicted stock price for different companies.A different visualization was used for each of the three study applications resulting in three identical applications, one with a set of six intuitive visualizations, a second with a set of six detailed visualizations and a third with a set of six compact visualizations.

Data
A typical scenario for decision-making is the investment in stocks.Such a task is subject to uncertainty due to many of the variables that are involved, (e.g., consumer trust, worldwide news, economic growth, Dow Jones index).We used stock prices data to compare different model representations that support nonexpert users in their decision-making process while investing in stocks.
To simulate a real scenario we asked for advice from the financial expert.Based on experience, the financial expert selected six companies from the Dow Jones share that had a similar behaviour in their stock prices: Disney (NYSE: DIS), Home Depot (NYSE: HD), Apple (NASDAQ: AAPL), Chevron (NYSE: CVX), IBM (NYSE: IBM), and McDonalds (NYSE: MCD).We used the Quandl API [44] to get the current and one-year historical records of stocks from the selected companies.With the advice of the financial expert, we collected tweets mentioning the selected companies for topics such as worldwide news, and consumer trust that might serve as an indicator of the stock market [7].We aimed for tweets containing the following hashtags and keywords: "news", "#news", "trust", "confident", "confidence".Then, we used the sentiment analysis API [54], to assign a score from negative (0) to positive (1).

Model
To estimate the stock price, we built a simple regression model: where CI is a compound indicator, β 0 is the slope of the line and β 1 is the Stock Price intercept.CI is based on the average normalized values of the market indicators: consumer trust, worldwide news, economic growth, and the Dow Jones index.This allowed users to adjust the market indicators using sliders in the user interface, updating the predictions in the visualization section.Note that we present this simplified forecasting model solely for the purpose of demonstration.

Interaction
We divided the application into three main sections: settings, visuals and questions, see Figure 1.The settings section used the following market indicators: consumer trust, worldwide news, economic growth, and Dow Jones index.The application presented the market indicators as checkboxes with sliders, see Figure 2 (left).Selecting a checkbox determines whether the model considers the market indicator as a variable in the prediction.Each slider enables the users to set the level of importance of the variable in the model, according to their beliefs.
When the user selected variables or changed their importance with sliders, the visualization outcome was updated instantly, see Figure 1b.Next, the questions section, see Figure 1c, shows a set of easy-to-hard questions reviewed by the financial expert.The users had to interact with a combination of variable settings to respond to all the questions.

Visualization
The intuitive, compact and detailed visualizations were separated in three different applications, see Figure 2.Each of the visualizations represented stock price predictions of the same six companies.In this visualization, the stock prices predictions were shown using time-series.We implemented a line chart to show the variation of the stock price divided in two sections: historical and forecast.The historical section depicted the stock prices from the past 12 months as historical data.The forecast section showed the expected predicted value for the next 12 months.We also showed the value of the last stock price for the historical data, the value of the fore-casted stock price and the percentage difference between these two values, coloured by red (negative) or green (positive).In addition, we showed the prediction interval for such prediction.We used an opacity scale that indicated the likelihood for the predicted value from unlikely (light grey) to likely (dark grey).In other words, the opacity scale highlighted the quantiles of the predicted probability distribution and showed the range of accuracy for the value.The visualization was updated immediately after when the user adjusted the importance of different variables in the settings section.
Figure 2b: detailed visualization (scatterplots).The visualization consisted of the plot of the regression model together with the observations of trust, worldwide news, economic growth and the Dow Jones index.The spread of stock prices of a variable was mapped to the X-axis, the corresponding distribution of the variables describing the economic factors to the Y-axis and then plotted into the coordinate system.Different colours were used to distinguish between the variables.The predicted value for the stock price was written next to the visualization at the Y-axis coordinate of the corresponding point of the regression line.
Figure 2c: compact visualization (dot plots).We used opacity in the dots to represent the accuracy in the forecast.The most likely predicted stock value was shown above the graph.The percent differences were displayed below.Colour was used to depict increase (green) or decrease (red) of the stock price in all visualizations.

Study Formulation
The financial expert analyzed the data, the model and the visualizations to create a set of 20 candidate questions for the main study.We conducted a preliminary study in Amazon Mechanical Turk (AMT), where we evaluated the 20 questions with ten participants (10 males) with no expertise with data analysis or visualization.Based on the number of actions required by the participants to answer the questions, we ranked them and selected the top 12 questions.After, we organized the questions into three different categories (easy, medium and hard).

Main study
We recruited 360 participants on AMT, 120 per application.To qualify for the study, users had to indicate that they had no professional experience in data analysis and that they were reasonably proficient in the English language.Also, participants were required to have a minimum approval rate of 95%.To catch "gaming" users that simply clicked through the questions, we included five "gold standard" questions randomly into our task list [63].These were simple questions, such as "What is 1 + 2?", which used similar controls as the "normal" tasks.In addition to these, we added three training questions to familiarize users with the application.We deemed this necessary as a pilot study showed low performance at the beginning with a learning effect.Each user could only participate once and received a minimum payment of $0.50.To motivate users to try harder, we offered bonuses.For answering at least 66% of questions correctly, users received $2.50, for more than 80%, they received $3.50.After a short tutorial, the application presented the training and study questions.The average time of the study for each participant was 21 minutes.At the start of each task, the visualization and controls were hidden, with only the question and a "start" button being visible in the middle of the screen.Upon clicking "start", the visualization, controls, and questions became visible.Recording started when users clicked "start" and ended when they clicked "submit".The following properties were recorded: time spent, number of actions, and the answer given.After users had completed all tasks, they were invited to answer questions regarding usability: visual appeal, ease of use, suitability, likability, ease of tasks and questions related to uncertainty: trust, confidence, credibility, understanding, accuracy representation, presented in Table 1.Answers were given on a five-point Likert scale, from strongly disagree (1) to strongly agree (5).Users who had failed more than one of the gold questions, had less than three correct answers or did not finish the evaluation were removed, leaving the groups presented in Table 2.

Results
We ranked the answers of the participants by their accuracy and grouped them in three levels of difficulty, see Table 3.Based on this ranking, we assigned easy, medium and hard levels of difficulty to each question.Also, to further understand the usage of the interface and answer our research questions, we calculated the participants efficiency.Frøkjaer et al. [12] defined efficiency as the relation between the accuracy with which users achieve certain goals and the resources expended in achieving them.
The evaluation of VA applications is a big challenge in the visual analytics field [61].So far, evaluation guidelines are not well established.However, accuracy and efficiency are amongst the most accepted criteria [46].Moreover, VA studies have used efficiency in the past to measure human-computer interaction techniques in the decision-making process.As such, efficiency is a relevant measurement in our study, as it can tell us how the visual representation is used to efficiently communicate the outcome to the end-user [33].We define efficiency as follows: where N is the total number of tasks; R is the number of users; a i,j is the accuracy of task i by user j; r i,j is the amount of resources expended by the user j to solve task i. Resource is defined as the actions per minute performed by the user.
Figure 3 shows the mean values for accuracy and efficiency, using 95% CI, and sorted by level of difficulty.Factorial ANOVA tests were conducted to compare the accuracy of participants, and their efficiency with the visualizations and levels of difficulty.As a follow up to the findings, a post-hoc Tukey HSD test was used.Also, we report the results in Table 4.
Accuracy: participant's accuracy was significantly different for all levels of difficulty (p < 0.001, η 2 p = 0.23).In general, participants tended to be significantly (p < .01,η 2 p = 0.018) less accurate while using the compact visualization.No other significant differences were found.
A Kruskal Wallis U non-parametric test was used to find a difference in the responses related to usability and uncertainty.Mann-Whitney tests were used as a follow-up in the findings.-Visual appeal: no significant differences were found.Participants preferred the intuitive (M dn = 5) visualization, followed by the compact (M dn = 4) and the detailed (M dn = 4) visualizations.-Ease of use: participants had a significant preference (p < .05)towards the intuitive (M dn = 5) visualization, followed by the detailed (M dn = 5) and compact (M dn = 4) visualizations.-Suitability: participants had a significant preference (p < .001)towards the intuitive (M dn = 5) and detailed (M dn = 5) visualizations (p < .01),followed by the compact (M dn = 4) visualization.-Likability: no significant differences were found.Participants preferred the intuitive (M dn = 5) and compact (M dn = 5) visualizations, followed by the detailed (M dn = 4) visualization.-Tasks were easy: participants had a significant preference (p < .001)towards the intuitive (M dn = 5) and the detailed (M dn = 4) visualizations (p < .01),followed by the compact (M dn = 3) visualization.-Trust: participants had a significant preference (p < .001)towards the intuitive (M dn = 4) visualization, followed by the compact (M dn = 4) and the detailed (M dn = 3) visualizations.-Confidence: participants had a significant preference (p < .001)towards the intuitive (M dn = 5) and the detailed (M dn = 4.5) visualizations (p < .001),followed by the compact (M dn = 3) visualization.-Credibility: no significant differences were found.Participants showed a similar preference towards the intuitive, detailed and compact (M dn = 4) visualizations.-Understanding: participants had a significant preference (p < .001)towards the intuitive (M dn = 5) visualization, followed by the compact (M dn = 4) and the detailed (M dn = 4) visualizations.-Uncertainty: participants had a significant preference (p < .001)towards the intuitive (M dn = 5), followed by the detailed (M dn = 4) and compact (M dn = 4) representations.
5 Answering the research questions RQ1: Regarding objective factors such as accuracy and efficiency, how do the intuitive, detailed, and compact visualizations relate to the task difficulty?
The selection of a suitable visualization for the task at hand has a significant impact on evaluation results.Results indicate that an intuitive visualization works well for easy and medium-level tasks where the participants were the most accurate and efficient.In both studies, participants that used the detailed visualization tended to be more accurate as the level of difficulty increased, particularly for hard tasks.Participants tended to be more accurate and more efficient while using this representation, particularly for hard tasks.The compact visualization was in general the least accurate and efficient.
RQ2: Regarding subjective factors such as usability and uncertainty, what are the benefits and challenges of intuitive, detailed, and compact visualizations in an interactive decision-making process?
The intuitive visualization scored best on most usability and uncertainty factors.These results indicate that the careful selection of an easy to use and suitable visualization has a strong impact towards uncertainty factors such as trust, understanding and confidence.Participants also indicated that the tasks seemed easy to solve and the accuracy representation was good.Participants that used the detailed visualization indicated the visualization was suitable for solving the tasks and tended to show a positive attitude towards the uncertainty representation.Although the compact visualization was rated good for factors such as understanding, accuracy representation and likability, further exploration is required on factors such as trust, confidence, and task difficulty.

Recommendations for Implementation
Visualization for non-expert users requires careful measurement of qualitative and quantitative analysis of behaviour to solve a problem.Modern approaches such as immerse analytics demand that the choice of the visualization should be progressively adapted to the complexity of the task [8].Communication of uncertainty is essential in the decision-making and should be an integral part of visualization.Based on our findings, we give the following recommendations.
-Detailed visualization: we recommend the use of this visualization as the level of difficulty increases, mainly when looking for efficiency and accuracy on hard level tasks.However, one of the potential drawbacks is the responsiveness of the visualization on small screens.A detailed visualization would shine in larger spaces, where users can manipulate the variables and inspect the details as the information is updated.-Intuitive visualization: the visualization scored best on most usability and uncertainty aspects and was highly accurate for easy and medium level tasks.Participants tended to be more accurate and efficient with the intuitive representation of easy and medium level tasks.-Compact visualization: we recommend being cautious using a compact visualization when accuracy is a critical factor for medium and hard tasks.
A compact visualization has potential because it can show a deep insight towards explaining uncertainty.According to participant's feedback, the visualization was credible and understandable, but they were not sure when asked about their trust or confidence.Further research is required about the effects of a compact visualization in the decision-making for VA applications.

Conclusion
Exploring the challenges of VA for non-experts is a growing research field [35].While advice on mapping types of data to an ideal representation does exist [64], information about different visualizations' qualitative and quantitative performance in an interactive decision-making process for non-experts does not.To address this research gap, we have examined the suitability of intuitive, compact, and detailed visualizations in the risk-based decision-making process for non-experts in an application facilitating the exploration of variables involved in a prediction model.By allowing the users to interactively change the importance of different variables, users could steer an algorithm predicting these aspects from a dataset.Using objective and subjective means proposed by earlier work, we determined the benefits and trade-offs of different uncertainty representations using tasks with different difficulty levels.We found that intuitive visualizations can be a good choice for easy and medium level tasks and that a more demanding visualization may be required for hard level tasks, potentially at the cost of usability aspects.Our research extends previous work on the challenges of supporting non-experts with VA: Kwon et al. [35] reported a set of "roadblocks" inhibiting non-experts from successfully using popular VA applications.As these are commonly built for experts, their utility can be "unreachable" to non-experts due to their limited domain experience, low graph literacy, and inaccurate mental model.Some applications of our work include the education domain where visualization of prediction outcomes plays a critical role to support the decision-making process of academic advisers [17].Moreover, in the employment domain, interactive VA tools can be useful to visualize prediction and recommendation output and support a dialogue between job mediator and job seeker [10].Here VA tools increase explainability, providing mediators with control over the information that is presented to job seekers.
Therefore, if we want to extend the user-base of VA to non-experts, we may have to take a step back from these specialized and robust applications.Instead, this endeavour may call for a new generation of closely tied applications to a user's abilities, preferences, and the difficulty of the problem to be solved.Huang et al. [23] explored the challenges of PVA and suggested that an application that is to be successful in this field needs to consider the user's context and provide "appropriate baselines to support reasoning about data".Our work presents a step towards providing such a baseline: defining user context as the difficulty of the problem to be solved or level of insight to be gained, we have evalu-ated the adequacy of a certain visualization for supporting the VA process of non-experts.Following Shneiderman's visual information seeking mantra [52] of increasing information density as required, we suggest increasing the complexity and capability of a model's representation as the task difficulty and the required level of insight increases.
Regarding the relationship between accuracy and uncertainty, we observed an increment of accuracy with the intuitive visualization for medium and easy tasks where participants also indicated an increase of trust and confidence.This was followed by the detailed representation where participants where significantly more accurate for hard tasks and they reported to have confidence with the visualization.However, with the compact representation, participants where less accurate and gave the lowest scores regarding uncertainty factors.
Our results and those of Kwon et al. [35] have indicated that confronting nonexperts with a powerful, but unnecessary complex visualization is detrimental to solving simple problems.Instead, our results indicate that by adapting the visualization to the difficulty of the task, a favourable balance may be struck between various objective and subjective aspects and the decision-making process optimally supported.By defining strengths and weaknesses of three visualization types for problems of varying difficulty, we provided a step towards the definition of a set of visualizations that may be chosen to support non-expert users.

Limitations
We used a crowd-sourcing platform to gather data.Previous work has shown the suitability of AMT for conducting studies in the visualization domain [21,65].Even though our interface was simple, guiding the users with questions to facilitate evaluation, and only four control variables were used to simplify interaction, results should be interpreted with caution, as studies were unsupervised.Moreover, we have detailed the accuracy results per difficulty level, however, the subjective data related to perceived uncertainty is not captured individually for the different difficulty levels.Also, while we were able to identify the benefits and challenges of the visualizations, the effect sizes are relatively small.Future work will investigate further the perceived uncertainty and examine performance in a purely exploratory environment.We will also investigate whether the findings can be extended to other domains.

Fig. 1 .
Fig. 1.Layout used in the applications.a) The settings section shows sliders for adjusting the variables.b) Visualization section where predictions are shown.c) Questions section that users have to answer together with the visuals.

Fig. 2 .
Fig. 2. The three different applications used in the financial decisions study: a) intuitive, b) detailed, and c) compact.

Figure 2a :
Figure2a: intuitive visualization (time-series).In this visualization, the stock prices predictions were shown using time-series.We implemented a line chart to show the variation of the stock price divided in two sections: historical and forecast.The historical section depicted the stock prices from the past 12 months as historical data.The forecast section showed the expected predicted value for the next 12 months.We also showed the value of the last stock price for the historical data, the value of the fore-casted stock price and the percentage difference between these two values, coloured by red (negative) or green (positive).In addition, we showed the prediction interval for such prediction.We used an opacity scale that indicated the likelihood for the predicted value from unlikely (light grey) to likely (dark grey).In other words, the opacity scale highlighted the quantiles of the predicted probability distribution and showed the range of accuracy for the value.The visualization was updated immediately after when the user adjusted the importance of different variables in the settings section.Figure2b: detailed visualization (scatterplots).The visualization consisted of the plot of the regression model together with the observations of trust, worldwide news, economic growth and the Dow Jones index.The spread of stock prices of a variable was mapped to the X-axis, the corresponding distribution of the variables describing the economic factors to the Y-axis and then plotted into the coordinate system.Different colours were used to distinguish between the variables.The predicted value for the stock price was written next to the visualization at the Y-axis coordinate of the corresponding point of the regression line.Figure2c: compact visualization (dot plots).We used opacity in the dots to represent the accuracy in the forecast.The most likely predicted stock value was shown above the graph.The percent differences were displayed below.Colour was used to depict increase (green) or decrease (red) of the stock price in all visualizations.

Fig. 3 .
Fig. 3. Results regarding actions, time spent, actions per minute, accuracy and efficiency, grouped by easy, medium, and hard levels of difficulty.

Table 1 .
List of objective and subjective measures considered.

Table 2 .
Final groups of participants from the user study

Table 3 .
Questions by level of difficulty.We show the mean accuracy from all participants.
You have very precise expectations, consumer trust at 39%, worldwide news at 68%, economic growth at 38% and the Dow Jones index at 62%.The (...) stock is expected to gain more than (...) but less than (...).0.74 hard You expect the economic growth to be moderate (50%).How does the Dow Jones index have to be, so that the growth of apple is exactly the opposite of the growth of IBM?0.68Consumer trust is moderately bad (25%), economic growth and the Dow Jones index are good (75%).What is the value for the worldwide news when the home depot's stock is $129.99?

Table 4 .
Results of the study.Objective measures are presented with the mean, subjective measures are presented with their mean and median.Best-scoring values are highlighted in bold.