Designing Natural User Interfaces Scenarios for All and for Some: An Analysis Informed by Organizational Semiotics Artifacts

. The design of Natural User Interface (NUI) technologies is still in its early stages; therefore, it does not have well-established guidelines, especially ones that consider the context of Accessibility. This increases the challenges for designers of these technologies to achieve products fulfilling their purpose. . In this paper we present a research project that aims at exploring NUI devices within the Accessibility context, with the goal of proposing ways to promote a better design for NUI technologies. We present this project from an Organizational Semiotics perspective, so that the context we aim to focus on shows itself clearly during the entire design process. Our ultimate goal is to promote better NUI designs, especially for people with disabilities, supporting their autonomy and inclusion in society.


Introduction
Known as Natural User Interfaces (NUIs), this interface paradigm in Human-Computer Interaction (HCI) encompasses devices and technologies that, as the term "natural" implies, only require users to do what comes naturally to them, instead of having to develop technical skills to be able to interact with the interface.This, as [1] argue, can both attract users who do not feel comfortable with the traditional mouse and keyboard interfaces, and elicit new design and engineering challenges that will inspire research in several different knowledge areas.However, to become actual usable products, these ideas need to consider, from the very beginning, the context in which the new technology will be used and who its potential stakeholders are.Organizational Semiotics (OS) provides tools with the potential for this analysis of organization and context, but given the novelty of NUIs there is the additional challenge derived from considerations of Accessibility and users with disabilities.Although there are some initiatives in this direction, be it in the form of general NUI guidelines [2] or in the form of design and evaluation heuristics for NUIs [3], creating new tech-nologies with NUI that are both useful and usable by a vast diversity of users' capacities is still a challenge.
The work described and discussed in this paper is part of a research project that aims at exploring existing NUI devices and testing both their own Accessibility (i.e., its interactability by people with disabilities) and their usage in the context of facilitating everyday actions of people with disabilities.The latter implies in either employing these devices as they are or by altering them or combining them with other devices.This project has two main goals.The first is to propose a conceptual framework for the design of NUI devices that consider the context of Accessibility.The second objective is to design and engineer new devices that not only help people with disabilities, but also that meet the requirements of Universal Design (UD).Once we achieve these goals, products may help in the design of new NUIs that encompass as many users as possible.Therefore, we aim at designing NUI-based technologies with potential of being more inclusive for people with disabilities and more interesting and useful for people without disabilities.
We believe that Organizational Semiotics (OS) can help to achieve these goals by providing a contextualized view of the problem we are dealing with.In this work we use the OS artifacts to clarify and represent our research project, showing how they allow pointing either in the direction of UD or of Assistive Technologies (AT).On one hand, UD means creating products that are usable by anyone, regardless of features such as age, culture, language or disability [4].In Computing, there is a related term, "Universal Access", which refers to ensuring that all people have access to technology and information, and that these computing services are usable by anyone [5].Although UD might seem very hard or even impossible to achieve, it should at least inspire designers to create better solutions [6].Therefore, we believe UD is a design goal that should guide the design process from the very beginning of a computing project.On the other hand, AT refers to devices or computing systems created to compensate, relieve or neutralize body impairments [7].This encompasses assisting people with disabilities in the execution of tasks and, in turn, improving their social participation and autonomy.Despite their potential benefits, ATs commonly suffer from abandonment by their users, usually because of difficulties related to adapting to the AT or in learning how to use it.To overcome these problems, it is important to involve the stakeholders in the design process, especially those closest to the users, such as family, friends, caretakers, doctors and nurses [7].
Looking at the two concepts of UD and AT it is possible to see they contrast in how they treat their target users: while UD tries to tend to as many users as possible, ATs focus on helping specific users perform specific tasks.However, they have in common the fact that they both need a deeper consideration of context and both can benefit from SO artifacts.In this paper, we will show how different uses of the artifacts can best promote either AT or UD.
The paper is organized as follows: In Section 2 we give an overview of the theory behind the OS methodology we employed.Then, in Section 3 we present the OS artifacts and the practical scenarios instantiated from them.Section 4 discusses how the different problem-solving approaches taken in each scenario reflect on the SO artifacts and, ultimately, in the final solution.Finally, Section 5 presents our concluding remarks and overview of future work.

Theoretical Basis
The Organizational Semiotics (OS) proposes a comprehensive study of organizations in different levels of abstraction (informal, formal and technical), and their interdependencies.For OS, organizations can themselves be information systems with norms and patterns of well-defined behaviors that regulate the internal processes within the organization.In this sense, an organization is composed of three layers [8]: informal (outer layer), formal (middle layer), and technical (inner layer).The premise behind the use of OS in information systems is understanding the situated context of the organization that the system will be inserted in, and clarifying the main forces that act on it, to propose a technical information system that makes sense for these organizations.Understanding organizational functions from the social level is essential for achieving this goal [8].To enable a better understanding, development, management and use of information systems, a set of methods known as MEASUR (Methods for Eliciting, Analyzing and Specifying User's Requirements) was developed in the OS [9].In this paper, we make use of some of these methods aiming at clarifying the problem and proposing solutions, which can have an impact on the design of both Universal Design and Assistive Technologies.For this, we use three artefacts, two of which are from the OS: the Stakeholders Identification Diagram (SID) and the Semiotic Framework (SF).The third one is the Evaluation Frame (EF), used to discuss problems the stakeholders may find and anticipate solutions to them [10].We briefly describe them as follows, and in the next sections, we discuss their instantiations in our research project.
The SID [11] facilitates the identification of those involved in a particular design process.SID pays attention to different levels of involvement, interests, and expectations, allowing the visualization of stakeholders and their organizations inside five different categories: Operation, Contribution, Source, Market, and Community.In turn, the EF is intended to support reasoning about problems and solutions related to each stakeholder identified through the SID.Therefore, it favors the clarification and identification of requirements as well as the anticipation of issues that may impact/influence the solution to be designed.EF is represented in a table format where the columns contain problems and solutions, and each line references one of the five SID categories.The idea is to raise, in each of these layers, the identified problems and solutions for each group of stakeholders.
Finally, the SF [9] favors the identification and organization of requirements according to six different levels that represent different aspects of signs.The first three levels can be related to technological issues (the Physical, Empirical, and Syntactic), and the other three levels can be related to aspects of human information functions (Semantic, Pragmatic and Social World).The Physical World indicates the features and signs that can be measured by physical analysis and engineering.Empiric studies the properties of the signs.Syntactic analyzes the relationship between signs (whether formal or structural).Semantic describes the relationship between a sign and their meanings.Pragmatics studies the relationship between a sign and the behavior of the involved agents.Finally, the Social World evokes the need to understand how the rules of interactions between the groups work.The SID and EF were used to clarify the scenarios in which we experienced NUI technologies, while the SF supported the organization and specification of requirements and design decisions to be made.Therefore, these three artifacts contributed with a perception of the problem domain and its possible solutions.

The Instantiated Artifacts and Case Studies
Although the term Natural User Interface (NUI) has gained power after the advent of innovative devices [1] such as the Microsoft ® Kinect [12], NUI-based devices can be created using any kind of input modality, as long as the experience feels natural and the technology best reflects the abilities of its users [2].This means that it is possible to create a Natural User Interface with a combination of older technologies (like mouse and keyboard) with newer ones (like gestural or touch), because the potential for naturalness is in those technologies, but not the guarantee of it.This trade-off between potential and guarantee creates excellent design and research opportunities, since the possible combinations of input and output devices for the creation of new NUIs are countless.However, it also elevates the challenges behind the task, since employing a technology that is recognizably in the NUI paradigm does not give certainty that the resulting device will actually provide a natural experience for the user.Part of the challenge also lies in understanding the NUI-based devices' potential for users and their context of use.This means taking into account different user needs, based on the characteristics of the users themselves and on where they would utilize the technology.In our research project, we look at this problem from the perspective of people with disabilities.We want to investigate how NUIs can help these users in several ways, such as in accomplishing daily tasks, gaining autonomy, being included in society and interacting with other people.
The first step to understand the problem is to look at the stakeholders of the NUI technologies and devices we aim to explore and design.This is where the SID comes in.In the inner layer, Operation, we have the users of the NUI devices, and, as we move to the outer layers, we define other stakeholders such as researchers and developers (Contribution) the users' families and friends (Source and Community), technology companies (Source and Market), Government and society (Community).In the EF, this brings out questions like "How does the device affect user's interactions and relationships with friends and family?" or "How does Government regulation adapt to the new devices?".Furthermore, although we guide our project with the principles of Universal Design [4], we are also trying to understand how to better design NUI-based devices for people with disabilities.Therefore, some of our stakeholders are specific to this audience, such as the industry of accessibility materials and associ-ations for people with disabilities.In our EF, this leads to questions like "How do the associations for people with disabilities benefit from the devices and technologies created?", which we answer as "they have early access to prototypes and, later, can use the finished final products".Therefore, the SID and the EF give us a contextualized view of our research problem, helping us anticipate problems and solutions related to each stakeholder.To explore the artifacts even further, we have created from them three different case studies.The following subsections describe these case studies and explain how they contribute to our research problem.

Microsoft ® Kinect and the Visually Impaired
In the first case study, we started from two stakeholders in our SID: "people with disabilities" from the Operation layer, and "technology companies" from the Source layer.We instantiated each as "people with visual impairments" and "Microsoft", or more specifically, its NUI device, the Kinect [12].From there, we focused on the challenge of using the Kinect to help visually impaired people (i.e., the blind or people with low vision) in their daily tasks, such as navigation and recognizing objects, informative signs and people.The Kinect is a device composed of 3D depth sensors, an RGB camera and a microphone.Therefore, the main goal of this study case is to figure out how to translate the visual information input given by the Kinect into an output the target users can easily understand.For this, we chose 3D audio because of its capability of carrying information, such as the location of an object in relation to the user, in an indirect way.To test the applicability of this solution, we built a prototype, uniting the Kinect, bone conduction headphones and algorithms of Computer Vision (to interpret the visual input) and 3D Audio (to generate the auditory output).This prototype allowed us to conduct laboratory experiments with users and figure out technical, ergonomic and usability issues.Since until now we only have conducted experiments in controlled environments, so far we have tried to answer questions from the Operation and Contribution layers of the EF, such as "How efficient is the operation?","How comfortable does the user feel using the device?" and "How does the user benefit from the device?".Once the experiments reach real-world tests, we will also be able to get feedback on the questions and problems presented in the other layers, like "How can friends and family help the user to configure or learn how to use the device?",taken from the Source layer.Therefore, the main contributions of this case study to our research problem are the insights into the building process of a NUI-based device, especially when the starting point is an already existing NUI technology that needs to be adapted.Additionally, we will also continue to see how the analysis we made with the SID and the EF applies in this instantiation of our research problem.

Samsung ® Galaxy Gear and the Visually Impaired
In the second case study, we again started from two stakeholders in our SID: "people with disabilities" from the Operation layer, and "technology companies" from the Source layer.We instantiated each as "people with visual impairments" and "Sam-sung", or more specifically, one of its NUI devices, the first generation Galaxy Gear [13].From there, we focused on the challenge of using this device to help visually impaired people with the task of recognizing people in their surroundings.The Galaxy Gear is a smart wristwatch (or smartwatch) that has an 800 MHz processor, 512MB RAM, 4GB of internal memory, 2 microphones, a speaker, Bluetooth capabilities and a 1.9 Megapixel camera on the wristband.It can also communicate with the user's smartphone to execute tasks such as answering calls and reading messages.Therefore, the main goal of this case study is to figure out whether and how a wearable device such as the smartwatch can help the visually impaired with the task of recognizing people around them.This involves not only developing Computer Vision algorithms that are able to run on a device with limited hardware capabilities, but also figuring out the best ways to provide feedback to users in ways they can easily understand.Additionally, the feedback cannot overwhelm or embarrass the user.We have conducted experiments with users within laboratory conditions and found issues related to software, ergonomics and feedback.Therefore, similar to the first case study we have so far tried to answer questions from the Operation and Contribution layers of our EF.Once we carry on to real-world tests, we will be able to answer questions from the other layers, such as "How does the device affect interactions or relationships between the users and their families or friends?" (taken from the Source layer), or "How does the new device impact on NUI devices companies?"(taken from the Market layer).Additionally, in this case study we are again exploring a NUI technology (smartwatch) within the context of a specific group of users (the visually impaired); however, in this scenario we are not adapting the device on the hardware level so far, but on the software level.Therefore, the main contributions of this case study to our research problem are the insights into adapting, on the software level, a NUI wearable device to perform tasks it did not originally fulfill (recognizing people with the camera).Furthermore, we are also able to see how the analysis made with our SID and EF will continue to apply in a concrete instantiation of our research.

Web of Things in the Supermarket
In the third and last case study, we started from several stakeholders in our SID: "people with disabilities" and "other users" (Operation and Contribution), "technology companies" (Source), "NUI devices companies" (Market), "interested society" (Community) and "Academia" (Community).Their instantiations would be, on one hand, any person interested in receiving help with the task of selecting and finding items in a supermarket ("people with disabilities", "other users" and "interested society"); on the other hand, we have those involved in the area of the Web of Things [14] ("technology companies", "NUI devices companies" and "Academia").Then, we focused on the challenge of using the Web of Things concepts to help any kind of customer to find and select items in a supermarket.
The Web of Things (WoT) is a research field derived from another field called Internet of Things (IoT).On one hand, the IoT is concerned with the transition of the Internet from a network of computers to a network of trillions of smart "things", such as mobile devices, home appliances and sensors.On the other hand, the WoT revolves around reusing and adapting technologies and protocols that exist in the current Web to build applications that will run in the IoT.Hence, this case study has the goal of using the network of sensors, smartphones and other "things" that may exist into the supermarket to help people in the tasks of finding and choosing products in the establishment.Additionally, the case study also encompasses providing ways to use the WoT to, direct or indirectly, make users help each other.This means providing functionalities that will allow, for instance, people without disabilities to give information that may help people with disabilities, such as product reviews and translation or transcription of information presented on the packing.Notice that this can also be useful to other types of users, such as foreigners, elderly and people who are uncertain about the quality of a product.
After coming up with the general idea of the case study, we looked at the possible technologies that could be used to create the device we are aiming at.We decided an RFID (Radio-Frequency IDentification) reader, some RFID tags and a text-to-speech software were enough for a first experiment.Despite the controlled conditions of our simulated supermarket, we discovered important issues related to the sound feedback, especially regarding the semantics and the syntactic structure of the information given to the user.More specifically, several users could not comprehend the directions to find the sections of the supermarket, and others had trouble understanding reviews of products left by other users.These issues are direct reflections of questions from our EF, such as "How efficient is the operation?"(Operation layer) and "How do users report problems?"(Contribution).Once we move on our experiments to noncontrolled environments, we will be able to answer questions from other layers, like "How do supermarkets benefit from the device?"(Market layer) and "How can academia benefit from the device?"(Community layer).Therefore, the main contributions of this case study to our research problem are the insights into creating a device that, from the start, is aimed at any user and that helps people with disabilities.Additionally, also gain perspective on how our analysis made with the SID and the EF work on this instantiation of our research problem.

Results and Discussion
Each of the case studies described in the previous section was informed by analysis on the same SID and EF.However, while the first two scenarios adopted the approach of starting the design from an existing NUI technology, the third one started from the problem and looked for technological solutions to it.These two different approaches (and their consequences) can be evidenced in the Semiotic Framework (Fig. 1) If we think about the organizational onion [8], the first two case studies have taken the direction that goes from the technical layer to the informal layer, while the third case study started in the informal layer and went to the technical layer.Looking at this in the SF, the first two case studies started in the most bottom step, the Physical World, by defining the technologies they would use in their designs (Kinect or smartwatch) and moved to the top step, the Social World.In turn, the third case study went the opposite way, by defining the problem and the concern for Universal Design in the top step, and then moving down to reach the Physical World.Hence, from now on in this text we will refer to the first approach as "bottom-up" and to the second as "top-down".If we make a more detailed analysis, we can see the impact each approach has in each level of the SF (Fig. 1).The bottom-up approach starts in the Physical World by choosing a NUI device and combining it with the necessary auxiliary technologies, such as white cane for the blind user, headphones to receive audio output and, lastly, the multichannel communication needs to be considered.In the case of the Kinect, hardware and software modifications were made to implement one of the channels: the audio.In the case of the smartwatch, only software adaptations were necessary since the device's hardware already has multichannel capabilities.Moving on to the Empiric layer, the efficiency of the audio feedback is tested and adjusted, usually by software.Then, in the Syntactic layer, the pattern and format of the feedback are defined and possibly require more adjustments.In the Semantic step, we consider how much the audio cues are understandable to the users, and possibly make more adjustments.In the Pragmatic layer, we see how much the device as a whole is actually helping the user in the execution of a task.Finally, in the Social World step we consider issues related to embarrassment and segregation.In contrast, the top-down approach starts by defining the problem (finding and selecting products on a supermarket) and committing to Universal Design, i.e., helping as many types of users as possible in the accomplishment of the chosen tasks.Moving to the Pragmatic layer, we think about how to provide communications between users, and how to make these communications useful for them and compatible with the proposed problem.Then, in the Semantic step, we design the input (commands) and output (feedback) messages so that they are understandable by as many users as possible.
In the Syntactic layer this reflects upon the format we will choose for the messages, the types of senses (vision, tact, hearing…) we will choose to reach and which databases we will use.This also carries on to the Empiric layer, where we consider which communication channels to use and how efficient each one is.Finally, in the Physical World we actually select the devices that will be used and combine them to achieve a prototype of the solution.Therefore, it is possible to note that, on one hand, the bottom-up approach requires software and hardware implementation from the very start, and for each step we move up on the SF, adjustments are required, which can be very costly both for designers (time and labor) and users (time).Additionally, the bottom-up approach also offers less flexibility in terms of the technologies employed, since they are chosen very early.This also implicates that once we reach the Social World it may be very difficult to adapt the current physical apparatus to fit all.On the other hand, the top-down approach only commits to specific physical devices in the very last step, allowing designers to consider UD-related issues much earlier.This gives them freedom to select the NUI devices and technologies that best fit the requirements they came up with during the descent from the Social World to the Empiric layer.This can save both time and money, since it will be possible to compare, in the Physical World, the different options that satisfy the requirements and then select the one that costs the least.We believe that these crucial differences in the two approaches point to the contrast between designing an Assistive Technology (AT) and designing a solution that follows Universal Design (UD).While AT usually refers to a device or computing system that assists people with disabilities [7], UD defends the creation of products that are usable by the greatest possible extend of categories of users [4].Hence, the SF indicates that, in the context of designing new NUI-based devices with a special attention to users with disabilities, the bottom-up approach may lead to an Assistive Technology that will most likely to address a specific category of user.In contrast, the topdown approach seems to promote Universal Design solutions.It is important to note, however, that within an iterative design process, it is possible to adapt the designs so they can address a wider variety of users, but this adaptation seems to be much more difficult if we are adopting the bottom-up approach.

Conclusions and Future Work
In this paper, we used the lens of Organizational Semiotics on a research project that aims at exploring the existing NUI-based devices in the context of Accessibility, either by testing their potential as assistive technology or by investigating how they can allow everyday actions of people with disabilities and others as well.The ultimate goals of this project are creating new NUI-based devices and proposing guidelines or conceptual frameworks for designing these devices in the future.The results of this study showed the usefulness of the Stakeholders Diagram and the Evaluation Frame to guide the creation of three different case studies scenarios and, in return, how the discoveries made in each case study reflected differently in the artifacts.In addition, we observed that the different directions of designing the scenarios reflected in the organization of requirements in the Semiotic Ladder.Finally, we observed evidence that starting a design from the top or from the bottom steps of the Semiotic Framework can have a huge impact on the way the resulting technology will address the user, either leading to an assistive technology or a solution for all, aligned to the principles of Universal Design.
We believe this evidence illustrates the contribution of Organizational Semiotics artifacts towards a design for all, especially in the context of using NUI state of the art devices.Hence, our future work includes proceeding with further iterations of each case study scenario to understand other semiotic aspects of those technologies in their real world usage.