V-Mail: 3D-Enabled Correspondence About Spatial Data on (Almost) All Your Devices

We present V-Mail, a framework of cross-platform applications, interactive techniques, and communication protocols for improved multi-person correspondence about spatial 3D datasets. Inspired by the daily use of e-mail, V-Mail seeks to enable a similar style of rapid, multi-person communication accessible on any device; however, it aims to do this in the new context of spatial 3D communication, where limited access to 3D graphics hardware typically prevents such communication. The approach integrates visual data storytelling with data exploration, spatial annotations, and animated transitions. V-Mail “data stories” are exported in a standard video file format to establish a common baseline level of access on (almost) any device. The V-Mail framework also includes a series of complementary client applications and plugins that enable different degrees of story co-authoring and data exploration, adjusted automatically to match the capabilities of various devices. A lightweight, phone-based V-Mail app makes it possible to annotate data by adding captions to the video. These spatial annotations are then immediately accessible to team members running high-end 3D graphics visualization systems that also include a V-Mail client, implemented as a plugin. Results and evaluation from applying V-Mail to assist communication within an interdisciplinary science team studying Antarctic ice sheets confirm the utility of the asynchronous, cross-platform collaborative framework while also highlighting some current limitations and opportunities for future work.


INTRODUCTION
S PATIAL data visualization has long been essential for breakthroughs in medicine, science, engineering, the humanities, and more (e.g., [1], [6], [19], [42]).Sharing these visualizations with others, however, remains a major challenge for many 3D, scientific, and/or big data visualization applications because they typically require highend devices and data processing-the type of computing infrastructure that is not portable and not accessible to most non-experts.One strategy for addressing this problem is to port the exploratory visualization tools scientists use on their current high-end visualization systems (e. g., advanced volume rendering) to the commodity devices that people use regularly, such as through remote rendering [27] or new web technologies [30], [39].A fundamental limitation, however, is that scientific visualization applications are often complex, requiring substantial expertise on the data and on how to use software features.Thus, even when it is possible for applications to run on commodity devices, there are still serious challenges to overcome in order to engage new audiences (e. g., people who may have little incentive to explore the data) and, often, even our team-science collaborators, who take on advisory roles in developing visualization techniques.
In light of this situation, we take a different, complementary approach.We argue that bringing full-fetched data exploration capabilities to every device may not be the most important requirement of high-end collaborative Manuscript received December 7, 2022.
visualization for spatial 3D data.Instead, what is fundamentally needed is to improve human-to-human communication about the data, and the way to do that is to make this communication work on everyone's existing devices.This approach conceptually builds closely on prior work in storytelling with data visualizations [8], [20], [32], which has been explored extensively for 2D abstract data but less so for spatial 3D data.By adopting a keyframe-based storytelling metaphor [3], [41], we have developed a system that creates animated data stories from data views captured directly from exploratory visualizations along with a mechanism of using an URL of a standard video file as the base 'token' for distributing a data story.By packing the communication about the data into a standard video format and sending this critical human-to-human communication rather than the raw data and visualization application, our system leverages existing e-mail and social media applications as tools for distributing stories.
Many forms of storytelling are, however, collaborative activities by nature [21], [24], [43], and people often take on different roles in this process [7], [12], [24].We, thus, also support tasks that data story audiences may carry out after an initial viewing of a story such as providing feedback to the original author, verifying claims made in the story, or even adding new content to the story.
To turn data storytelling into a collaborative and iterative process, we extend our system and introduce the V(isualization)-Mail framework.The key approach is flexible export of data story where a single V-Mail story token can trigger experiences with different storytelling fidelities, based on the absence or presence of dedicated software (Fig. 1).By having options from simply viewing the story on a video player to opening it within specialized visualization software to explore and annotate the 3D data, it is possible to cover Fig. 1.Three working platforms that demonstrate different storytelling fidelities of V-Mail for analyzing Antarctic ice-sheet data: (right) anyone with a video player can at least passively view the story, including annotations made by others; (left) in the highest-fidelity case, the story unlocks data on a V-Mail server than can be loaded via a plugin for desktop-based visualization applications, where users can explore and annotate the 3D data more deeply; (middle) the mobile client works as a custom video player with mechanisms for interactively viewing the story and adding annotations.
a wide range of use-cases for collaborative discussion and 3D data analysis, including making it possible for users with different tasks and levels of familiarity with visualization applications to work in unison.
We also present several extensions to this core idea to further enhance 3D-specific data storytelling.For example, since understanding the spatial context for specific data views is often critically important, the views are represented as 3D frames floating in the same space as the data visualization, enabling the audience to see them from various vintage points.In addition, to increase clarity, we employ a staged animation when the transition between data views would require changes to several visualization parameters, for example, when new data filters or color maps have been applied.
In summary, the main contributions of this article are: • the design of a novel visualization framework to facilitate 3D-enabled multi-person correspondence about spatial datasets for team science, • an extension of keyframe-based storytelling metaphors to support co-authorship, treating the created stories as collaborative documents authored asynchronously, • the design of a data story server and communication protocol for using V-Mail across an ecosystem of hardware platforms that seamlessly adapt the fidelity of the experience to the hardware, and • user feedback and lessons learned from a first evaluation of using V-Mail in a real team-science setting where team members utilized all V-Mail client platforms we created (desktop app, mobile app, and native video players on multiple mobile and web-based devices).

RELATED WORK
V-Mail relates to research on interactive tools, techniques, and formats for storytelling with data visualization and builds upon multiple topics (e. g., animation and annotation, interactivity, co-authorship and collaboration) that have become consistent themes in this research.We note that, although storytelling techniques have been used for presenting both spatial [28], [29], [37], [41] and non-spatial [4], [11], [23], [24], [33], [35] data, recent work has focused more consistently on non-spatial data.V-Mail, therefore, builds upon a trend of customizing storytelling support tools for specific types of data (e.g., [3], [26], [36], [41]), but does so for the specific case of spatial 3D data, as often encountered in domains such as physics, astronomy, biology, and medicine.

Co-Authoring and Interacting with Stories
Many forms of storytelling are, by nature, collaborative activities [21], [24], [43] and people often take on different roles in this process [7], [12], [24].In a recent survey, Tong et al. [37] describe authoring tools for parallel storytelling, where multiple authors work simultaneously to create the final visualization content.Like V-Mail, this type of storytelling blurs the lines between storyteller and audience; the audience partakes in the storytelling and the initial storyteller is also an audience member.In sense.us[14], for example, users communicate collaboratively using annotations and comments posted online on web-based data representations.
The audience also partakes in the storytelling process in GeoTime [9].In an accompanying text editor, authors write text-based data stories using bookmarked data views as references to claims.By scrolling the editor filled with the links, the audience has the ability not only to see the message with text descriptions but also to flip through saved visualization states for further exploration.Further, by modifying text in the editor, the audience can add new content or even new data views found in their own data analysis sessions.
V-Mail is similar to GeoTime in that multiple users collaborate on visualization presentations.Unlike GeoTime, however, V-Mail employs several simple yet effective techniques to convey different users' contributions.For example, as in the work by Heer, Viegas, et al. [14], [40], each comment shows the author of the edit.Further, V-Mail incorporates lower-fidelity options for viewing and collaborating with others on low-cost devices such as mobile phones.By sharing a visualization state via a URL [14], [40], V-Mails can be referenced in other communication media such as personal blogs.Similarly, by using video files as the means of distribution [15], V-Mails can be widely distributed, including through social media.V-Mails provide users with different options for consuming data stories, from passive viewing to interacting with them, as there is no one correct level of interactivity on the spectrum for constructing a narrative visualization [12], [29], [35].While interaction can distract viewers from an initial story arc [21], the ability to interactively view data stories is typically seen as giving the audience a feeling of involvement and lends credibility [22], [28], [29], [41], [43].
Existing web-based systems that rely on remote rendering [27] or new web technologies [30], [39] can also allow people to carry out visualization tasks on the commodity devices they use regularly.These systems, however, often target data exploration tasks, whereas we focus on data communication with the support for collaborative annotation and storytelling.Also, our first prototype provides storytelling techniques on top of visual outputs from exploratory visualization systems, so the V-Mail concept could be applied to visualization software built as standalone applications.

Annotating Spatial 3D Data
When compared to creating data stories for 2D or nonspatial data, authoring stories about 3D spatial data includes another special consideration in that it requires connecting text, speech, drawings, or other input from authors to specific spatial views or data elements.These software features are often described as 3D annotations, which can be tied directly to data objects [38], [41].Alternatively, graphical annotations can be drawn in 'screen-space' [3], [14], [26], [38].Without the need to explicitly reference data elements, this 'screenspace' drawing with 'free-form' annotations enables users to be more expressive [14] and even to draw in 'empty space,' which can be useful in sparse 3D scenes [38].However, free-form 'screen-space' annotations are typically not useful when viewed from other vantage points, as the spatial context is only valid from the exact viewpoint where the annotations were made.Some form of navigation between annotations is needed along with an ability to see what viewpoints include annotations.V-Mail thus implements screen-space annotations in the style of Boom Chameleon [38].After creating an annotation from one viewpoint, we add a snapshot of the annotated view to the scene as a 3D widget (view frustum with a textured film plane) that can be seen from other vantage points and returned to later.

Arranging and Transitioning Between Multiple Views
Many data storytelling systems provide multiple complementary data views, and they are considered critical for 3D data storytelling [3], [37], [41].One reason is that understanding the complex spatial relationships in 3D data often requires multiple and/or animated perspectives [21], [25], [29], [35].By smoothly transitioning between these perspectives, well-designed animations can allow the audience to build a congruent mental model of the spatial data [11].Different mechanisms exist for authoring animations for data storytelling.These can be as simple as recording a sequence of interactions performed by the storyteller [41] or providing animation templates to assist users in creating a final animation [3].Authoring a meaningful animation, however, can be complicated.Automated techniques are thus also useful, especially in time-critical scenarios such as urgent care in a hospital [15] and even for science dissemination [22].V-Mail, adopts a semi-automatic approach, similar to that of Lidal et al. [26]; the animations are created by interpolating the values the storyteller has captured/created in discrete states.We extend this concept by utilizing a staged animation [3], [13] to increase clarity when the transition would require changes to several parameters at once.Additionally, V-Mail includes a linked 2D storyboard view to define the temporal context for each key state and to re-sequence the story if desired.This feature is motivated by prior work that created guided 3D visualizations by capturing and ordering multiple views and found that breaking the presentation into steps can improve comprehensibility [41].

DESIGN CRITERIA AND CORE DESIGN CONCEPTS
Based on this discussion of existing means for communicating about or using spatial 3D data and the challenges we outlined in the previous sections, we now translate these points into specific design criteria and introduce the design concepts of V-Mail that guided us toward our vision.

Design Criteria
Our overall approach is driven by today's huge spectrum of data communication possibilities, yet with no solution that matches our vision.On one end of the spectrum, people use e-mails enriched with screenshots from generalpurpose visualization tools such as ParaView [2].This has the advantage that one does not have to worry about the device of the addressee, yet messages are limited to texts with few images, with little expressiveness.Video exports can also be used to showcase visual representations with higher fidelity and in a continuous fashion, yet this only supports one-way communication.Moreover, communication features such as drawing and commenting are typically missing in e-mail based communication.The other end of the spectrum is marked by storytelling tools, such as that by Wohlfart and Hauser [41], in which recipients can load a shared visualization not only to view but also to explore.Unfortunately, such tools require collaborators to have a similar hardware and software setup and often hours of training to use the visualization application.As argued previously, this often makes them accessible only to experts.Our key goal is thus to facilitate communication about spatial 3D data without having to worry about recipients' devices and skill levels, yet with an ease of communication, responding, and commenting that approaches that of e-mail.To achieve this vision, we formulated the following design criteria, which are closely informed by ongoing data visualization research on storytelling, communication, and collaboration: As audiences have different goals and needs, C1 we need to provide them with different options for consuming the presented material, e. g., with different levels of interaction and detail [12], [24], [43] and C2 we want to support a range of devices to view messages and provide feedback so users can pick a visualization suitable for their tasks and current situations [16].As there can be multiple ways of interpreting results, C3 messages should capture the thought process that lead to a conclusion so others can make informed decisions [26], [31], [41] C4 and, when going through the steps of presenting spatial data, it is important to a method should be provided for the audience to maintain spatial correspondence as the presentation proceeds and to understand where, spatially within the data, each step takes place [22], [29], [37], [38], C5 presenters should thus have various options to express their ideas and thoughts and to draw the audience's attention to particular viewpoints and data features [26], and C6 the audience should have the ability to add to and edit the content so that groups can exchange ideas back and forth iteratively as they work collaboratively to understand and interpret the impact of the data [24].

V-Mail Core Metaphor and Design Concepts
The core metaphor behind V-Mail is that of a bookspecifically, we recall children's' picture books that include text and imagery on each page and are often organized so that each page functions as a mini "episode" of the story.A child looks at a page in a book and reads the text at their own pace to understand the setting and characters pictured.Then, to advance, the child flips to the next page to continue to the next episode.Books for adult readers follow a similar progression of steps, but this is often arranged at the level of chapters.The relationship of one chapter to the next may not be immediately apparent, but the perspective provided by each chapter is, ultimately, critical to the interpretation of the whole book.Books of all forms naturally lend themselves to a variety of annotations, from hand-drawn illustrations to remarks scribbled in the margins.In V-Mail's approach to storytelling with spatial 3D data, a 'page' is a view of the 3D dataset that shows a particular situation.Annotations can be provided by sketching onto the 2D projection of the 3D view or by attaching text messages to it.'Flipping to the next page' animates from one 3D view to the next, allowing viewers to follow along while maintaining spatial context.Several page flips can be connected into 'chapters', but the transition from one chapter's end to the start of the next can be more abrupt (i.e., without an animated transition).As in the children's picture book, it is possible to animate forward and backward through the data story, with different playback modes to control the time with precision or to quickly skip to a new chapter.With this approach, we break complex messages down into more digestible, visual, and expressive forms (C3-C5).
The V-Mail frameworks requires just three additional concepts to be added to this core "book metaphor" functionality: Floating 3D pages-When presenting visualization results with a set of captured data views, we often think of a panel listing the views; e. g., thumbnails on the viewer in AniViz [3].V-Mail includes such a panel, but in addition to this panel, the data views are also included directly within the spatial visualization of the underlying 3D data.Together with the smooth transitions between the data views achieved with animations, these floating pages assist users in understand the spatial correspondences between views as they progress through a data story (C4).
Living documents-To remove the one-way character of existing 3D data correspondence and turn data communication into a collaborative and iterative process accessible to non-experts, V-Mail treats data stories as living documents (C1, C6).The framework makes it possible for users to explore a V-Mail, provide annotations, and extend stories in a way that scales with the capabilities of their used device.
Communication via a lightweight token-To realize this device flexibility (C2) and thus be able to engage a broad audience (C1), V-Mail does not rely on dedicated rendering support.Instead, stories are shared as a video URL-online token that can be interpreted based on the used device and the absence or presence of dedicated software.The video-based, low-fidelity version of the data story makes it possible for users to open it on a mobile phone with a video player, for instance while traveling.However, this lowest level of fidelity supports only simple viewing options (e.g., the web client in Table 1).A range of higher-fidelity versions that require dedicated software offer additional exploration and annotation capabilities.On a high-end desktop, users can freely explore the data depicted in the story and add additional pages or chapters (e.g., the desktop client in Table 1).A middle-fidelity version only supports exploration at certain view locations but with fairly advanced annotation capabilities and story navigation features (e.g., the mobile client in Table 1).To make it easy to manage such device flexibility, V-Mail associates all story data with the video, and shares it as a URL token that is opened appropriately by the different devices.

DETAILED DESIGN AND IMPLEMENTATION
We now describe the visual components, interaction techniques, and distribution mechanisms required to realize this vision.

Visual Components
A V-Mail story is composed of data views with drawings and comments from different users.We add animations in-between the data views to convey their spatial correspondences in a continuous fashion.Our V-Mail desktop client is one example of a V-Mail client (Section 4.3.3)and is a plugin to an exploratory visualization application [18].The plugin is comprised of three visual components: (1) the storyboard, (2) floating 3D data views, and (3) animated-transitions.

The Storyboard with the Comment Panel
This first visual component shows an overview of a data story and the input from contributors (Fig. 2(b)).Thumbnails of data views provide a quick glance at important points in a story and its presentation order (Fig. 2(c)).We also overlay drawings on the thumbnails to indicate visual highlights from users.Sometimes it may be more appropriate to textually describe what is shown in a shared data view; e. g., for writing a case number in a data ensemble or the name of a rendering algorithm.The comment panel provides an interface for leaving such notes and viewing comments left by others (Fig. 2(f)); each comment includes the author's name.

Animated Transitions
Animated transitions help to convey the spatial and contextual correspondences between data views in a story.Instead of jumping from one data view to the next, we use animated transitions that the audience can gradually adapt to the visual changes in different data views.The animated transitions are created automatically whenever a new data view is added to minimize the time and cognitive effort needed to create a V-Mail story.Four types of animated transitions (Fig. 4) can be used, depending on the differences between the viewpoints and visualization states in the adjoining data views.If only the camera location changes, the animation (1) translates the camera from a starting location to the next location. 1When only the visualization state has changed, the animation (2) gradually changes the values of visualization parameters from the previous state to the next by interpolating the values of just the parameters that have changed.Users may stop the transition to explore the visualization in an intermediate frame, but this exploration is only possible when the set of visualization parameters of two states are the same and the parameters can be changed continuously (e. g., changing a slicing plane in volume rendering).In cases where there are no obvious intermediate frames (e. g., changing a visualization type or dataset) we ( 3) use an imagebased blending approach.This blends two images where the screenshot from the previous data view fades-out and gradually fades-in into that of the next data view (see the second row of Fig. 4).When both camera location and state change, we (4) break the transition into two stages.First the camera moves to the next location.Then, the visualization state gradually changes to that of the new data view (see the last row of Fig. 4).

Basic Story Interaction
We introduce several interaction techniques to create, view, and explore V-Mails.
1.In our prototype, a linear interpolation is used; however, other motions that optimize camera motion to support specific goals could be useful.We thus implemented the animated transitions as abstract methods to enable future developers to override default animations with alternatives.

Creating a story and annotating data views
We begin creating a story with an empty storyboard.Users can navigate in the data visualization application (Fig. 2(a)) using typical 3D view manipulation, and then add a data view to the storyboard via a dedicated button ("+" in Fig. 2(b)).We then capture the view's thumbnail and insert it as a reference into the storyboard, also extending the timeline (Fig. 2(d)) appropriately (for ≥ 2 thumbnails) to account for the needed transition animations.Users can also re-arrange the data view sequence using simple drag-anddrop interaction, and they can remove data views as desired.
To be able to tell a story effectively or to respond to one, it is also often useful to sketch directly on a view to direct someone's attention to particular features, which we support on captured data views (Fig. 5).To avoid conflicts between sketching and view manipulation, we differentiate explicitly between these two modes and users can switch between them using two buttons at the bottom of the data view.In addition to the sketches, longer textual comments can be typed and associated with the data views (Fig. 2(f)).

Uploading a story
When ready to share the V-Mail, users upload the story to a server.Instead of uploading a story to the server whenever there is a change, we initiate this upload process explicitly (via an entry in the drop-down menu).We chose this design to enable the user to ideate and draft a story before sharing it to others, similar to drafting an e-mail before sending it.

Viewing an animated data story
As a sequence of data views with animated transitions, a V-Mail story is akin to a video with chapters.We thus borrow different playback modes from traditional video players and adjust them to our application (Fig. 6).
As in normal video players, playback continues until the end of the story (top row in Fig. 6).However, different from normal videos, the V-Mail player pauses briefly at each 'chapter mark' to devote additional time to and indicate the significance of these key perspectives.To further emphasize the data views, the player also includes mode where playback stops completely at each data view so users may read annotations in detail (second row in Fig. 6) and respond to them-similar to the playback style used by Wohlfahrt and Hauser [41].Finally, for faster navigation, the player also supports jumping between data views without animations (bottom row in Fig. 6). 2 Another aspect of data stories that is different from normal videos is that it can be useful to view data stories, especially the transitions between views, in reverse.Thus, all three of the play modes include support for reverse playback.
To allow users to explore V-Mails in a more fine-grained manner, similar to video players we also provide a timeline slider (Fig. 2(d)).To emphasize the aspect of chapter transitions we constrain the timeline dragging to between two neighboring data views, so that it is easy for a user to focus on the visualization view while controlling the timeline with a mouse.This constraint is reset on each mouse down (for exploring a new transition) and can be removed with a modifier key.

Linking free exploration and storyboard timeline
Again different from traditional video players, the interaction in a data story is not restricted to just playing back a precreated video stream.Instead, a user may exploring the 3D data visualization freely, potentially to capture slightly different shots or to better understand a feature in the data.After such a free exploration (and if no new shot was added to a timeline), the viewer may want to transition back to the timeline.We support both of these types of transitions by keeping the 3D visualization view and the timeline exploration linked and use animation transitions from the set described in Section 4.1.3.Each time a user switches from timeline to 3D view, we store the current position in the timeline to be able to transition back.In the opposite direction, we create a transition from a current state in the 3D viewer to the last active one.We also allow users to jump to stored data views, not only by interacting with the 2. The desktop client does not have these skip buttons because the skipping navigation can be done by clicking thumbnails (Fig. 2(c)).In the mobile client, however, we place the buttons in the bottom corners of the display to provide easier access (Fig. 9).storyboard view, but also by clicking on the respective data view frames in the 3D view (Fig. 7).

Cross-platform Collaboration Mechanisms
To truly support our cross-device collaboration vision (C2) we cannot rely on all users having access to a high-end PC, which may be required for the interaction style described this far.We thus now describe how users respond to V-Mails on different devices and introduce the platform-specific prototypes (i.e., V-Mail clients) we have developed so far.

URL as a lightweight token for story sharing
While we export a collection of files for each story (Table 2), we do not want to bother users with having to manage and send these files.We thus manage the file collection on a dedicated server (that only has to provide file management and remote access) and instead share a story in form of a URL that points to the repository.An essential benefit of using a URL is that this allows users to not only link to a story in an e-mail but also to share a story on social media.We create this URL after each export (Fig. 8(b)).When opened in a web browser, the URL simply shows a video player (Fig. 8(c)).On devices with access to our dedicated clients the website, the URL provides a means to open the story in the dedicated software.Alternatively, the clients also provide a list of stories stored on the server (Fig. 8(a)).In both cases, the clients then access the files needed for their specific capabilities and provide the respective interaction tools, as we detail next.

A device ecology for collaborating on data stories
We implemented three specific clients as a proof of concept and to mirror the data levels we described in Section 4.3.1.
In Fig. 1 we show photos of these solutions and in Table 1 we compare them by the features they support.As just described, our most basic web client relies on the video player available on virtually all connected devices (Fig. 8(c)) to facilitate passive storytelling [41].Depending on the functionality of this player, one may only perceive the story as a simple playback with graphical annotations for the data views.If the player supports it, however, one may also jump between chapters and/or see the textual comments that are available as video subtitles.
The web client can be accessed on virtually any connected device and uses the device's built-in video playback.With it we can thus reach completely new audiences with data stories.Even if someone never used a data story before and thus does not have or is not willing to install dedicated software, they can still view them.Depending on the media channel used to share the URL, they can also respond to a story such as by commenting in a separate e-mail.Moreover, this form of consuming also allows users to explore the story on their own first [24], before, potentially, adding to it and working collaboratively with other authors.
To explore the situation where users are willing to install a dedicated tool such as a dedicated mobile app on their phone or tablet, we developed an Android V-Mail client (Fig. 9) to facilitate a higher interaction fidelity for both phones and tablets.On top of playing back the video of the data stories, this app also allows users to use our advanced playback modes, add new annotations (both graphical and textual), and have limited 3D exploration via the pre-computed 360°i mages for each data view.As it is likely used on small-screen devices, the app does not support changing the chapter order.And instead of using thumbnails for the data views in the storyboard, the app places crumbs on the timeline to provide visual and navigational support to users.The mobile app facilitates a much richer form of collaboration by means of data stories than the web client, including being able to directly comment within the viewing environment and some 3D exploration.However, since this functionality relies only on the built-in video playback available in virtually all devices, the processing power required is essentially the same as the web client.Also, because it runs on mobile devices, it inherently allows people to engage with data stories like they do with other online media, for example, during a break, at home, or while traveling.In fact, the URL format makes it possible for people to share V-Mails on a variety of communication platforms, from e-mail to messaging apps, already installed on their phones.
For full-fidelity interaction, however, we created a desktop V-Mail plugin to a data visualization application [18] for both Windows and MacOS.This setup provides the functionality described in Section 4.2.It requires a highperformance computer, but it provides full flexibility in data exploration, story creation, and editing.At least one desktop client like this is needed to create an initial sharable data story, and this high-end client also makes sure that any updates from mobile app users get re-embedded into the video exports.This client also supports creating story branches with new sharable URLs.

SYSTEM AND IMPLEMENTATION NOTES
Our current V-Mail implementation comprises three clients (Fig. 10).We built the desktop plugin on top of an exploratory visualization software; it provides all the storytelling capabilities.When the user of the plugin uploads a story, files shown in Table 2 are generated and uploaded to the V-Mail server.One of these files is a video, and the mobile and web clients work with this video file.The mobile client is a special video player for data videos with the dedicated play modes we described in Section 4.2.3 and annotation features.The web client relies on a video player within a browser.

V-Mail server and general operation
The V-Mail server comprises a database and a file storage.The purpose of integrating the database is to check available stories without opening files in the storage.It has a single table; each row represents a V-Mail story, and the columns are: an unique ID of the story, a name given by the user who created it, and last times modified by each client.The file storage contains additional files for V-Mail stories (Table 2).Files for each story are saved in separate folders, and names of the folders are the unique IDs listed in the database.
When a user opens the desktop or mobile client, the client accesses available stories from the database and shows these as a list (Fig. 8(a)).When the user clicks a button to open a specific story, files used by the current client ("Use by" column in Table 2) are downloaded and loaded.When the

Desktop and mobile clients details
We built the desktop and mobile clients using C# in Unity3D.
We handle storytelling and collaborative activities in V-Mail with a StoryManager (Fig. 11).In addition to providing features for loading and uploading a story to the server, the StoryManager has three components for viewing and editing a story.A Message manages content in the current story and has an ordered list of Page(s).Each Page contains view and visualization state information captured from an underlying visualization application.It also contains comments and sketches drawn by users.The MessagePlayer updates the story time depending on the current playback option (Fig. 6).

APPLICATION CASE AND EVALUATION
To evaluate the effectiveness of our concept for team collaboration, we recruited members of a leading, interdisciplinary climate science research team to integrate V-Mail into their normal workflow over a two-week period.This section reports on training, observations, results, user feedback, and other lessons learned from this real-world application.We also showed the tool to members of a neuroscience research team, and report lessons and user feedback from their initial encounter with our tool in Section 6.3.Unlike the climate science team, they decided not to be part of a longitudinal study.In the section, we also report limitations and areas could be addressed in future work.

Methodology
The interdisciplinary climate science research team uses a combination of tools, including some of the world's most advanced high-performance computing (i.e., supercomputer) simulations, to study complex interactions of Antarctic water masses that may explain the global rise in sea temperature.The team includes oceanographers, geologists, computer scientists, a visual artist, and a science writer.Like many research teams today, they place an increasing emphasis on improved communication with the public.The artist and writer have played leading roles in these efforts, and the team has produced five short videos describing the science in the past seven years, all of which were national competition finalists (two won awards as best submissions).One of their recent videos was created with the Unity-based Artifact-Based Rendering (ABR) engine [18]-the same engine used for our first implementation of a V-Mail plugin.Since one of our goals was to understand how V-Mail might integrate with whatever data communication work is at hand in a real, research-active science team, we asked the team to use V-Mail for whatever tasks they thought made sense during the two-week trial we had agreed upon.The team decided that the most interesting communication task at hand was to develop one or more complementary videos to go deeper into certain aspects of the science and provide useful material for the scientists to use in their presentations.Thus, exploring how V-Mail might be utilized to support this work became the primary task of the V-Mail trial, and the two primary users became the visual artist, a professional sculptor and painter for more than 30 years whose collaborations with scientists led her to data visualization, and the science writer, a Ph.D. student in cultural geography.During the first few days of the twoweek period, we met individually with the artist and science writer to provide training for the V-Mail clients they chose to use and to assist them in their tasks.We recorded qualitative feedback by taking notes from interviews we conducted after the training sessions as well as via questionnaires before and after the two-week period.In addition, we saved each V-Mail that they created and configured the V-Mail clients to log all edits.After the two weeks, the artist shared the resulting V-Mail in a regularly scheduled, "live" video-conference with the scientists, gathering feedback on whether the story covers important data features and whether it employs appropriate storytelling techniques.

Observations and Results
We organize this section by the type of data we collected.

Pre-study Questionnaire
In the questionnaire we asked for the teams' pre-existing communications and related needs, and the two primary participants filled out the questionnaire.Their responses showed that they generally take different approaches for data communication.For urgent matters, participants typically communicate in-person or send e-mail-like messages.When answering a question that could be answered in few words, participants use an e-mail-like tool.However, when it takes multiple steps to explain a concept, they typically use synchronous communication, such as a live video conferencing tool.The choice was the same for describing spatial features in their data and for public-facing communication.When asked about their use of visuals in e-mails, participants responded differently.One does not use e-mails for sharing 3D visualization results.The other said that about 60% of her e-mails contains images or videos, and 5% of these contain annotations.With regards to device usage, both participants typically have some access (2-8 hours) to high-end or specialized devices.However, they typically use personal PCs or mobile devices for science-related communications.One participant indicated that her responses to questions may vary depending on goals, as she often uses a high-end PC when working on a video or visualization project.

Produced V-Mails
In total, the final V-Mail contains ten data views (Fig. 12).The first three (#1-#3) provide an overview of data visualization, highlighting all visible variables.The last shot (#10) is another overview shot.Data views in the middle (#4-#9) describe two different concepts about interactions of water masses and currents on the ice shelf.Each concept was communicated with three shots.The first (#4, #7) gives a spatial context, the second (#5, #8) zooms in further to focus on visual decisions, and the last (#6, #9) zooms in further to relate the visual decisions to specific data features.Each shot is discussed in detail with textual comments contributed by both artists.
Interested in sharing their work to collaborating scientists and possibly using it as video footage for their next video projects, the artist requested another version of the V-Mail that shows the fly-through animation at a higher resolution (1920 × 1080) and without the floating 3D pages.Treating it like a final draft of their work, the "final" V-Mail shows only the voice-over script, without all the low-level communications such as questions in a form of comments.

Software Logs
Our log data and users' comments made in the story indicate that there were five back-and-forth communications over the two weeks (Table 3).At each turn, after making edits, the participants used their normal e-mail to pass the V-Mail link to each other and request comments.Both URL and name were used to reference updated V-Mails.
The textual comments were added to data views and were used mainly to provide guidelines and script a voiceover.Questions were also asked in a form of comments.For example, confused with the scripting guidelines, the science writer left questions on corresponding data views.We also found that the comments contained imperfections, like typos and this accidental use of caps lock, "hERE WE CAN SPEAK about."Capitalization was also used for emphasis, as in "IGNORE the above."The artist and science writer did not use the sketching feature to annotate data views.
The visual artist used the desktop client and the science writer used the mobile client.While working on the script, the science writer realized that a few data views were not needed.While working on the mobile client, she left comments on these extra data views to request a removal.Because the artist using the desktop client forgot how to remove data views and  we (the authors) were also monitoring the communication, we were able to quickly help out and resolve the request in the desktop client.

Observations during introductory sessions
At the end of each training session, we asked users to now begin to use the tool while we observed their actions and answered any questions that came up.
In the first V-Mail, the artist captured data views with scripting guidelines written as comments (Table 3).The installation steps and training time for this stage took about 20 minutes for the desktop-client.In the next hour session, she created three V-Mail drafts from scratch (from an empty storyboard) and did not report any major difficulties.She discarded two of them, and proceeded to refine to third by adding a few more data views, rearranging their placement, and adding notes and guidelines to send to her colleague.The artist created V-Mails with multiple sections (i.e., data views) to cover different data features (Fig. 12) and used different approaches to extend and strengthen a story line.We noticed that the most recently added data view was often used as a spatial reference frame for the next data view.For example, after adding view #6, view #7 was described using the language, "now looking down and reveal the currents."The participant also added data views "out of sequence" and rearranged previously added views to strengthen the story.
In the second V-Mail, the science writer opened the message from the artist, then proceeded to work on drafting a voice-over script for each data view using the guidelines contained in the V-Mail (Table 3).The science writer used the mobile-client because it provided all features needed for her tasks.The installation and training took her about 5 minutes.She flipped through pre-existing data views and added the script as comments, finishing this task in next 30 minutes.She did not report any major difficulties while using the tool.

Feedback from User Interviews
After they created their first V-Mails, we interviewed the participants to learn about their views of both tool and concept.
The artist described gaps in their current tools and relevant cases where V-Mail could be most useful.The artist described her typical approach of creating a fly-through animation as, first, establishing the concepts to convey and, second, capturing the shots needed for the concepts.She described the process as "melding story into visuals."For all but simple topics, in-person communication or video conferencing are preferred for describing such visual and spatial concepts, but in practice, they often resort to sending several fly-though animations to the scientists to confirm the concepts.In their traditional workflow, this process takes a long time as they often require technical assistance from additional team members to create their animations.V-Mails, in contrast, enable artists, science writers, or other team members to control this process on their own.They can focus on identifying the right views to convey the concepts, and the animation is automatically created.The ability to quickly mock-up animations from data views expedites the creation process, and the ability to view the results on mobile devices and web browsers simplifies the distribution.
Imagining how their experiences might translate to a future trial with the larger group, the artist pointed out that the mobile client will provide a means for the scientists to provide feedback without the need to go through complex installation steps and a long training.She also pointed out the importance of the web-client as it enables them to reach people outside of the group, e. g., the public and scientists from other groups.The artist did not use the sketching feature to annotate data views, even though, in several data views, she described features at specific locations of the views (e. g., eddies formed in a current shown in the bottom).When asked about it during the interview, the artist explained that she thought this would be useful in some situations, but in this case she knew the science writer was already familiar with data features and visual mappings from their close recent work together.
The participants made several feature requests.She requested to implement additional V-Mail clients for ParaView [2], a visualization tool the team frequently uses.The science writer suggested supporting audio recording to annotate existing data views as it is often easier than writing thoughts down.The writer told us that opening a V-Mail takes a long time on her device.She suggested another data presentation mode that shows a V-Mail without the animated transitions, as this can reduce the download time and the animated transitions might not be needed for some users who are already familiar with data presented in the shared views.Also, in our current implementation, the user first needs to add a data view before annotating it.She suggested another approach where a data view is automatically added when the user starts drawing on the current camera view.

Feedback from scientists
After the two weeks, the artist shared the resulting V-Mail in a regularly scheduled, "live" video-conference with the larger group, and led a discussion about the work using the webclient.The scientists were most interested in seeing details of water-mass distributions, current trails, and their impacts on melting glaciers.The created V-Mail narrative covers these regions of interests and employs proper storytelling techniques-e.g., a series of first overview, then zoomed-in data views [34] connected with transitioning animationsto show not only details of these regions but also their contextual information.They were also interested in seeing changes of specific regions over time.Unfortunately, timeseries data is not yet supported in the underlying visualization software.The scientists requested this as a future work and brainstormed ideas for storytelling techniques to convey these in future iterations.They envisioned to station the camera to focus on a specific region and then show changes over time.They would do this for multiple data regions and employ V-Mail's interactive storytelling techniques to return to previous camera locations to facilitate comparative analysis of the sites.

Post-study Questionnaire
One user from the climate science team participated in the questionnaire.Her responses reaffirm what she described about our tool in the interview session (Section 6.2.5), giving positive responses on features that enable her to make movies or tell data stories.However, she noted several additional points.The participant noted that connecting views with animations is a crucial component in data communication.The ability to pause in-between and play backwards as well as the ability to see the 3D frames, however, received neutral responses.Nonetheless, she was concerned about using small mobile displays as they may not be able to convey the visual detail of the visualizations.

Initial Feedback from a Brain Science Team
We integrated V-Mail into a different system that visualizes crossing pathways in white matter areas in a human brain (see Appendix B).We showed the V-Mail during a live video conference to members of a neuroscience research team, who was already familiar with the visualization system.Their initial reaction was somewhat mixed; they told us the tool itself is novel and could be useful in presenting findings to a broader audience, but there is no urgent need for such tool for their team-science collaboration.The team already meets regularly, almost weekly, using a live video conference or in a meeting room, and the team is led by one senior member to whom his students report.They use remote asynchronous communication tools such as e-mails to coordinate the meetings or share datasets.
Participants described situations where cross-platform collaboration tools like V-Mail may not work and could be useful.In clinics, radiologists use specialized monitors for viewing lossless DICOM images to see subtle differences in tissue contrasts.Also a participant pointed out that there is a limit to what information they can share with the public as certain information is sensitive and private.In cases where sharing information does not have such constraints, V-Mails could be used to enhance the credibility of presented results.They envision that stories created in V-Mail are submitted as supplementary material for research papers.Linked from figures or text, V-Mails could allow reviewers or readers to validate the authors' claims and further explore the data.

Discussion, Limitations, and Future Work
The success of adding a cross-device, asynchronous communication tool, like V-Mail, into the existing science team's workflow depends greatly on how the team already works.The brain science team decided not to partake in a two-week trial of V-Mail, as this team included just one senior member and several grad students who already meet regularly to discuss their work.They also raised concerns about security with respect to patient data that should not be widely shared (Section 6.3).The climate science team also expressed a preference for synchronous communication.However, schedules are more complex within this larger team, so the ability to communicate asynchronously is more valuable.
In addition to using the comment feature for drafting a voice-over script, participants used it to ask questions and delegate tasks, and these comments often contain typos and accidental use of caps lock.While these low-level communications were removed from the "final" V-Mail, they can help people to clarify tasks and enable other team members to provide immediate help.All of these reinforce the chat-like immediacy and draft-in-progress qualities of V-Mail.These qualities are unusual in science communication, especially high-end data visualization, but we believe they may, in fact, help to put a human face on science communication, helping scientists and stakeholders more freely discuss scientific data.
The science writer pointed out that communicating data in a form of stories might introduce unnecessary overhead for audiences who are already familiar with data (Section 6.2.5).For these users, the ability to disable certain storytelling features and share only portions of a data story could be useful.By adapting the feature that encodes all viewing properties in a shared link [30], we could associate an URL parameter to each data view in a story, so that a specific view could be referenced in e-mails or comments.The feature would also be required if V-Mail is used with other storytelling media, e. g., for providing interactive figures in a paper submission as discussed by the brain science team.In the same spirit, other storytelling paradigms that rely less on animations could also be integrated in V-Mail.For instance, scrolly-telling [35] can be used to rapidly control the speed and direction of viewing.We could also spread out data views in a layout, e. g., as done in data comics [5], which could be more useful for audiences who are already familiar with spatial locations of data features.
A major limitation of the current V-Mail implementation is that it can only be integrated into applications built using Unity3D, and a separate instance of the V-Mail plug-in is needed for each system.To ease the integration process, we are considering a more loosely coupled implementation where a separate storyboard application communicates over a network with visualization software.Since the webpage and the mobile app do not rely on other data visualization applications (these clients play a video), we can continue using these clients for showing and interacting with stories created from new connected software.
We observed that participants used lightweight clients if the features required for their tasks were available in those clients.We wonder, if the clients supported some lightweight data exploration features, participants might have explored data even though their tasks did not require this.Indeed, the brain science team viewed facilitating interactive exploration on the web-client as an important direction for future work.Thus, upgrading the web-client with remote rendering [27] or new web technologies [39] could be a useful next direction.In addition, as mentioned by the artist and the brain scientists, seeing visualization on small screens, like a phone, is just limiting.Further studies are needed to better characterize exactly how much value the capability provides to science teams.In addition, further studies with more and other domain experts should be carried out to confirm our findings from our qualitative evaluation.

CONCLUSION
On a conceptual level, our work extends the state of art on storytelling in visualization and provides a rich tool set for asynchronous remote collaboration.We build upon existing spatial 3D data storytelling techniques that tightly couple storytelling with data exploration and employ semiautomatic procedures to expedite the story creation process.What is unique in V-Mail is the ability to collectively work on data stories, which is often seen with abstract data visualization, e. g., via web-based visualizations, but less so with visualization of spatial 3D data.Essentially this multiauthor editing in V-Mail is a type of parallel storytelling, with the difference that users not only view or edit shared data stories but also do this across different devices.V-Mail is thus an example of using multiple display devices in collaborative data visualization [16], [17].V-Mail focuses, however, on asynchronous remote communication and provides lightweight mobile and web clients for partaking in collaboration and simplifies the distribution process using URLs to reference data stories.
The desktop-client is an example of asynchronous collaboration frameworks that support collaborative activities around exploratory visualizations.The mobile-client, however, supports these activities around video-based visualiza-tions created from the desktop-client.The ability to provide visual feedback directly onto shared visualization results is similar to the ability to annotate on shared data views and stories in the web-based visualization.However, there are two major differences in our approach: (1) communication mechanisms are tailored for spatial 3D data visualization, e. g., with spatial annotations and animated transitions, and (2) the app is designed to solely work with shared visualization results.The latter point may be of interest to visualization practitioners as this approach enables them to bypass installation steps and computation requirements often imposed by using exploratory visualization software such as ParaView [2].Also this separation suggests an interesting future direction of turning V-Mail into a collaborative framework that unifies well-known software in the field for creating data stories and conducting discussion.
In our first V-Mail deployment and evaluation, we took the approach of asking a real science team to integrate V-Mail in whatever ways they found most useful for the tasks at hand during a two-week period.The results were generally positive, re-confirming the urgent need for improved tools for spatial communication in this context, while also suggesting areas of required improvements to advance V-Mail from the prototype stage to a production-level tool.In this first application, only the artist and science writer actively used the tool to draft a video-based visualization, and the scientists were involved at the last stage to provide feedback on the created visualization.The brain science team provided initial feedback on the tool but did not partake in a longitude study.Notably, if we are able to secure funding to support additional work on V-Mail, we plan follow-on studies to understand how V-Mail might be used and/or tailored to best support specific scientist-to-scientist, public-to-scienceteam, and lay-person-to-specialist communications.

Fig. 2 .
Fig. 2. The V-Mail desktop client allows users to create a story directly from a visualization application: (a) the top portion is the visualization application; (b) the storyboard panel is placed on the bottom; (c) captured data views in a story; (d) the timeline interface provides playback options and indicates the current time in a story; (e) the drop-down menu provides options for opening, uploading, and sharing a story; (f) the comment panel shows comments on the current data view.

Fig. 3 .
Fig. 3. Example usages of 3D data views in data communication: (a) the user talks about data features focused in the views; (b) a bird's eye view shows the locations being discussed in the data.

Fig. 4 .
Fig. 4. Different animations to transition from one data view to another.Each type shows the changes in-between two adjacent data views.

Fig. 5 .
Fig. 5.The user sketches and leaves comments on the current view in the annotation mode.A drawing pallet is for specifying the color and the size of a brush.Clicking the toggle on the top of the storyboard switches the current mode to the exploration mode.

Fig. 6 .Fig. 7 .
Fig. 6.Playback options & corresponding buttons.Top: plays the animation from start to finish, or in reverse; middle: plays it until it reaches the next/previous data view; bottom: jumps to the next/previous data view.

Fig. 8 .
Fig. 8.A story URL works as a lightweight token for sharing and opening the story.(a) our V-Mail clients show several stories with their URL; (b) after a story's upload, the client shows the URL; (c) opening the URL takes the user to a webpage where the story video can be played and the story can be opened in a dedicated client via a button (bottom).

Fig. 9 .
Fig.9.We created the V-Mail mobile client (a) as a custom video player with visual components and mechanisms for interactively viewing and editing a story; clicking on the top-left share button in the viewer copies the story's URL and shows messaging apps to share the URL with (b); when clicking an e-mail app, we create a message with the name of the story as a title and the story's URL pasted into the main body (c).

Fig. 10 .
Fig.10.System architecture.V-Mail is composed of a server and three visualization clients.The desktop plugin is connected to an exploratory visualization software.The mobile and web clients support a portion of data storytelling activities around visual outputs stored in the server.
Page andTransition in a story.A Transition is composed of two Pages and a transitioning amount.Viewers are visual components that show the current Message and provide a means for editing it.CommentViewer updates the comment panel (Fig.2), and AnnotationViewer shows drawings on the render view.The VisualizationViewer updates an underlying visualization.If the current time is on a page, the viewer updates the visualization from the current Page accessed from the MessagePlayer.If on a transition portion, the viewer interpolates between two Pages from the Transition.As the mobile client plays video files, the StoryManager in the client is different from that of the desktop plugin.The VisualizationViewer is removed as the client does not rely on an underlying visualization, and we add a SphericalImageViewer to provide limited 3D exploration.The MessagePlayer updates the timeline slider based on the current time in the video, and skips time upon clicks on the corresponding buttons or a specific point on the slider.

Fig. 12 .
Fig. 12. Visualizations of data views' placements in the V-Mail participants created.Each circle represents a data view with a number indicating the view's order and with color indicating its thematic section.

TABLE 1
Availability and fidelity of operations that facilitate collaboration around shared stories and their prerequisites, by client/platform.

TABLE 2
* Fig. 11.Three components of viewing/editing a story in StoryManager

TABLE 3
Five back-and-forth communications between participants.