Storytelling and Visualization

Abstract

Source: Chao Tong, Richard Roberts, Rita Borgo, Sean Walton, Robert S. Laramee, Kodzo Wegba, Aidong Lu, Yun Wang, Huamin Qu, Qiong Luo, and Xiaojuan Ma, https://www.mdpi.com/2078-2489/9/3/65/htm
This work is licensed under a Creative Commons Attribution 4.0 License.

Introduction and Motivation

"We believe in the power of science, exploration, and storytelling to change the world" - Susan Goldberg, Editor in Chief of National Geographic Magazine, from "The Risks of Storytelling", October 2015.

Throughout history, storytelling has been an effective way of conveying information and knowledge. In the field of visualization, storytelling is rapidly developing techniques that enhance understanding. By visualization, we refer to "interactive data visualization" as defined by and described extensively by Ward et al. We note that there are many references to interactive visualization throughout the survey. For example, the term "interactive", appears over 40 times throughout the text. We also note that animated transitions are a major theme in the survey and receive their own section in the text. See Table 1. Many communities have commented on the importance of storytelling in data visualization. Storytellers tend to be integrating complex visualizations into their narratives in growing numbers.

Table 1. Our classification of the storytelling literature. The y-axis categories fall into who-authoring-tools and user-engagement, how-narrative and transitions, why-memorability and interpretation. See Section 1.2 for a complete description.

As contributions, we present a survey reviewing storytelling papers in visualization and present an overview of the common and important elements in storytelling visualization. We also describe the challenges in this field and present a novel classification of the literature on storytelling in visualization. Our classification highlights both mature and unsolved problems in this area. The benefit is a concise overview and valuable starting point into this rapidly growing and evolving research trend. Readers will also gain a deeper understanding of this rapidly evolving research direction.

1.1. Definition and Storytelling Elements

A story can be defined as "a narration of the events in the life of a person or the existence of a thing, or such events as a subject for narration" or "a series of events that are or might be narrated". Storytelling is a popular concept that is used in many fields, such as media, education and entertainment. Storytelling is a technique used to present dynamic relationships between story nodes through interaction. According to Zipes, storytelling can involve animation and self-discovery, incorporating models, ethical principles, canons of literature, and social standards. In education, a storyteller can improve and strengthen the literacy of students. Also, the storyteller can engage audiences so they feel a desire to read, write, act, and draw. Audience members can learn to express themselves critically and imaginatively with techniques they may learn from the storyteller or teacher.
In the context of the visualization literature. Lee et al. argue that "the community has been using the term 'storytelling' in a very broad way without a clear consensus or discussion on what a visual data story encompasses". They state that a visual data story includes a set of story pieces. Most of the story pieces are visualized to support one or more intended messages. Story pieces are presented with a meaningful order or connection between them to support the author's high level communication goal.
Furthermore no agreed definition of "visual data story" has yet emerged in the visualization literature. For a full-length 6 page discussion on this topic, we refer the reader to Lee et al.

1.2. Classification of Literature and Challenges in Storytelling and Visualization

Although storytelling has been developing in other fields for years, storytelling is a relatively new subject in visualization. As such, it faces many challenges. In this survey we have extracted the fundamental characteristics of storytelling both as an entity and as a creative process. Our literature classification is based on the logical notions of who are the main subjects involved in storytelling for visualization (authoring tools and audience), how are stories told (narratives and transitions), why can we use storytelling for visualization (memorability and interpretation). From these characteristics we have then developed the following dimensions which are common to storytelling in visualization.

Authoring-Tools: Authorship addresses who creates the story and narrative. Authorship commonly refers to the state or fact of being the writer of a book, article, or document or the creator of a work of art and its source or origin. Central to this definition is the writer or author. Rodgers defines an author as "an individual solely responsible for the creation of a unique body of work".

User-engagement: Engagement is about the audience and also concerns why we use storytelling. How can we ensure that the message comes across to the audience? Can we measure engagement?

Narratives: Narrative concerns how an author tells a story. Narrative structures include events and visualization of characters. Narrative visuals contain the transition between events. This entails, "Using a tool to visually analyze data and to generate visualizations via vector graphics or images for presentation", and then deciding "how to thread the representations into a compelling yet understandable sequence".

Transitions: Transitions are about how authors may tell the story. Transitions seamlessly blend events within a story and are key to its flow. Successful transitions vary actions as little as possible to strengthen overall coherence. Transitions in visualization can be either dynamic or static.

Memorability: Memorability addresses why authors present data in the form of a story. Memorability is an important goal of storytelling. A good visualization technique draws the viewer's attention and increase a story's memorability.

Interpretation: Data interpretation refers to the process of critiquing and determining the significance of important data and information, such as survey results, experimental findings, observations or narrative reports.

When examined in the context of storytelling in visualization each dimension raises interesting questions: Are current storytelling platforms taking into account the role of the author and supporting the authorship process? What forms of narrative structures and visuals best apply to storytelling in visualization? Are static transitions or dynamic transitions more effective for storytelling in visualization? Can visualization increase the memorability of data information or knowledge? Does storytelling and visualization aid with data interpretation? What is the most effective way to engage an audience? Data preparation and enhancement is another challenge for which there is currently no literature. Thus we include it as a future research direction but not in our classification.

Starting from the logical notions of who, how, why, and these open questions we have chosen these dimensions to form the basis of our literature classification on storytelling in visualization. See Table 1. It is important to note that some papers address multiple topics in Table 1 and in our classification. We placed papers by what we determined to be the main focus of the paper. This is very useful for obtaining an overview. However some papers address more than one theme, e.g., authoring tools and narratives.

1.3. Classification of Literature: The Second Dimension

In addition, the literature is also classified by the ordering or sequence of events, which refers to the traversal the path viewer takes through the visualization. This dimension is adapted from Segal and Heer. It forms our second categorization for Table 1. The classification includes:

1. Linear: A story sequence path in linear order is prescribed by the author.
2. User-directed path: The user selects a path among multiple alternatives or creates their own path.
3. Parallel: several paths can be traversed or visualized at the same time.
4. Random access or other: There is no prescribed path. There is currently no literature prescribing random order. Therefore we replace this with a column called "overview".
   

1.4. Literature Search Methodology

We search both the IEEE and ACM Digital libraries for the terms "storytelling", "narrative visualization", "memorability", "transitions in visualization", "user-engagement", and various combinations of these phrases. We focus primarily on the IEEE TVCG papers. We check the references of each paper and looked for related literature on storytelling. We also search the visualization publication data collection for these major themes in visualization and storytelling. Google scholar is also used as part of our search methodology.

In summary, our literature search includes:

1. IEEE EXPLORE Digital Library
2. ACM Digital Library
3. Visualization publication data collection
   
4. the annual EuroVis conference
5. the Eurographics Digital Library
   

Several other papers were discovered by looking at the related work section of the papers we found.

1.5. Survey Scope

The storytelling visualization papers summarized in this survey include the subjects of scientific visualization, information visualization, geo-spatial visualization, and visual analytics. In order to manage the scope of this survey, storytelling papers from other fields are not included, such as:

Virtual reality and augmented reality: For example, Santiago et al present "mogre-storytelling" as a solution to interactive storytelling. This tool provides different functionalities for creating and the customization of scenarios in 3D, enables the addition of 3D models from the Internet, and enables the creation of a virtual story using multimedia and storytelling elements.

Education: For example, Cropper et al. address the extent of how scientific storytelling benefits our communication skills in the sciences, and the connections they establish with the information itself and others in their circle of influence.

Gaming: Alavesa et al. describe the development of a small scale pervasive game which can take storytelling from camp-fire sites to modern urban environments.

Multi-media and Image Processing: For example, Chu et al. describe a system to transform any temporal image sequence to a comics-based storytelling visualization. Correa and Ma present a narrative system to generate dynamic narrative from videos. Image processing falls outside the scope of this survey. Video processing also falls outside the scope of the survey.

Language processing: Theune et al. develop a story generation system. It can create story plots automatically based on the actions of intelligent agents living in a virtual story world. The derived plots are converted to natural language, and presented to the user by an embodied agent that makes use of text-to-speech.

Annotation: The topic of annotation is included in the survey. For example, Annotation is discussed in Section 5.1. Narrative Visualization for Linear Storytelling where Hullman et al. describe a system called contextifier, which automatically produces custom, annotated visualizations from a given article. Hullman et al. is based on previous work in storytelling in visualization and Kandogan's automatic annotation analytics. It develops a system that can automatically generate custom, annotated visualization from a news article of company. The theme of annotations arises again in Section 5.4 on Narrative Visualization Overviews where we discuss literature on annotated charts.

Details-on-Demand: The theme of details-on-demand is included in the survey and is often used throughout the literature. For example in Section 3.2. Authoring-tools for User-directed and Interactive Storytelling, Figure 1 shows an image sequence taken from a sample linear volumetric story. The distinct story nodes refer to the key events in the story, which provide an overview first, then details on specific features on demand. Again in Section 5.4 Narrative Visualization Overviews, an Afghanistan nation-building development project example shows an interactive geographic visualization with details on-demand sliders that present the status of Afghanistan nation-building development projects. In another example, the Minnesota Employment Explorer shows how mouse-hover provides details-on-demand, double-clicking an industry triggers a drill-down into that sector while an animated transition updates the display to show sub-industry trends.

This article is much more up to date than the article by Segel and Heer which is already more than 7 years old now. The field has evolved substantially since then. Also, the article by Segel and Heer is not a comprehensive survey like this one. However it is a highly cited paper and makes a big contribution to narrative visualization.

There are other fields that study storytelling as well. In the next sections we describe the literature on storytelling in visualization. Our classification is presented in Table 1. An alternative classification is presented in Table 2. Figure 2 shows the visualization techniques used in storytelling for data visualization literature.

Table 2. An alternative classification of the storytelling literature based on scientific, information, and geo-spatial visualization. Both mature areas and unsolved problems are apparent.

Related Work

Ma et al. state that a story that is well paced exhibits deliberate control over the rate at which plot points occur. They present a selection of scientific storytelling visualizations from NASA related work and describes various examples.

The Scientific Visualization Studio (SVS) at NASA uses storytelling visualization to investigate observational data collected by instruments and sensors and make it more suitable for consumption by the public.

The science museum presents visualization to the public with complex and abstract geographic phenomena at extreme size scales for explanatory animations. The science museums provide further interpretation through labels, videos, and live demonstrations. See Figure 3.

Storytelling enables the user to interact with geographic data such as the Earth's climate or the collapse of a star by using a story model, such as story nodes or story transitions. Ma et al. is based on previous scientific visualization work at NASA, based in the scientific research center and scientific museum and describe how visualization can be used to tell a good story, and tell it well. This is a topic that the scientific visualization research community paid little attention to at that time.

Tong et al. published a storytelling visualization survey paper as a short paper in abridged form. It contains no image or paper summaries. This is a full-length, comprehensive, extended version of that survey. It is approximately double the length.

Authoring-Tools for Storytelling and Visualization

Authorship refers to writing or creating a book, article, or document, or the creator of a work of art according to The Oxford English dictionary, especially with reference to an author, creator or producer. For our purposes, we will adopt a definition of author described by Rodgers, "An author is best described as an individual solely responsible for the creation of a unique body of work". Hullman et al. state, "Story creation involves sequential processes of context definition, information selection, modality selection, and choosing an order to effectively convey the intended narrative".
An author is best described as an individual solely responsible for the creation of a unique body of work.

Presenting the findings of a qualitative study of undergraduate writers at The City University of New York, Hullman explores student perspectives on models of authorship, the relationships between these models and student experiences of authorship in different writing situations, and proposes the importance of distinguishing between the multiple models and definitions of authorship and the rhetorical contexts associated with each. Rodger develops a qualitative study of 800 students on the definition of authorship and their rhetorical contexts over a one-hour interview. Students defined authors as "[people] who see writing as being beyond a hobby", and as a term that should be applied only to those individuals for whom writing is "something he or she has to do", "a career", or "an act that will lead to something being published".

All papers in this section focus on authoring-tools for storytelling. Wohlfart creates new volume visualization stories for medical applications. Gershon and Cruz present general storytelling for information visualization. Kuhn, Lee and Plowman all develop unique creator tools for storytelling visualization.

It is important to note that our survey is not simply a list of papers. Individual papers are summarized according to a special methodology. This process connects related papers together such that the connections and relationship to previously published literature is made clear.

3.1. Authoring-Tools for Linear Storytelling

The literature in this sub-section focus on visual designs for authoring in a linear style that is prescribed, automatic, or semi-automatic (as opposed to interactive) or decided by the users. In other words, creators are provided with assets to formulate a linear story.

Gershon and Page state that storytelling enables visualization to reveal information as effectively and intuitively as if the viewer were watching a movie. They introduce the concept of storytelling and presents advantages of storytelling.

One example presents a situation in which a number of enemy positions surround a school with children trapped inside as de facto hostages as the crossfire fills the space overhead and both sides move toward confrontation. Gershon and Page is based on previous work of Denning and explain the usage of storytelling in information visualization.

Lu and Shen propose an approach to reduce the number of time steps that users required in order to visualize and understand the essential data features by selecting representative datasets. They design a flexible framework for quantifying data differences using multiple dissimilarity matrices. A new visualization approach that filters data analysis results, which is achieved by measuring the degree of data similarity/difference and selecting important datasets that contain essential data features. See Figure 4.

They interactively select representative datasets that include a significant portion of features of scientific data, whose data distribution requires more analysis than time sequence, reduces the amount of data to necessarily visualize and still keeps the essential data information. This can be used to improve the efficiency of time-varying data visualization.

An interactive storyboard is used to visualize and explore the overall content of time-varying datasets through composing an appropriate amount of information that can be efficiently understood by users.

Lu and Shen is based on the previous work of time-vary visualization and design a general method for comparing data dissimilarities. They do not require a dense sampling frequency to capture the object evolution and their work is not limited to specific feature models, such as geometry or interval volumes, and their attribute designs.

Storytelling, in the context of this article, deals with the core of information visualization by extracting relevant knowledge and enhancing its cognition. Cruz et al. present generative storytelling as a conceptual framework for information storytelling. They create stories from data fabulas using computer graphics as a narrative medium. Data fabulas are a set of time-ordered events caused or experienced by actors.

A story is formed by characters. It involves the representation of the fabula's actors and the definition of a temporal structure. The engine transforming a fabula into story consists of two models. The event model creates a story timeline and an action model creates a set of actors behaviours. For example an empire's decline visualizing western empire's decline in the 19th and 20th centuries. See Figure 5.

Cruz et al. is based on previous work of narrative theory and presents generative storytelling as a conceptual framework for information storytelling.

3.2. Authoring-Tools for User-Directed and Interactive Storytelling

A large body of research has been carried out for authors wishing to create their own user-oriented or interactive stories. This literature focuses on interactive, user-driven authorship (as opposed to automatic or semi-automatic authorship). Storytelling is a relatively new form of interactive volume visualization presentation. Wohlfart explores the usefulness of storytelling in the context of volume visualization. He presents a story telling model and divides the concept of volumetric storytelling into story authoring and storytelling constituents. He presents a volumetric storytelling prototype application, which is based on the RTVR (real time volume redering) Java library for interactive volume rendering. See Figure 1. The storytelling model contains a range of hierarchy levels, in top-down order, which are: story node, story transitions, story action group, story action atoms. See Figure 1. The story nodes form the corner marks of the story and store the state of the whole scene. Story nodes are connected by story transitions, each consisting of one or multiple story action groups. Each story action group stores the scene changes relative to its preceding action group (or story node).

The story authoring process contains two steps: a story recording process and a story editing process. The outcome of this recording process is a raw prototype of a story told through volume visualization. In the story editing step, this raw story is refined until the final story outline is reached.

This process presents a volume visualization following the storytelling model. And the key feature is interaction, including viewing interaction, representing interaction and data interaction. Wohlfart and Hauser also discuss the paradoxical integration of storytelling and interaction, also called the narrative paradox.

Figure 1 shows an image sequence taken from a sample linear volumetric story visualized with their prototype. The distinct story nodes refer to the key events in the story, which provide an overview first, then details on specific features in the dataset, and at the end a conclusion made by the story author. The necessary story transitions are represented as orange arrows from one story node to the next and are animated in the prototype application. The story consumer may take over some story parameters (e.g., camera angle) already during playback or at the end of the story to further investigate the dataset.

This story guides the observers through the visualization, puts the contained visual representations into context with each other and finally introduces them to important features in the data. See Figure 6. Wohlfart is based on previous work of volume visualization  and interactive visualization and combine these concepts together to develop a storytelling model for volume visualization.

Geological storytelling is a novel graphical approach for capturing and visualizing the reasoning process that leads to a geological model. Lidal et al. present a sketch-based interface for rapid modelling and exploration of various geological scenarios. The authors present a concept that handles sketching processed over time and a novel approach for externalizing the mental reasoning process. The process can be presented and evaluated. The geological storytelling model contains three main parts. See Figure 7.

The canvas is a sketch-based interface where the geologist can draw the geological story on a 2D seismic slice backdrop, utilizing a pen and paper interaction style. The StoryTree is a tree graph representation of all the geological stories, each with its own subtree of story nodes. Individual story nodes can be selected for editing in the canvas. One or more complete story trees can be selected for playback or comparative visualization in the InspectView. The InspectView serves two purposes. First, it is a view where a story can be played and evaluated. In addition, multiple stories can be played synchronized for a side-by-side visual comparison.

The data-based story is recorded in SketchStory as a sequence of charts in XML files. The charts are linked with specific sketch gestures. The presenter draws an example icon and then draws a sketch gesture for chart invocation. Sketchstory recognizes the gesture and creates the corresponding chart. Lee et al. is based on previous work for storytelling of information visualization and sketch-based interaction, and develops the SketchStory system to enhance storytelling in a presentation.

Lidal et al. is based on a previous storytelling model for scientific visualization and develops a storytelling model for geological visualization.

Lee et al. present SketchStory, a data-enabled digital white board to support real-time storytelling. It enables the presenter to stay focused on a story and interact with charts created during presentation. See Figure 8.

Storytelling is one of the most impactful ways to teach, learn, and persuade. Lundblad and Jern present geovisual analytics software with integrated storytelling. It can be applied to large spatial-temporal and multivariate data through dynamic visual user interfaces.

Using a scatter plot matrix gives the analyst a good overview of all correlations between the selected indicators. The analyst can use the scatter plot matrix as an overview and then steer the scatter plot for interesting detailed combinations over time. See Figure 9. The distribution plot presents a special visualization technique that displays the variation within individual European countries. The Motala River map is visualized for different stories divided into different layers, such as a glyph layer, stream layer, polygon layer and background map layer. It shows the local and total water flow, and water path from source to ocean.

Lundblad and Jern is based on the previous work of the storytelling concept and work of web-based geovisual tools, integrates storytelling with geovisual analytics software.

3.3. Authoring-Tools for Parallel Storytelling

In this category of literature, authors create stories in parallel. In other words there may be multiple authors working in parallel i.e., simultaneously for the final outcome. This is opposed to a single author as in the previous subsection.

GeoTime events are recorded in x, y, t, coordinate space. This is used in observation analysis and can make a major contribution to a storytelling model. Eccles et al. presents the GeoTime stories prototype that combines a geo-spatial map with narrative events to produce a story framework. See Figure 10. This system provides functions for simple pattern detection in simultaneous movement activity. These functions look at possible interactions between people within the narrative, the speed at which they travel, and the type of location that they visit. Narrative text authoring enables the analyst to create and present stories found within the data. The story window displays this data as well as discovered patterns. The system enables multiple stories to connect together if they follow a linear flow. Also simultaneous narratives can be shown in a single image for a direct comparison.

This system uses a similar approach to Sense.us. Instead of using a blog-type discussion workflow for adding text, Geotime is designed for authoring a single story and annotations are integrated into the data itself.

The CodeTimeline visualization by Kuhn and Stocker enables developers who are new to a team to understand the history of the system they are working on. Designed to show a development team's tribal memory, the software offers a partial replacement for exhaustive documentation. See Figure 11.

A collaboration view presents visualizes code ownership and historical patterns in collaboration. A sourcecloud flow view presents a word cloud of added and removed vocabulary between software releases. Lifetime events can be added by users as a frame of reference in each of the visualizations. This method of linking also enables new users to learn more about the history of the software development. These events can be include anything from email threads to pictures of the team during work.

Prior to Kuhn and Stocker, Ogawa presents "software evolution storylines" and "Code Swarm", which focus on the interactions between developers on projects but do not focus on telling a story about the software history. Codebook, a concept presented by Begel et al., outlines a social network that connects software engineers with their shared code base. It encourages interaction with their code and others, enabling a broader understanding of the project they share with other developers.

User Engagement

The literature in this category addresses an important but less developed research topic, namely user engagement. In other words, who do we engage with storytelling and how can we engage an audience?

Mahyar et al. address how prior research in different domains define and measure user engagement. They discuss existing frameworks for engagement from other related fields and propose a taxonomy based on previous frameworks for information visualization.
They present five levels of user engagement in information visualization. See Figure 12.

1. Expose (Viewing): the user understands how to read and interact with the data.
2. Involve (Interacting): the user interacts with the visualization and manipulates the data.
3. Analyze (Finding trend): the user analyze the data, finds trends, and outliers.
4. Synthesize (Testing Hypotheses): the user is able to form and evaluate hypotheses.
5. Decide (Deriving Decisions): the user is able to make decisions and draw conclusions based on evaluations of different hypotheses.
   

Their work is based on previous work of Bloom's taxonomy and adapts it to information visualization.

4.1. User Engagement for User-Directed Visualization

The literature in this subsection focuses on interactive, user-driven visualization for user engagement. Engagement specifically focuses on each user's investment in the exploration of a visualization. Boy et al. use low-level user interaction e.g., the number of interactions with a visualization that impact the display to quantify user engagement. They present the results of three web-based field experiments, and evaluate the impact of using initial narrative visualization techniques and storytelling on user-engagement with exploratory information visualizations. The main contribution of their work include: the design of three web-based experiments on user-engagement information visualizations. They hypothesize narrative elements should effectively engage the user in exploration of data and analysis the result. They conclude that storytelling does not help engage users in visualizing their experiments.

Boy et al is based on previous work on narrative visualization and user-centred metrics. The negative outcome of their study clearly indicates that more future work is needed to investigate whether or not storytelling increases user engagement.