Storytelling and Visualization
Completion requirements
Storytelling has always been an effective way of conveying information and knowledge. In the field of visualization, storytelling is rapidly gaining momentum and evolving cutting-edge techniques that enhance understanding as storytellers are integrating more complex visualizations into their narratives. Read this article to explore the survey of storytelling literature in visualization and present an overview of the common and important elements in storytelling visualization.
Introduction and Motivation
"We believe in the power of science, exploration, and storytelling to change the world" - Susan Goldberg, Editor in Chief of National Geographic Magazine, from "The Risks of Storytelling", October 2015.
"In a world increasingly saturated with data and information, visualizations are a potent way to break through the clutter, tell your story, and persuade people to action" - Adam Singer, Clickz.com, "Data Visualization: Your Secret Weapon in Storytelling and Persuasion", October 2014.
Throughout history, storytelling has been an effective way of conveying information and knowledge. In the field of visualization, storytelling is rapidly developing techniques that enhance understanding. By visualization, we refer to "interactive data visualization" as defined by and described extensively by Ward et al.. We note that there are many references to interactive visualization throughout the survey. For example, the term "interactive", appears over 40 times throughout the text. We also note that animated transitions are a major theme in the survey and receive their own section in the text. See Table 1. Many communities have commented on the importance of storytelling in data visualization. Storytellers tend to be integrating complex visualizations into their narratives in growing numbers.
Table 1. Our classification of the storytelling literature. The y-axis categories fall into who-authoring-tools and user-engagement, how-narrative and transitions, why-memorability and interpretation. See Section 1.2 for a complete description.
| Linear | User-Directed/Interactive | Parallel | Overview | ||
|---|---|---|---|---|---|
| Who | Authoring-Tools | Gershon et al., 2001 Lu and Shen, 2008 Cruz et al., 2011 |
Wohlfart, 2006 Wohlfart et al., 2007 Lidal et al., 2012 Lee et al., 2013 Lidal et al., 2013 Lundblad et al., 2013 Fulda et al., 2016 Amini et al., 2017 |
Eccles et al., 2007 Kuhn et al., 2012 |
|
| User Engagement | Figueiras, 2014 Boy et al., 2016 Borkin et al., 2016 |
Mahyar et al., 2015 |
|||
| How | Narrative | Hullman et al., 2013 Hullman et al., 2013 Gao et al., 2014 Amini et al., 2015 Bach et al., 2016 |
Viegas et al., 2004 Hullman et al., 2011 Figueiras, 2014 Figueiras, 2014 Nguyen et al, 2014 Satyanarayan et al., 2014 Gratzl et al., 2016 |
Akashi et al., 2007 Fisher et al., 2008 Hullman et al., 2011 Bryan et al., 2017 |
Segel and Heer, 2010 Lee et al., 2015 |
| Static Transitions | Ferreira et al., 2013 | Robertson, 2008 Chen et al., 2012 Tanhashi et al., 2012 Liu et al., 2013 Ferreira et al., 2013 |
|||
| Animated Transitions | Heer et al., 2007 Liao et al., 2014 |
Bederson and Boltman, 1999 Akiba et al., 2010 Nagel et al., 2016 |
|||
| Why | Memorability | Bateman et al., 2010 Borkin et al., 2016 |
Saket et al., 2015 | ||
| Interpretation |
As
contributions, we present a survey reviewing storytelling papers in
visualization and present an overview of the common and important
elements in storytelling visualization. We also describe the challenges
in this field and present a novel classification of the literature on
storytelling in visualization. Our classification highlights both mature
and unsolved problems in this area. The benefit is a concise overview
and valuable starting point into this rapidly growing and evolving
research trend. Readers will also gain a deeper understanding of this
rapidly evolving research direction.
Definition and Storytelling Elements
A
story can be defined as "a narration of the events in the life of a
person or the existence of a thing, or such events as a subject for
narration" or "a series of events that are or might be narrated". Storytelling is a popular concept that is used in many fields, such
as media, education and entertainment. Storytelling is a
technique used to present dynamic relationships between story nodes
through interaction. According to Zipes, storytelling can involve
animation and self-discovery, incorporating models, ethical principles,
canons of literature, and social standards. In education, a storyteller
can improve and strengthen the literacy of students. Also, the
storyteller can engage audiences so they feel a desire to read, write,
act, and draw. Audience members can learn to express themselves
critically and imaginatively with techniques they may learn from the
storyteller or teacher.
In the context of the visualization
literature. Lee et al. argue that "the community has been using the
term ‘storytelling' in a very broad way without a clear consensus or
discussion on what a visual data story encompasses". They state that a
visual data story includes a set of story pieces. Most of the story
pieces are visualized to support one or more intended messages. Story
pieces are presented with a meaningful order or connection between them
to support the author's high level communication goal.
Furthermore no
agreed definition of "visual data story" has yet emerged in the
visualization literature. For a full-length 6 page discussion on
this topic, we refer the reader to Lee et al.
Classification of Literature and Challenges in Storytelling and Visualization
Although
storytelling has been developing in other fields for years,
storytelling is a relatively new subject in visualization. As such, it
faces many challenges. In this survey we have extracted the fundamental
characteristics of storytelling both as an entity and as a creative
process. Our literature classification is based on the logical notions
of who are the main subjects involved in storytelling for visualization
(authoring tools and audience), how are stories told (narratives and
transitions), why can we use storytelling for visualization
(memorability and interpretation). From these characteristics we have
then developed the following dimensions which are common to storytelling
in visualization.
User-engagement: Engagement is about the audience and also
concerns why we use storytelling. How can we ensure that the message
comes across to the audience? Can we measure engagement?
Narratives:
Narrative concerns how an author tells a story. Narrative structures
include events and visualization of characters. Narrative visuals
contain the transition between events. This entails, "Using a tool to
visually analyze data and to generate visualizations via vector graphics
or images for presentation", and then deciding "how to thread the
representations into a compelling yet understandable sequence".
Transitions:
Transitions are about how authors may tell the story. Transitions
seamlessly blend events within a story and are key to its flow.
Successful transitions vary actions as little as possible to strengthen
overall coherence. Transitions in visualization can be either dynamic or
static.
Memorability: Memorability addresses why authors present
data in the form of a story. Memorability is an important goal of
storytelling. A good visualization technique draws the viewer's
attention and increase a story's memorability.
Interpretation:
Data interpretation refers to the process of critiquing and determining
the significance of important data and information, such as survey
results, experimental findings, observations or narrative reports.
When
examined in the context of storytelling in visualization each dimension
raises interesting questions: Are current storytelling platforms taking
into account the role of the author and supporting the authorship
process? What forms of narrative structures and visuals best apply to
storytelling in visualization? Are static transitions or dynamic
transitions more effective for storytelling in visualization? Can
visualization increase the memorability of data information or
knowledge? Does storytelling and visualization aid with data
interpretation? What is the most effective way to engage an audience?
Data preparation and enhancement is another challenge for which there is
currently no literature. Thus we include it as a future research
direction but not in our classification.
Starting from the logical
notions of who, how, why, and these open questions we have chosen these
dimensions to form the basis of our literature classification on
storytelling in visualization. See Table 1. It is important to note that
some papers address multiple topics in Table 1 and in our
classification. We placed papers by what we determined to be the main
focus of the paper. This is very useful for obtaining an overview.
However some papers address more than one theme, e.g., authoring tools
and narratives.
Classification of Literature: The Second Dimension
- Linear: A story sequence path in linear order is prescribed by the author.
- User-directed path: The user selects a path among multiple alternatives or creates their own path.
- Parallel: several paths can be traversed or visualized at the same time.
- Random access or other: There is no prescribed path. There is currently no literature prescribing random order. Therefore we replace this with a column called "overview".
Literature Search Methodology
We search both the IEEE and ACM Digital libraries for the terms "storytelling", "narrative visualization", "memorability", "transitions in visualization", "user-engagement", and various combinations of these phrases. We focus primarily on the IEEE TVCG papers. We check the references of each paper and looked for related literature on storytelling. We also search the visualization publication data collection for these major themes in visualization and storytelling. Google scholar is also used as part of our search methodology.In summary, our literature search includes:
- IEEE EXPLORE Digital Library
- ACM Digital Library
- Visualization publication data collection
- the annual EuroVis conference
- the Eurographics Digital Library
Several other papers were discovered by looking at the related work section of the papers we found.
Survey Scope
The
storytelling visualization papers summarized in this survey include the
subjects of scientific visualization, information visualization,
geo-spatial visualization, and visual analytics. In order to manage the
scope of this survey, storytelling papers from other fields are not
included, such as:
Virtual reality and augmented reality: For
example, Santiago et al. present "mogre-storytelling" as a solution
to interactive storytelling. This tool provides different
functionalities for creating and the customization of scenarios in 3D,
enables the addition of 3D models from the Internet, and enables the
creation of a virtual story using multimedia and storytelling elements.
Education:
For example, Cropper et al. address the extent of how scientific
storytelling benefits our communication skills in the sciences, and the
connections they establish with the information itself and others in
their circle of influence.
Gaming: Alavesa et al. describe the
development of a small scale pervasive game which can take storytelling
from camp-fire sites to modern urban environments.
Multi-media and
Image Processing: For example, Chu et al. describe a system to transform
any temporal image sequence to a comics-based storytelling
visualization. Correa and Ma present a narrative system to generate
dynamic narrative from videos. Image processing falls outside the
scope of this survey. Video processing also falls outside the scope of
the survey.
Language processing: Theune et al. develop a
story generation system. It can create story plots automatically based
on the actions of intelligent agents living in a virtual story world.
The derived plots are converted to natural language, and presented to
the user by an embodied agent that makes use of text-to-speech.
Details-on-Demand: The theme of details-on-demand
is included in the survey and is often used throughout the literature.
For example in Section 3.2. Authoring-tools for User-directed and
Interactive Storytelling, Figure 1 shows an image sequence taken from a
sample linear volumetric story. The distinct story nodes refer to the
key events in the story, which provide an overview first, then details
on specific features on demand. Again in Section 5.4 Narrative
Visualization Overviews, an Afghanistan nation-building development
project example shows an interactive geographic visualization with
details on-demand sliders that present the status of Afghanistan
nation-building development projects. In another example, the
Minnesota Employment Explorer shows how mouse-hover provides
details-on-demand, double-clicking an industry triggers a drill-down
into that sector while an animated transition updates the display to
show sub-industry trends.
Figure
1. The proposed method to author a story is to record the user's
natural interaction with the visualization software. This image shows
the process of the story creation by Wohlfart. Green annotations
represent user interaction and red annotations refer to internal system
processes. As soon as the software starts recording, a new story is
created and all interactions are logged.
This article is much more up to date than the article by Segel
and Heer which is already more than 7 years old now. The field has
evolved substantially since then. Also, the article by Segel and Heer is
not a comprehensive survey like this one. However it is a highly cited
paper and makes a big contribution to narrative visualization.
Figure
2. A table summarizing the visualization techniques used in each
storytelling paper. The papers are sorted alphabetically by the first
author's surname.
Table
| Scientific Visualization | Information Visualization | Geo-Spatial Visualization | |
|---|---|---|---|
| Authoring Tools | Wohlfart, 2006 Wohlfart et al., 2007 Lu and Shen, 2008 |
Gershon et al., 2001 Cruz et al., 2011 Kuhn et al., 2012 Lee et al., 2013 Fulda et al., 2016 Amini et al., 2017 |
Eccles et al., 2007 Lidal et al., 2012 Lidal et al., 2013 Lundblad et al., 2013 |
| Narrative | Viegas et al., 2004 Akashi et al., 2007 Fisher et al., 2008 Segel and Heer, 2010 Hullman et al., 2011 Hullman et al., 2013 Hullman et al., 2013 Figueiras, 2014 Figueiras, 2014 Nguyen et al., 2014 Amini et al., Lee et al., 2015 Bach et al., Bryan et al., 2017 Gratzl et al., 2016 |
Gao et al., 2014 Satyanarayan et al., 2014 |
|
| Static Transitions | Robertson, 2008 Chen et al.,2012 Tanhashi et al., 2012 Liu et al., 2013 |
Ferreira et al., 2013 |
|
| Animated Transitions | Akiba et al., 2010 Liao et al., 2014 |
Bederson and Boltman, 1999 Heer et al., 2007 |
Nagel et al., 2016 |
| Memorability | Bateman et al., 2010 Borkin et al., 2013 |
Saket et al., 2015 |
|
| Interpretation | |||
| Engagement | Figueiras, 2014 Mahyar et al., 2015 Boy et al., 2016 Borkin et al., 2016 |