During the last years, the use of 360° videos in journalism gets more and more common. Beside the New York Times, Blick or the Süddeutsche Zeitung also non-journalistic producers offer non-fictional content for a growing audience. The new technologies (360° cameras and ambisonic microphones) and the related emerging storytelling format carry the potential to immerse its audience to an (in journalism) probably unseen degree. A sense of presence and emotional reactions (resulting in increased empathy) characterize the experience. This study complements existing practitioners reports by presenting the insights from a project at a Swiss university. Over the course of nine weeks, the graduating class of a degree program in digital storytelling realized the publication of three non-fictional 360° videos. Two teachers, an external expert and a journalism researcher accompanied the seven students and their project. The study presents insights from a systematic analysis of the data gathered by the involved researcher conducting participatory observation, informal talks and formal interviews. The results consist of general experiences from the process and four identified opportunities and challenges working with the new format for non-fictional storytelling. Insights are being discussed from a practical and an ethical perspective. Implications for future research are provided.
Technological innovations have changed journalistic storytelling during the past decades (Pavlik 2000). Digital , mobile (Westlund 2013), interactive (Nip 2006; Hujanen and Pietikäinen 2004) and participatory (Deuze 2005), multimedia (Deuze 2007), gamified (Bogost, Ferrari, and Schweizer 2012), more visual (Wallace 2009; Bock 2011), (big) data-based or algorithm-supported (Coddington 2014) and social media-optimized (Hermida 2012; Hermida 2010) formats have evolved. The new formats changed working routines and working processes at the newsroom (Pavlik 2000; Steensen 2011; Deuze 2007), the relationship between journalists and users and journalists’ professional identity (Grubenmann and Meckel 2015).
Amongst the youngest media trends belongs virtual reality (VR) (Stale 2017). Even though the technology has characterized the gaming industry for years now (Jensen 2016), Nonny de la Pena was a pioneer working with VR in journalism back in 2012 presenting her work “Hunger in Los Angeles” at the Sundance Film Festival (Goodman 2012). VR refers to „immersive, interactive, multi-sensory, viewer-centered, three-dimensional computer generated environments and the combination of technologies required to build these environments“ (Cruz-Neira 1993). To create corresponding worlds is an expensive job – particularly for already economically stressed editorial teams (Polgreen 2014). In this context, 360° videos are a lower-hanging fruit opportunity (Hernandez 2016): 360° videos are easier to produce and reach a broader audience as users can watch these videos on their smartphone or a desktop computer whereas for VR experiences they need a headset (or Head-Mounted Display HMD).
The New York Times was the first major media brand pushing the technology. In 2015 they not only published their proof of concept “Walking New York”, they in fact delivered more than a million pre-assembled Google Cardboard sets (a cheap headset for watching 360° videos) to home delivery subscribers (Welsh 2015). In Mai 2016 they also included 300’000 of their most loyal digital customers, based on subscription length (Robertson 2016). Beside the NYT’s spectacular PR event, other media brands published first 360° stories: Frontline took its viewers in “Ebola Outbreak” into the heart of the Ebola crisis, the Times Magazine’s portrayed in “The Displaced” three children from South Sudan, eastern Ukraine, and Syria and The Associated Press partnering with the Los Angeles-based production company AMD released a series of VR films in February 2016. Also VICE or ABC News were among the first exploring the format. But ahead all was Gannett’s Des Moines Register with “Harvest of Change”, an interactive experience about Iowa farm families. Whereas the other publishers worked with 360° videos, Des Moines Register combined them with full VR, offering the experience for the Oculus Rift.
Compared to normal videos (or “flatties”), 360° videos increase user agency. Watching the video, the user controls the direction she’s looking: left, right, up, down, to the back, to the front, to the protagonist or away from her (see figure 1). The user is however fixed to one position and cannot move in space. This is a main difference compared to VR experiences where the user can walk around and influence the proceedings. And it’s also the reason why people speak of a rather passive immersion referring to the degree to which users “feel involved with, absorbed in and engrossed by stimuli from the virtual environment” (Palmer 1995). Compared to VR worlds in video games, you also don’t find direct interaction between the user and the protagonist in a way that users could react on protagonists’ actions and influence the course of the story or the experience (that said, technology developments continue to provide new opportunities for content, including greater interactivity (Watson 2017)). Due to these reasons, some experts even ask into question if 360° videos are VR at all (Adams 2016).
Despite these limitations, the use of 360° videos – which represents the majority of published work in what we call VR or Immersive Journalism – still has a great influence on user experience and means interesting implications for the storytelling. Beside these changes, editorial teams are further confronted with new challenges concerning the purchase costs and use of new technology, new work processes and increased production costs, and the need for new distribution channels and business models.
Interested particularly into new storytelling and production, we at ZHdK CAST audiovisual media decided in Summer 2016 to realize a non-fictional storytelling project in 360° with our graduating class. Over the course of nine weeks (from November 2016 to mid January 2017), our students, a class of 7 people, developed and produced a story from the scratch. The students have been supervised by two teachers with a practical background in media production and were advised by an external 360° video expert for questions regarding the equipment. Additionally, I accompanied the project applying participatory observation to research the group’s experience with the new technology and the related storytelling technique. As a follow-up of the project, I interviewed all students and the two teachers to learn from their experience in the project. In this paper I describe our insights and reflect on it from a journalistic point of view. Our study takes place at the intersection of journalism and design.
Part of the design department, our focus lies on the production but I will integrate some ethical thoughts.
Our study was guided by the following research questions:
- How to tell a non-fictional story in 360°?
- What potentials offers 360° video for non-fictional storytelling?
- What are the challenges that come with producing non-fictional story in 360° degrees?
- What are our recommendations for producing non-fictional stories in 360° in the future?
In a first step, I’m going to give a short overview of what scientific research says about how technology changed journalistic work and storytelling during the past years. I will then give a brief overview of what we know from practitioners and their experiences and how they think the new technology will change journalism. After a short description of our project and the methodology I applied in the study, I will afterwords present its findings and discuss it in a final chapter.
Immersive Storytelling in Journalism
Until today, there are four (practice-based) reports (Doyle, Gelman, and Gill 2016; Watson 2017; MacVey 2015; Aronson-Rath et al. 2015), several interviews with journalists and professional articles, and a few academic papers (Sirkkunen et al. 2016; de la Peña et al. 2010) published yet we can gain insights from.
One thing they all agree on, are the high production costs mainly of VR but also of 360° stories (Polgreen 2014). Equipment (hard- and software) is an investment (see figure 2) but technology is rapidly evolving. However, lacking experience and the time-consuming need for trail and error to develop new forms of storytelling as well as the time for stitching and rendering are additional costs compared prohibitive for many small and medium-sized news organizations (see Doyle, Gelman, and Gill 2016; Sirkkunen et al. 2016).
Due to their lack of experience, pioneering brands, as The New York Times, The Des Moines Register, the Huffington Post, ARTE or the Süddeutsche Zeitung have been collaborating with early VR specialists as Within (formerly Vrse), Jaunt VR, Here Be Dragons (formerly Vrse.works), RYOT or RE’FLEKT. Today, some of them have the skills in-house in multidisciplinary teams responsible for 360° productions (e.g. the New York Times, the Guardian or the Swiss BLICK). “These teams bring together editorial leads with software developers and product owners, designers and motion graphics producers with print journalists and photographers, and in some cases also have dedicated business development support to manage tech partnerships” (Watson 2017, 26).
Ease of use and costs of purchase are also important reasons on the user side: 360° videos can be watched in apps on users’ smartphones or through the browser-version of YouTube or Facebook, who made the publication of the format possible in March respectively in September 2015, on any device. For a real VR experience, users however still need access to a headset (for two examples see figure 3). This circumstance of course gave rise for 360° videos and made them accessible to a broader audience – which is important for the format’s acceptance and dissemination.
Both, the considerable higher production costs and limited audience are the reason why the use of 360° videos exceeds the work with full VR in journalism today by far. And even the promise that purchase costs for headsets will come down won’t change this fact too fast.
To distribute their 360° videos, some publishers offer their own app (e.g. The New York Times, the Süddeutsche Zeitung, The Guardian or BLICK) and/or integrate the YouTube video on their website. A third option is to distribute the video additionally over a partner’s channel (e.g. abc NEWS on JAUNT VR). Even though there are some trials to produce (hard) news coverage (e.g. VICE NEWS VR or the foreign reporting of the Swedish newspaper Dagens Nyheter), in journalism the technology is merely being used for features and documentaries. Live VR also offers opportunities. Some organizations have already begun to explore these – as the coverage of events from the Presidential Campaign – but opinions are split on how significant live VR would be in the near future (Watson 2017). Concerning length and quality, (Watson 2017) differentiates two main types of content: 1) Documentary-style 360° films, usually five to 15 minutes long, with high production values to view them on VR headsets, usually delivered via apps and 2) short-form 360° videos under two minutes, generally intended for browser viewing, usually produced relatively quickly and cheaply for distribution on social channels (YouTube 360/Facebook).
Beside publications from editorial publishers, there are however also commercial producers and even NPO and NGO publishing non-fictional stories (e.g. JAUNT, Discovery VR, the United Nations or the Bill & Melinda Gates Foundation). For those of course the empathic potential is of particular interest.
Another main topic being discussed it the technology’s immersive potential: VR is seen as the “empathy machine”. First brought up by film maker Chris (Milk 2015) in a TED Talk, it has probably been the perceived singularity of the format most discussed (see (Naylor 2017; Bloom 2017; Constine 2015; Alsever, 2015). These articles are celebrating the new format as a game changer. Authors describe the experience as walking “in someone’s shoes, but [seeing] the world through their eyes” (Constine 2015) or as “an intimacy that traditional cinema can’t [offer]” (Robertson 2014). For producers the potential to provide a “first person experiences of the events or situation” (de la Peña et al. 2010) as the Syrian war, becoming blind or living in a refugee camp is a crucial motivation.
The technology’s immersive potential unfolds from creating a sense of presence, “a psychological state in which virtual […] objects are experienced as actual objects […]” (Lee 2004, 37). Spatial Presence, as one of three dimensions of Presence (beside involvement, and realness) (Schubert, Friedmann, and Regenbrecht 2001), describes the aspect of “being there” (Steuer 1992) which seems to be important for VR Journalism (Garling, n.d.; Bilton 6ADAD). (Sirkkunen et al. 2016) call (Spatial) Presence and immersion the two key concepts to be supported for working with VR in journalism. The two concepts are related in the way that “researchers in the field of Presence describe features that give rise to Presence as immersive properties of the media” (Wirth et al. 2007, 496).
Referring to these concepts, Paisley Smith, an assistant from Nonny de la Pena, describes the effect as “a disconnect from the world you know, and you’re immersed somewhere else” (cited in Malmo 2014). Because in VR users live the events in such a personal experience, their empathetic response to what takes place is much stronger (Malmo 2014). This can be seen as a potential for journalism to get people involved (as Dolnick describes it) but also as a risk for misuse: Does the story convince by reasons or by getting people emotionally involved? How to deal with journalistic and ethical standards?
“We’ve written hundreds of stories about the refugee crisis, but what we kept hearing from people was that they had become somewhat inured to it. But with this new medium, they were making eye contact with these kids and it shocked them in a profound way.” (Sam Dolnick from the New York Times cited in (Bilton 6ADAD)).
Paul Bloom brings up a counterpoint and argues that there are feelings one “can’t tap into […] by putting a helmet on your head” (referring to VR headsets) as the duration of the experience or if your forced into it make them particular (i.e. becoming paralyzed or being tortured) (Bloom 2017). (Watson 2017) argues that the discussion about empathy has moved on: “every medium can elicit empathy, but its power to do so lies in the skill of the storyteller” (21). Jason Farkas from CNN wonders if an over-emphasis on empathy in the early days of VR experimentation limited the range of content explored (cited in Watson 2017, 21).
It’s interesting to note that authors do not differentiate between 360° videos and true VR cases emphazising the empathic potential, discussing for example Nonny de la Peña’s VR experience “Project Syria“ (VR) and Spike Jonze’s “VICE News VR: Millions March” (360° video) in the same article (version: speaking of a “virtual reality film” (Alsever, n.d.)). This however seems important considering the expected differences in the immersive potentials of the formats – systematic users studies are though still missing.
There are other ethical concerns as how to protect privacy or get consent in a spectrum that makes it difficult to isolate coverage? Or that immersion could be too realistic or manipulative, creating uncomfortable or misleading experiences (Doyle, Gelman, and Gill 2016). And then there is an interesting question related to producing in 360°as it’s unnaturally staged in a manner that changes the authenticity of the story (Doyle, Gelman, and Gill 2016): producers have to get off the set while recording, interviews need a special setting (if the interviewer should not be visible) and interviewers find themselves on an abandoned set.
This leads us to another challenge for journalists working with the format: the provision of information and reasons. If the user controls her viewing direction, how to design the storytelling and narrative structures without using the editing conventions based on film and TV productions? 360° videos are basically (a sequence of) single shots that the users can explore as they want. Within a shot, there are only limited cues to direct users’ attention (as audio, light or abrupt movements but no framing the shot or zooming in and out) (Mufson 10ADAD; Doyle, Gelman, and Gill 2016) and also camera movements are different (reduced) (Sirkkunen et al. 2016). (Doyle, Gelman, and Gill 2016) propose to apply gaming strategies. Jaunt VR combined team of three directors, from film, from theatre and from VR for the production of “Ashes to Ashes“. Over all, storytelling has to be developed from the scratch with increasing user agency meaning a ”radical shift” for the journalistic mindset (Sirkkunen et al. 2016, 299).
So, how to make sure the user doesn’t miss important bits of information? Or is it rather about self-directed exploring, user agency and the related experience than about the controlled provision of information? Related to this question is the one about content: which topics, stories and places work in VR? Max Boenke, Head of Video at Berliner Morgenpost, is worried that more and more news organizations are using the format for stories that are not interesting what will keep people away from watching it in his eyes (cited in Watson 2017, 33).
Based on the last year’s experiences, there are already some rules producers can stick to:
- “[You] need to think in a fresh way: You can’t apply what you know about documentary film making and just put it on to VR, it doesn’t work. You have to think more like a UX designer or about human-centric design or immersive theatre”(Jessica Lauretti from RYOT cited in Watson 2017, 23).
- “[L]ocation-based stories work particularly well: Stories where the place is significant in the story, where it is almost the character in the story” (Marcelle Hopkins from the New York Times cited in (Watson 2017, 22f.).
- Marc Jungnickel from Bild (cited in Watson 2017, 22) says “two good starting points are stories that enable people to ‘Be them or be there’: ‘Be them’ provides visceral experiences such flying aeroplanes and jumping off cliffs. ‘Be there’ gives you unique access to special locations”.
- The New York Times deliberately chose to not use reporters on screen and make this different from television news reporting (Marcelle Hopkins from the New York Times cited in Watson 2017).
- Matilda Hanson from the Dagens Nyheter stresses that VR is as much an audio as a visual experience (cited in Watson 2017, 23). In VR (even in 360° video) producers can work with Spatial Audio. Google describes Spatial Audio in their documentation for developers the following: “Spatial Audio is a powerful tool that you can use to control user attention. You can present sounds from any direction to draw a listener’s attention and give them cues on where to look next. But most importantly, Spatial Audio is essential for providing a believable VR experience.”
So, somehow related to all of this, (Doyle, Gelman, and Gill 2016) are asking: “What will happen when the novelty of VR wears off, and whether the quality of the storytelling and the VR experience will bring people back to look at the content on a regular basis” (p. 6). Our study contributes insights to both aspects, storytelling in VR and technological aspects that add up to what Doyle, Gelman, and Gill call the “VR experience”. What also will contributes to the question if people are willing to watch journalistic VR pieces in the future, is to what degree VR is going to be replaced by Augmented Reality (AR), a technology to which experts and investors ascribe by far higher (business) potential than to VR due to the higher flexibility and applicability of the technology. This discussion will however not be part of this paper but will be followed with much interest.
Our study took place in the context of the transmedia seminar with our graduating class. At the Zurich University of the Arts, they run through the specialization CAST audiovisual media and are trained to become digital storytellers. At the core of their studies we teach them fictional and non-fictional storytelling, video production and animation (moving image) – for web-publication. Our program is part of the design department. That said, our students not just become digital storytellers in agencies or corporate communication but also in journalism (or they get self-employed).
In the final term of their studies, the students get the chance to realize their last bigger project as a class. Always into new forms of storytelling and exploration, the graduating class of 2016 produced the first Swiss short film in 360° “Gegen die Regeln” (against the rules). Building on this experience, in 2017 we decided to stick to the technology but to produce in non-fiction. The students, a class of 7 people, were open to chose the topic and were guided by two teachers of whom one was experienced in working in 360°. The class received further support by a well-known professional 360° producer, being available for questions concerning the equipment. We further established a cooperation with students from the specialization composition and sound design at the school as producing in 360° was an issue there, too. For these students, the cooperation was voluntary. Over all, nine students took the chance to gain experiences and became involved. Me as a research assistant of our faculty accompanied the project from the beginning.
Over the course of nine weeks (from November 2016 to mid January 2017), the students developed and produced the story from the scratch. The seminar was organized in compulsory meetings in the class and autonomous working where students had to organized themselves but teachers were around or available for questions via email or on skype. All of the students had some previous experience producing in 360° from a precedent seminar. The students decided to go for the topic angst and represent it in three 360° episodes (i.e. Arachnophobia, Generalized Anxiety , and Paranoia). Overwhelmed by the depth and width of the topic they discovered with their research, they in the course of the project decided to present the experience on a website supplemented by written information and normal videos. The result of their work can be seen here (in German). You’ll find the 360° videos in the section “ERLEBEN” (experience). The publication of the website was guided by Facebook and Instagram posts as well as PR activities. The videos have further been distributed on YouTube and Facebook.
Data gathering and analysis
Interested in the systematic analysis of the potentials and challenges that would evolve during the project, we decided that I would accompany the whole project from a research perspective. Beginning with the project and my research, I wasn’t sure yet about my role in the project itself and, related to that, if I should apply passive or participatory observation for my data collection. However, rather early I realized that, as I knew all the involved people and the work they were doing, it was hard for me to not become involved. And also for the others, the situation wasn’t easy: having me around and knowing about my knowledge of the topic, they wouldn’t accept a passive role. In the end, we found a way to deal with the situation by having me involved into discussions and conceptual decision but not into the actual production and the process management what makes me speak of participatory observation.
In addition to my observations and informal talks I had with all the involved people during the project, I interviewed all of them plus the head of group of the involved sound designers about two weeks after the publication of the story by the end of January 2017. The nine interviews lasted between 17 and 97 minutes and were all recorded. The interview data supplemented my notes from the observations and the informal talks. For the analysis of the data I applied inductive, thematic analysis (Mischler 1986). In the presentation of the insights of our study, I will differentiate between outcomes of the interviews and my personal view.
My data collection and the data analysis was guided by the following research questions:
- How to tell a non-fictional story in 360°?
- What potentials offers 360° video for non-fictional storytelling?
- What are the challenges that come with producing non-fictional story in 360° degrees?
- What are our recommendation for producing non-fictional stories in 360° in the future?
I will now present the insights from the study, structuring the results by differentiating between challenges and potentials the involved people experience during and after the production of the transmedia story “angst360”.
In a first step, I will now describe two rather general but important experiences from the process. I will add some notes on the division of work between the students during the process and a brief description of the three 360° videos. In a third step, I will discuss four opportunities and four challenges we identified working with the new technology and the emerging storytelling format.
Define the topic
Starting the project, it was the students’ first task to define a topic they wanted to tell a story about. It turned out that students experienced this decision as extremely difficult. In fact, they’ve spend so much time on it that it put a lot of pressure on them realizing the story in the remaining time. Observing their struggling, the two teachers advised them to start thinking from places: Where do you find interesting places or rooms? Which places or rooms seem promising from a 360° perspective? Why does it make sense to look around, to explore the place?
Starting from there, they would advise them to move on to the topic and the protagonist(s): Which topic lays at the core of the story? Why should users watch the video? And: Who are good storytellers? What is she keen on talking about? Where can she bring us to? What’s her motivation to talk about a given topic?
They illustrated their advice with figure 4.
It’s interesting to realize that these guiding questions are quite similar to the two starting points Marc Jungnickel from Bild gives at hand: “Be them or be there” (cited in Watson 2017, 22).
Relying on the advice, the students decided to go for a story about bunkers dropping six other ideas. Exploring the topic bunker, the students went round in circles. After many discussions and close to giving up, they realized that the story they wanted to tell wasn’t about a place but about an emotion. It became apparent that bunkers would indeed make a place people want to explore but that the interesting story behind was the fear that goaded the Swiss to build so many bunkers on their territory. That’s how they found their way from an interesting place to their story: angst.
Reflecting on the difficulties our students experienced in this process, we soon realized that these were probably related to the artificial nature of the task. Because in journalism you would of course not start with the storytelling format you’re intend to use but exactly the other way around: You would start from the story to be told and would then, in a given case, realize that 360° video would make the perfect format to tell this story (see figure 5).
That not all publishers show and showed respect for this maxim exploring the new technology may be the reason for Max Boenke’s (Head of Video at Berliner Morgenpost) fear that bad stories will keep people away from 360° stories (cited in Watson 2017, 33). The professional who advised our students in the project told them about a simple rule guiding the decision if 360° video was the best choice telling a given story: If your watching the 360° video and the story makes totally sense watching it without changing the perspective a single time it should not have been produced in 360° but in a different format.
After their decision for the topic angst, the students immersed themselves into research. All students have been involved into this task. They however turned towards different aspects of the topic to share the work. From their analysis of the topic, the structure of the story emerged (see sketch in figure 6).
Involve the sound designers
Forced by the two teachers, we had an early meeting with a group of sound design and composition students from our school. We’ve visited them in a class and told them about the project, proposing a collaboration. The ones interested in contributing to the project wrote their names on a list and got contacted for a first meeting. Nine students offered their voluntary help and got involved. Starting from there, we kept them posted about our progress to provide them with an idea of the story as soon as possible. They appreciated the early briefing, bemoaning only late involvement in most other projects limiting the chance to give advice from their professional perspective and develop their own understanding of the story.
One of the sound designers who had some previous experience with 360° projects, stressed the need (and wish) for experimentation – which is only possible with enough time at hand. He told us that professionals would haven even less experience with audio in 360° videos (i.e. Spatial Audio) than with the visual concerns and that standards wouldn’t be established yet – making it an interesting time to realize projects. Spatial Audio offers the possibility to attach a certain sound to a (moving or stationary) object or protagonist in the video. The user will be able to localize the sound in the video surrounding her. The producer can also record 3D audio at the scene with a special microphone (Sennheiser AMBEO® VR Mic). The microphone combines four capsules, recording the sound in the room or the environment at one point resulting in a spherical Ambisonics-Sound (Sennheiser).
Despite their early integration into the project, the sound designers in the end still had to work over the Christmas Holidays due to a (too) tight schedule. Because our students had lost too much time defining the topic of the story, the time for the production run short. This also limited the room for experimentation.
The three 360° videos
To present the topic angst, our students developed the website www.angst360.ch where they published different texts, nine videos and three 360° videos (and photos of the interviewees). Not all students got involved into all jobs during the process. The division of work helped them to concentrate on a task and work more efficient. All students have been involved into research but some more deeply. These two woman also took care of most of the organizing (interviews and locations). A small group shot the normal videos (all interviews), another group shot most of the 360° videos. However, all students got involved into a least one 360° shoot. Two students looked after the whole post production (stitching, cutting, color grading etc. and the collaboration with the sound designers). One was in charge of the website, another of the communication.
Each of the three 360° videos offers an experience related to a particular form of fear: Arachnophobia, Generalized Angst, and Paranoia. In the the short film about Arachnophobia the user meets Ursina who suffers from Arachnophobia since she was a young girl. In the film she visits the spider lover Bastian in his cellar with 130 terrariums and more than 600 animals. In the film about Generalized Angst the user gets to know Reto, the owner of the shop “SicherSatt” for emergency supply. He tells the story about his business idea and presents different products. In the third film on Paranoia, the user finds herself in a dark room with screens, cameras and strange people watching her. A voice off-camera talks about omnipresent surveillance. The videos last 05:37, 05:57 and 04:59 minutes.
Beside these experiences during the process, I will now discuss four identified opportunities and challenges in more detail.
Opportunities of 360° non-fictional storytelling
Evoke emotions and empathy
Of course, VR is not the only format or medium that establishes closeness and evokes empathy. However, the immersive effect (to be at that place, to feel close), respectively the experience of presence fosters emotional reactions (Västfjäll 2003). The series is based on the experience of fear. In all three videos our students tied to evoke a particular form of fear. The students tried to use the new format’s immersive potentials to (roughly) induce the feelings users are reading/hearing about. They stoke corresponding feelings with techniques the new format gives us at hand (i.e. sound and/or movements in the user’s back, placing the tarantula on the camera in combination with a close/sudden sound, placing the user in a dark room with flickering light). It’s a lack of the study that we couldn’t do any user studies to learn about the effect of these techniques.
Trying to evoke different forms of fear, it’s an interesting fact that VR is actually being used for the investigation of pathological processes in mental disorders, especially anxiety disorders as VR can actually induce emotional reactions (Diemer et al. 2015).
As the students used the 360° videos in combination with other formats, it’s interesting to see how they applied them: The mainly used text but also the normal videos to provide information and used the 360° videos for offering experiences, emphasizing and portraying people. In the combination, they tried to allow users to put themselves in someone else’s state or situation and create a certain atmosphere that supports the reception of the information provided on additional channels (text and video). I remember how RYOT used the same mechanism to (try to) make people spend money to NGO and NPO using a “take action” button below 360° videos presenting a particular grievance.
Spectacular and unknown places, work with special angles
The place where the film takes place receives a crucial role in 360° videos. Boring, unspectacular places aren’t inviting to look around. Vice versa, interesting, spectacular, manifold places can be entertaining. Of course, producers will bring users to vulcanos, into space shuttles and to war scenes. We did the same by placing the user into the tarantula’s terrarium. But you don’t always have to go that far: For the video about Generalized Anxiety, we started the experience in a can – a symbolic start as cans play an important role in the video. Starting in full darkness, we used the sound of opening a can as a cue. With the opening of the can, the user gets out of the can into the room – and into the action.
Such mechanisms require a new way of thinking of the producers, a creative approach to work with special rooms, places and angles. What makes sense from a storytelling perspective? What works for the user? And where to position the camera to create a credible perspective? (As the user will probably be standing or sitting watching the video, not every camera position will have a credible effect.)
Spatial Audio and navigation through audio
Spatial Audio completes the 360° format: Studies show that both emotional reactions and ratings of presence increase with spatialized sound (Västfjäll 2003). Positioning sounds in the room or the scene, intensifies the immersion and the 3D effect. We made use of this effect particularly in the Arachnophobia and the Paranoia film (you might like the scene where you suddenly find the tarantula on top of you, introduced by a close noise from above).
One of the sound designers experienced the difference in the immersive potential of smartphones and VR headsets. He was about to design the sound for the tarantula video and watched the scenes on his smartphone. After finishing the design, he received the opportunity to watch the scenes again on a VR headset and was shocked by the size of the tarantula. He decided to design the sound all over again (for the experienced bigger size of the spider).
Sound can also be used as a cue to make users turn around, explore the room or not miss an incident (as the tarantula). We played with this in the Paranoia film, triggering breath or whisper in the user’s back (respectively 180° away from the last trigger). But also in the two other videos to indicate in which direction currently action takes place.
Increase user agency
With the possibility to look around in the video, the producer provides the user with increased agency. This also means that if users want to watch the ceiling, they can and will. And it also means that users don’t necessarily follow the plot if they don’t feel like. This has important implications for the storytelling: Producers can design a plot that doesn’t build on mandatory (visual) content but just offers different bits of information users can pick up or not – and make the exploration of the setting or rather the whole experience the aim of the content.
An alternative is to act on the assumption that users will watch the video repeatedly to discover the whole story – but we cannot be sure about that. Our students tried that in the Paranoia video: users can watch the video again and again and discover new things to see – but they don’t have to as information is communicated through audio. So, audio receives increased importance in 360° video.
But then again, we know that for mobile watching (in public) audio is not always accepted because not everyone carry earphones. In the beginning, producers used longer texts in the video to communication instructions or information (e.g. The Displaced). However, you don’t see that too often anymore because people don’t read longer texts (and you also have the risk that people don’t see it, looking in a different direction).
However, if users can look around they will. In the Arachnophobia video the students placed the camera for some shoots in a corner of the room. If users look around, they at some point can only see the wall and some terrariums from close (you can also see the same problem in the first scene of “After Solitary”). Such angles aren’t appealing and should be avoided. The cameramen have to get used to think in 360° and consider every possible angle.
Challenges of 360° non-fictional storytelling
Due to their lack of experience with the new technologies (camera and microphone) and the new storytelling format, the students needed much more time for the production according to their own evaluation. It started with the search for the topic but didn’t stop there. Hardly every step of the production process required more time than they were used to. Of course, most processes will become more efficient with time. But the stitching is currently still additional (even though 360° cameras for smartphone offer automatic stitching) and rather spontaneous shooting (beside 360° live streaming) is just impossible as setting up the scene just takes much more time and coordination.
As soon as most of the production processes will become automated, the experimentation with the storytelling in the new format will be what’s left – and this is of course exciting. Over the past years, the ones who pushed the format forward were the ones who really experimented with new storytelling techniques – in fiction and non-fiction (e.g. Jaunt VR’s “Ashes to Ashes“, the cooperation around ”Notes on Blindness” or BeAnotherLab’s “The Library of Ourselves”).
An important technique for non-fictional storytelling is the interview. The work with this in normal video long established technique is confronted with new challenges in 360° video as there is no behind-the-camera. Of course there is the possibility to have both, the interviewee and the interviewer, in the scene. However, our students didn’t want the interviewer to be visible – a common preference in documentaries and features. For the realization in 360°, the students tested three different alternative solutions: (1) First they had the interviewer present in the room and erased him in the postproduction. This however is very laborious. (2) Then they tried to use walkie-talkies: The interviewer was outside the room and asked questions through walkie-talkies. This worked but the students prefer a third solution: (3) The interviewer comes into the room, asks a questions, leaves the room and the interviewee answers the question by him- or herself. Both solutions are of course artificial for the interviewee as he or she talks alone in a room. There are some people who can better deal with this situation than others. Ursina for example, the protagonist in our Arachnophobia video, did a very good job.
Non-fictional but staged
As there is no behind-the-camera area, 360° videos are more staged than traditional non-fictional films. You always have to stage the whole room or area if you don’t want the crew and/or equipment to be visible. Our students’ videos were all staged and recorded with several takes. Spontaneously following a protagonist with the camera without giving any instructions is not possible as the producer has to place the camera on a suitable position and leave the scene.
This is also the reason why short documentaries in 360° are often just a sequence of scenes without camera movement (see for example the coverage of the Syrian civil war in “The Fight for Falluja” by The New York Times or documentaries from refugee camps as “Clouds Over Sidra” by The United Nations and Within).
As the all-around-aspect is central to 360° films, a producer should integrate the whole scenery – or at least make it a conscious decision which areas to stage and which not (there are producers who consciously only stage a limited area and guide users’ attention to this spot). Dramaturgy, orchestration and staging become very important – and don’t leave too much freedom for spontaneity.
Of course there are settings that allow for spontaneous records with visible equipment as the ascent of the Mount Everest (by Matthias Taugwalder for Mammut) when the advantages of a 360° capture outperform aesthetic trade-offs.
Influencing user’s perception
The format’s immersive potential bears the risk to not just present facts but influence users’ perception by manipulating their feelings. Our students to a certain degree influenced users’ perception of communicated information in all three videos: In the first film, it was rather playful by beaming the user into the tarantula’s terrarium. This effect should ease the understanding of Ursina’s fear of spiders. In the video about the emergency supply shop, the students complemented the storyline with (made up) scenes from a shelter with flickering light. With this, they disconnected from facts and created a (fictional) scenery to influence users’ perception of the main story – although with the same intention, to get users into a mood related to the content. In the third film, they again went one step further by making up the whole scenery. In my eyes, the applied techniques in the emergency shop and in the surveillance film place them in a gray area between fiction and non-fiction and I would no expect to see the use of such techniques in professional journalistic contributions.
Of course, emotions have always played a role in non-fictional storytelling. But the new format’s immersive potential – and with this also it’s manipulative power – exceeds that of precursor formats as text, image, audio and film (at least if its well done). This risk will even be increased as soon as full VR and computer-generated images (CGI) will become more common and widespread. How should users evaluate a situation’s origin? Some producers would possibly say that generated scenes don’t have to base on real ones if the aim is to let users experience certain conditions or situations. This may be true for a space flight or even blindness. But what about the Syrian war?
Today users make a conscious decision for journalism, to see a fictional movie or play a computer game. Emerging formats however bear the potential (or risk) to merge these formats. The resulting experience will then merge the effect of information/facts and emotions. How should users make sure to not become manipulated? In my eyes, there is a need for the development of contemporary ethical standards, working with new technologies and formats in journalism.
In our project, we realized a non-fictional 360° video production with our graduating class over the course of nine weeks. All the students had some previous experience with the new technology and the emerging storytelling format. The students have been supported by two teachers and an external expert and realized a collaboration with the school’s sound design and composition students. I accompanied the project with an scientific perspective, applying participatory observation, informal talks and formal interviews. From the collected data, I derived some insights concerning the work process and four opportunities and four challenges working with the new format I discussed in the paper.
All together, we can confirm (other) practitioners’ experiences previously presented in journalistic contributions: Storytelling in 360° seems to exceed the immersive potential of previous (journalistic) formats as text, picture, audio and video. Connected to the immersive potential are the stimulation of presence and content related emotions. Related to presence and emotions seem empathy and manipulative power – this implication however still lacks concluding empirical evidence. First insights however clearly point in this direction, calling for the discussion of ethical standards for journalistic coverage in this area. But also for media literacy on the user side as not only journalists are working with the new format.
We can further confirm the higher production costs of 360° videos not just due to the lack of practitioners’ experience with the new technologies and the related storytelling but also due to additional or more laborious work steps (e.g. stitching, conducting interviews, staging the scene). Of course, with time the work with the new technologies becomes more efficient and storytelling more experienced. Producers however should not stop experimenting with new, adapted storytelling techniques as, what we have learned so far, they are what makes watching this content interesting and worthwhile. To directly address the user of 360° videos seems to be a technique worth being further explored in my eyes as it appears to be in line with the experience of presence. The reduced interaction possibilities in 360° videos however seem to be impedimental here.
Another aspect to confirm is the importance of rooms and places – and the exploitation of the 360° spectrum. Producers don’t have to go that far as to give access to extraordinary places each time. Conscious staging can also make a scene worth exploring. If users should be guided trough a designed sequence of scenes, making sure they don’t miss any bit of information or if users should be free to explore a scenery, maybe watching a video several times to get all content, is up to the producers’ decision. What works best or if different options will coexist will be the result of further experience with the format and does also depend on the content and the category (fiction vs. non-fiction). My experience with another class shows that not all users enjoy the experience of (maybe) missing out some information or parts of the storyline.
In this study, I didn’t go into cutting, the length of scenes and of the whole video. These are also characteristics of this emerging format and standards are still under discussion.
Our study further discusses practical aspects as the use of Spatial Audio, unfamiliar perspectives (e.g. from inside a can) or conducting interviews that didn’t get too much attention in (popular) reporting so far. But also rather theoretical aspects as increasing user agency or the conflict around staged scenes and the manipulative power of the format should receive more attention in the discussion of its use particularly in journalism.
Beside the fact that our study just presents the insights from one realization of a 360° production and with this cannot be generalized yet (even though mostly in line with precedent reports form practitioners), we also have to be aware of the fact that our students are design and not journalism students. That’s the reason we decided to speak of non-fictional storytelling but not of journalistic work. Even though our students are trained to work in consideration of the most common journalistic work standards and did apply them in the project, they cannot (yet) be called journalists. This means that experienced journalists would probably identified further or different opportunities and challenges than we did in our work.
Another characteristic of our study to consider is the artificial environment of the project: It was the students’ work assignment to realize a publication in 360° and the project had to become realized within the course of the semester. Of course, many editorial teams probably created similar conditions with their decision to gain experience with the new storytelling format. And also in real-world projects the duration is most often predetermined (and probably even tighter scheduled).
What seems most important at this point of the emergence of the new storytelling format are systematic user studies. We definitely need more reliable information about what works for users storytelling-wise but also what effect the immersive potential develops on users’ perception. My own first experiences here show that technical challenges will make these studies rather difficult (e.g. most survey software doesn’t allow for the integration of 360° videos yet and at least in Switzerland still many interviewees lack experience with 360° videos) but researchers will develop solutions.
Special thanks for their support go to Martin Zimper, Marc Lepetit and Nicholas Schärer and of course to our students who not just realized the project but also have been interviewed for the purpose of this study. We also wish to thank the sound design and composition students for their great effort and their support.
Adams, Eric. 2016. “Improving Semantics. Most Virtual Reality Is Not Virtual Reality. Here’s Why”. Gear Patrol. https://gearpatrol.com/2016/07/18/virtual-reality-vs-360-video/
Alsever, Jennifer. 2015. “Is Virtual Reality the Ultimate Empathy Machine?”. WIRED. https://www.cjr.org/innovations/virtual_reality_journalism.php
Aronson-Rath, Raney, James Milward, Taylor Owen, and Fergus Pitt. 2015. Virtual Reality Journalism. New York, NY: Tow Center for Digital Journalism. https://towcenter.gitbooks.io/virtual-reality-journalism/content/
Bilton, Ricardo. 2016. “The New York Times Is Trying to Make VR Films That Aren’t One-Offs, and That Keep Readers Coming Back”. NiemanLab. http://www.niemanlab.org/2016/05/the-new-york-times-is-trying-to-make-vr-films-that-arent-one-offs-and-that-keep-readers-coming-back/
Bloom, Paul. 2017. “It’s Ridiculous to Use Virtual Reality to Empathize With Refugees”. The Atlantic. https://www.theatlantic.com/technology/archive/2017/02/virtual-reality-wont-make-you-more-empathetic/515511/
Bock, Mary Angela. 2011. “You Really Truly, Have to Be There: Video Journalism as a Social and Material Construction”. Journalism & Mass Communication Quarterly 88 (4). SAGE Publications: 705–18. doi:10.1177/107769901108800402.
Bogost, Ian, Simon Ferrari, and Bobby Schweizer. 2012. Newsgames: Journalism at Play. MIT Press.
Boyer, Paul D. 1998. “Energy Life, and ATP (Nobel Lecture)”. Angewandte Chemie International Edition 37 (17). Wiley-Blackwell: 2296–2307. doi:10.1002
Coddington, Mark. 2014. “Clarifying Journalism’s Quantitative Turn”. Digital Journalism 3 (3). Informa UK Limited: 331–48. doi:10.1080/21670811.2014.976400.
Constine, Josh. 2015. “Virtual Reality, The Empathy Machine”. TechCrunch. https://techcrunch.com/2015/02/01/what-it-feels-like/
Cruz-Neira, Carolina. 1993. Virtual Reality Overview. ACM SIGGRAPH ’93 Course Notes: Applied Virtual Reality.
de la Peña, Nonny, Peggy Weil, Joan Llobera, Bernhard Spanlang, Doron Friedman, Maria V Sanchez-Vives, and Mel Slater. 2010. “Immersive Journalism: Immersive Virtual Reality for the First-Person Experience of News”. Presence: Teleoperators and Virtual Environments 19 (4). MIT Press – Journals: 291–301. doi:10.1162/pres_a_00005.
Deuze, Mark. 2007. “What Is Multimedia Journalism?”. Journalism Studies 5 (2). Informa UK Limited: 139–52. doi:10.1080/1461670042000211131.
Deuze, Mark. 2005. “Towards Professional Participatory Storytelling in Journalism and Advertising”. First Monday 10 (7). University of Illinois Libraries. doi:10.5210/fm.v10i7.1257.
Deuze, Mark. 2007. “What Is Multimedia Journalism?”. Journalism Studies 5 (2): 139–52. doi:10.1080/1461670042000211131.
Diemer, Julia, Georg W. Alpers, Henrik M. Peperkorn, Youssef Shiban, and Andreas MÃhlberger. 2015. “The Impact of Perception and Presence on Emotional Reactions: a Review of Research in Virtual Reality”. Frontiers in Psychology 6 (January). Frontiers Media SA. doi:10.3389/fpsyg.2015.00026.
Doyle, Patrick, Mitch Gelman, and Sam Gill. 2016. Viewing the Future? Virtual Reality in Journalism. Miami, FL: Knight Foundation. https://knightfoundation.org/reports/vrjournalism
Ganz, Jason. 2017. “Virtual Reality Is the Global Empathy Machine. Learning in VR Will Inspire Action like Never before”. Medium. https://medium.com/singularityu/virtual-reality-is-the-global-empathy-machine-283b1ee4192c
Garling, Caleb. 2015. “Virtual Reality, Empathy and the Next Journalism”. WIRED. https://www.wired.com/brandlab/2015/11/nonny-de-la-pena-virtual-reality-empathy-and-the-next-journalism/
Goodman, Lizzy. 2012. “‘Hunger In L.A.’ Immerses Viewers In An Interactive Journalism Experience (And A Food Line)”. FastCompany. https://www.cjr.org/innovations/virtual_reality_journalism.php
Grubenmann, Stephanie, and Miriam Meckel. 2015. “Journalists’ Professional Identity”. Journalism Studies 18 (6). Informa UK Limited: 732–48. doi:10.1080/1461670x.2015.1087812.
Hermida, Alfred. 2010. “TWITTERING THE NEWS”. Journalism Practice 4 (3). Informa UK Limited: 297–308. doi:10.1080/17512781003640703.
Hermida, Alfred. 2012. “Social Journalism: Exploring How Social Media Is Shaping Journalism”. In The Handbook of Global Online Journalism, 309–28. Wiley-Blackwell. doi:10.1002/9781118313978.ch17.
Hernandez, Robert. 2016. “The Year Virtual Reality Becomes Reality”. NiemanLab Predictions for Journalism 2016. http://www.niemanlab.org/2015/12/the-year-virtual-reality-becomes-reality/
Hujanen, Jaana, and Sari Pietikäinen. 2004. “Interactive Uses of Journalism: Crossing Between Technological Potential and Young People’s News-Using Practices”. New Media & Society 6 (3). SAGE Publications: 383–401. doi:10.1177/1461444804042521.
Jensen, Thor. 2016. “The History of Virtual Reality Video Games”. Geek.com. https://www.geek.com/news/the-history-of-virtual-reality-games-1652225/
Jensen, Thor. 2016. “The History of Virtual Reality Video Games”. Geek.com, April.
Lee, Kwan Min. 2004. “Presence Explicated”. Communication Theory 14 (1). Wiley-Blackwell: 27–50. doi:10.1111/j.1468-2885.2004.tb00302.x.
MacVey, Matthew. 2015. Exploring Future Reality. Brooklyn, NY: NYC Media Lab. https://medium.com/exploring-future-reality
Malmo, Christopher. 2014. “A New Virtual Reality Tool Brings the Daily Trauma of the Syrian War to Life”. Motherboard VICE. https://motherboard.vice.com/en_us/article/jp5jx3/virtual-reality-is-bringing-the-syrian-war-to-life
Milk, Chris. 2015. “How Virtual Reality Can Create the Ultimate Empathy Machine”. TED Talk. https://www.ted.com/talks/chris_milk_how_virtual_reality_can_create_the_ultimate_empathy_machine/transcript?language=en#t-3470
Mischler, Elliot G. 1986. Research Interviewing: Context and Narrative. Cambridge, MA: Harvard University Press.
Mufson, Beckett. 2014. “Is Virtual Reality The Future Of Journalism?”. Creators VICE. https://creators.vice.com/en_us/article/8qvgpx/is-virtual-reality-the-future-of-journalism
Naylor, Aliide. 2017. “The Empathy Machine: Can VR Stop Bad City Developments before They Happen?”. The Guardian. https://www.theguardian.com/cities/2017/may/26/empathy-machine-vr-bad-city-developments-virtual-reality
Nip, Joyce Y. M. 2006. “EXPLORING THE SECOND PHASE OF PUBLIC JOURNALISM1”. Journalism Studies 7 (2). Informa UK Limited: 212–36. doi:10.1080/14616700500533528.
Palmer, Mark T. 1995. “Interpersonal Communication and Virtual Reality: Mediating Interpersonal Relationships”. In Communication in the Age of Virtual Reality, edited by Frank Biocca and Mark R. Levy, 277–302. Hillsdale, NJ: Lawrence Erlbaum Associates.
Pavlik, John. 2000. “The Impact of Technology on Journalism”. Journalism Studies 1 (2). Informa UK Limited: 229–37. doi:10.1080/14616700050028226.
Polgreen, Erin. 2014. “Virtual Reality Is Journalism’s next Frontier”. Columbia Journalilsm Review. https://www.cjr.org/innovations/virtual_reality_journalism.php
Robertson, Adi. 2014. “VR Was Sold as an ‘Empathy Machine’ — but Some Artists Are Getting Sick of It”. The Verge. https://www.theverge.com/2017/5/3/15524404/tribeca-film-festival-2017-vr-empathy-machine-backlash
Robertson, Adi. 2016. “The New York Times Is Sending out a Second Round of Google Cardboards”. The Verge. https://www.theverge.com/2016/4/28/11504932/new-york-times-vr-google-cardboard-seeking-plutos-frigid-heart
Schubert, Thomas, Frank Friedmann, and Holger Regenbrecht. 2001. “The Experience of Presence: Factor Analytic Insights”. Presence: Teleoperators and Virtual Environments 10 (3). MIT Press – Journals: 266–81. doi:10.1162/105474601300343603.
Sirkkunen, Esa, Heli Väätäjä, Turo Uskali, and Parisa Pour Rezaei. 2016. “Journalism in Virtual Reality”. In Proceedings of the 20th International Academic Mindtrek Conference on – AcademicMindtrek 16. ACM Press. doi:10.1145/2994310.2994353.
Stale, Grut. 2017. “The Battle for High-Quality VR”. NiemanLab: Predictions for Journalism 2017. http://www.niemanlab.org/2016/12/the-battle-for-high-quality-vr/.
Steensen, Steen. 2011. “ONLINE JOURNALISM AND THE PROMISES OF NEW TECHNOLOGY”. Journalism Studies 12 (3). Informa UK Limited: 311–27. doi:10.1080/1461670x.2010.501151.
Steuer, Jonathan. 1992. “Defining Virtual Reality: Dimensions Determining Telepresence”. Journal of Communication 42 (4). Wiley-Blackwell: 73–93. doi:10.1111/j.1460-2466.1992.tb00812.x.
Västfjäll, Daniel. 2003. “The Subjective Sense of Presence Emotion Recognition, and Experienced Emotions in Auditory Virtual Environments”. CyberPsychology & Behavior 6 (2). Mary Ann Liebert Inc: 181–88. doi:10.1089/109493103321640374.
Visbit. 2017. “Ending the Hype Cycle: Today’s Challenges and Opportunities in Improving the 360 VR Video Experience”. Medium Visbit Blog. https://medium.com/visbit/ending-the-hype-cycle-todays-challenges-and-opportunities-in-improving-the-360-vr-video-6dbdcdf1364
Wallace, Sue. 2009. “Watchdog or Witness? The Emerging Forms and Practices of Videojournalism”. Journalism: Theory Practice & Criticism 10 (5). SAGE Publications: 684–701. doi:10.1177/1464884909106539.
Watson, Zillah. 2017. “VR for News: The New Reality?”. Oxford: Reuters Institute for the Study of Journalism.
Watson, Zillah. 2017. VR for News: The New Reality?. Oxford: Reuters Institute for the Study of Journalism. http://reutersinstitute.politics.ox.ac.uk/publication/vr-news-new-reality
Welsh, Madeleine. 2015. “The New York Times Hopes Its First Virtual Reality Film, ‘The Displaced,’ Kicks off Mass Adoption of VR”. NiemanLab. http://www.niemanlab.org/2015/11/the-new-york-times-hopes-its-first-virtual-reality-film-the-displaced-kicks-off-mass-adoption-of-vr/
Westlund, Oscar. 2013. “MOBILE NEWS”. Digital Journalism 1 (1). Informa UK Limited: 6–26. doi:10.1080/21670811.2012.740273.
Wirth, Werner, Tilo Hartmann, Saskia Böcking, Peter Vorderer, Christoph Klimmt, Holger Schramm, Timo Saari, et al. 2007. “A Process Model of the Formation of Spatial Presence Experiences”. Media Psychology 9 (3). Informa UK Limited: 493–525. doi:10.1080/15213260701283079.
Zhou, Junmei, Zhixiong Xue, Ziyun Du, Teri Melese, and Paul D. Boyer. 1988. “Relationship of Tightly Bound ADP and ATP to Control and Catalysis by Chloroplast ATP Synthase”. Biochemistry 27 (14). American Chemical Society (ACS): 5129–35. doi:10.1021/bi00414a027.
Zinszer, Kate, Kathryn Morrison, Aman Verma, and John S. Brownstein. 2017. “Spatial Determinants of Ebola Virus Disease Risk for the West African Epidemic.”. PLoS Curr 9.