In her research conducted during a fellowship on immersion with the South West Creative Technology Network, Julia Scott-Stevenson explores: How can immersive media create shared experiences that imagine pathways towards a preferred future?
This piece was originally published in Immerse: creative discussion of emerging nonfiction storytelling on March 8, 2019.
I know, it sounds like I might be about to join the cacophony of grand claims around immersive media technologies, such as those relating to behaviour change, empathy development, and bias reduction. Meanwhile, a commonly heard concern is that digital media technologies distract us, take us out of the “here and now”— affixing our attention to a mobile phone perhaps, or isolating us behind a head mounted display.
Notwithstanding these fraught debates, it seems clear that one of the affordances of immersive media, and VR in particular, is the creation of a sense of presence, of “being there.” So why take a user out of the “here and now” and attempt to situate her somewhere else?
Vestige VR
One reason is to take her somewhere she cannot otherwise go. Some creators use the technology to visit someone or something lost or past (see, for instance, Vestige VR, or historical pieces such as Immersive Histories: Dam Busters). Others use the technology to venture inside another’s mind (Manic VR). Some explore future worlds (Biidaaban: First Light), or visit remote places that demonstrate the impacts of the anthropocene (Sanctuaries of Silence). Many future narratives, though, across multiple screen media forms, tend to be dystopian in flavour. Instead, I’m interested in what might be termed “preferred” futures — what is a future we want to get to? What is the world we want to make?
Manic VR
In September 2018 I was selected as one of 27 Immersion Fellows for the South West Creative Technologies Network project in the UK and given the task of exploring immersion. So, I developed a question:
Might there be possibilities within immersive media for creating shared experiences that imagine pathways towards a preferred future?
I explored this question by surveying existing media works across VR/AR/immersive experiences and talking to practitioners and industry members and began to identify a number of themes. Rapidly, I realised I had a manifesto on my hands!
In writing an immersion manifesto, I’m in the esteemed company of Kat Cizek and Janet H Murray, whose manifestos — taking slightly different angles — are absolutely worth a read. What I’m proposing here is a series of guiding principles for the development of immersive projects, particularly those that explore preferred futures (but, I hope, for other stories as well).
1. Stage an encounter
See also: connection, conversation
I had a wonderful, transcendent experience in The Collider, an immersive piece by Anagram (you can read more about it here). The most intriguing part was the curated encounter with another person, whom I had not met until that moment. I think using the notion of an encounter in an immersive piece neatly sidesteps some of the ethical conundrums apparent in claiming to develop empathy by enabling the user to walk in another’s shoes. With an encounter, the user is not trying to be someone else, she’s going to meet them for a while.
With an encounter, the user is not trying to be someone else, she’s going to meet them for a while.
Most of the experiences that led me to this understanding enabled an encounter with another human being — but perhaps we could have an encounter with ourselves? Or possibly even an encounter with the natural world? Rob McLaughlin, Executive Producer of Digital Content and Strategy with the Canadian National Film Board, suggested to me also that we could have an encounter with the maker of the work, through the work itself.
The Collider
Sparked by an ongoing conversation with i-Docs Director Mandy Rose, I borrowed the idea of encounter from the philosopher Emmanuel Levinas. Levinas noted that an ethical encounter with another recognises their infinity — that is, the complexity of that person, and an understanding that we will never fully know their entirety. He also noted that in representing others, we run the risk of instead totalising them — suggesting the incomplete picture we present is in fact all there is to them.
Consider, for instance, stories of people with disabilities. It can often be the case that the story of their disability is presented as the only story they have, ignoring the individual’s great complexity of being and of experience. So by crafting an encounter with another, instead of attempting to represent another or to have the user “be” another, I hope that immersive experiences move us towards an opportunity to understand the infinity of whom we encounter. (Hat tip here also to Freya Wright-Brough, who has explored Levinas in relation to digital refugee narratives during her PhD at Queensland University of Technology).
Finally, I also wonder if expanding the idea of encounter to specifically include conversation — allowing the user to speak back to the person they encounter — might be even more transformative. There is a deep rabbit hole here of questions that I won’t dive into now, around whether one can truly have an encounter with a simulation of a person, particularly in the age of artificial intelligence (something being explored by Silke Arnold-de Simine at Birkbeck University of London).
2. Be wild: Bewilderment is powerful
See also: Joy, awe
The world is an amazing, complex, messy, and beautiful place, and people are all of these things and more. But the world, and its inhabitants, are also in fairly dire straits. There’s a case to be made that the climate change situation is so desperate that the only route is to shock people out of complacency, and anything less is not enough (see, for instance, this piece in The New York Times recently).
On the other hand, there is a risk that viewers will simply switch off from relentless difficult stories because it all just seems so overwhelming and intractable. Imagining the future is clearly a fraught business.
I wonder, though, if the unique affordances of immersive media (the feeling of presence, for instance), might offer us a way forward. If we can see/touch/hear/interact with the things we love about the world now, are we better able to envisage a way to protect them in the future?
Sanctuaries of Silence
Kevin Berger writes about a conversation he had with novelist Richard Powers, who brings up an essay by Lewis Thomas from the ’80s, On Matters of Doubt. Powers points to Thomas’ reminder that bewilderment, in essence, means partaking in a state of being wild. So rather than the more common usage of perhaps a feeling of confusion or bamboozlement, it is a reminder of our animal state and our existence within a vast and complex system.
Bewilderment, in essence, means partaking in a state of being wild…a reminder of our animal state and our existence within a vast and complex system
Games designer Jane Friedhoff is onto something similar when she says she looks for catharsis over education in her games. Friedhoff, speaking at the Eyeo Festival last year, says she instead wants to create joy, while still pointing to a desired world. She defines joy as containing a number of elements: a capacity for action, a sense of feeling seen and heard, a feeling of being part of a larger whole, an ability to envision a new and better world, and something that can’t be fully explained or quantified. With this approach, she is advocating a move away from that extractive model of storytelling where we look from the outside and peer in at someone else. Instead, it’s about a joyful connection with a shared community — she says, “I can make mosh pits for my friends.” I think this sense of catharsis that Friedhoff is talking about is bewilderment by another name. We connect with others and are reminded of our position within a complex, majestic system.
It requires mentioning that I’m also not just talking about environmental narratives here (not that these exist separately from humans in any case) — my preferred future is an equal, equitable and diverse place that humans exist within, and there’s joy and bewilderment to be found in that.
3. Move from being to doing
See also: agency, interaction, control
I’ve lost count of the number of (usually) nonfiction 360 degree film experiences in which I’ve wondered why I’m wearing a headset, and whether I’m gaining anything more than I would by seeing the same piece on a flat screen. The creation of a feeling of presence can be enough in certain stories, but, for many, there needs to be a next step of some level of interaction. It’s a fine balance though, as the interaction needs to have some meaningful purpose to it.
Interaction needs to have some meaningful purpose to it.
Homestay VR is an animated VR story by Paisley Smith and the Canadian National Film Board. In it, the user explores a CGI-rendered Japanese garden while listening to Smith’s voiceover tell the sad story of Taro, a home-stay student with her family. Sparkling red leaves flutter around you, and you can reach out and collect them. While I thought the leaves were lovely, I was initially uncertain how they related to the story, or, specifically, how my catching them was relevant. Later, I heard Smith speak about the work, and she said, “Sometimes giving you something to do helps you stay in the present.” Smith has neatly encapsulated here the problem I noted above with some VR work and identified an important first step to encouraging a feeling of connection.
Collecting leaves in Homestay VR
How do we go from being in the present, though, to having an impact on it? I would suggest that interactivity can lead to a feeling of agency (some level of control) in the user only when the interaction has some meaning to the story. Janet H. Murray, in her seminal 1997 book Hamlet on the Holodeck, calls this participating in the “‘Active Creation of Belief.” We have to put in enough work to help continue the illusion, but not so much that the effort pulls us back out.
4. Embody the future
See also: bodies, voices
I love Lynette Walworth’s work, and Collisions is one of my favourite VR pieces. Exploring her latest work, Awavena, at IDFA DocLab late last year, I enjoyed the storytelling and was awed by the stunning CGI design of the forest. But, at 40 minutes long, it made me feel incredibly nauseous, and I was constantly aware of the heaviness of the headset and of what seemed like each tiny pixel in my view. I regularly had to close my eyes to attempt a self-reset. If I could have stood up from my chair and walked around, I might have been able to better ignore some of the distractions of the tech.
Awavena
Manifesto point number four, around using embodiment, is closely linked to the previous point on agency. By being able to take some control — in a physical, bodily way — over the experience, perhaps the user invests in a more complete way, or in a substantively different way? I want to be careful, though, not to suggest that engagement in a story or media form is only possible through movement, as of course there are countless traditional-screen films and documentaries that deeply and effectively engage the audience. Perhaps the key difference here is using the unique sense of presence and the consequent opportunity for embodiment as a conduit to a different kind of engagement. Or maybe it’s simpler than that — perhaps providing the opportunity to move or speak allows users to realise that they can move and speak.
Intriguingly, many of the immersive projects I’ve experienced over the past year that have engaged the body beyond the eyes and ears have been to some extent based on voice. VVVR, Make Noise, Injustice, Terminal 3, The Collider; all have called on the user to speak.
Make Noise, by May Abdalla, is a wonderful example of curating this interaction in such a way that users knows precisely what is required of them. A VR piece about the suffrage movement, it gives suggestions for what kind and level of noise one should make, and later, particular words one should speak (or shout) as a specific act towards smashing barriers. This removes a lot of the self-consciousness often associated with audience participation, which can come from a fear of doing the “wrong” thing.
VVVR
When done well, the embodied immersive experience can also create joy and catharsis. VVVR is a two person VR experience in which you sit on the floor, facing your partner, and make any sounds you like with your voice. The sounds are visualised in shapes and colours streaming from your mouth, and you can also see the results emanating from the person opposite you.
The creators, Ray McClure and Casey McGonagle, spoke at Open Cities Doc Festival Expanded Realities Symposium last year, and McClure noted that he was overjoyed to watch a five-year-old paired with an eighty-year-old in the experience. He heard them howl and laugh, and described them as being “reduced to the same thing.” He went on, “there’s a moment when you see them break through” (into the unselfconscious mode of just being there).
When done well, the embodied immersive experience can also create joy and catharsis.
The joyfulness in VVVR comes from several things—the sharing of the experience with the other person (the encounter), the recognition of and connection with our animal nature (bewilderment), and the feeling of making ridiculous noises and speaking ourselves present (agency and embodiment). And while I’ve focused here on voice, there are plenty of other projects that explore different sensory approaches to embodiment. For movement/walking, see It Must Have Been Dark By Then and Pilgrim; for touch, see Micro-utopia: The imaginary potential of home; for taste, see Leaked Recipes and Frankenstein AI; and for smell, see Munduruku: The Fight to Defend the Heart of the Amazon.
It Must Have Been Dark by Then: a geo-located audio walk and book
5. Care: the participants matter
At IDFA DocLab in November 2018, Steye Hallema presented The Social Sorting Experiment, which was a fun, hilarious, and slightly disconcerting experience inviting 48 people to move around a grid and use their smart phones to rate their grid neighbours in various ways. To participate, though, we had to visit a URL on our phones, the first page of which asked us to accept a data disclaimer along the lines of, “We have the right to use this data in any way we see fit.”
I paused, my finger hovering over the assent button, as I wondered what data they meant. The data I enter into the browser? Every other page currently open in my browser, too? Anything at all on my phone? Eventually, I gritted my teeth and clicked, “yes,” comforting myself that “this is art” and, therefore, the creators aren’t going to sell my data to shady advertisers or election engineers (hey, whatever helps me live the contradictions of active social media participation and a resistance mindset). One woman standing in the grid put her hand up and objected loudly. I was envious of her conviction, but it did mean that she couldn’t participate.
By care, I mean care for the participant in two ways. Firstly, through respect for their privacy and data — as creators make yet more personalised experiences and collect ever more data on how each version unfolds (and think here beyond immersion about things like streaming service algorithms, for instance), it is vital that we maintain an ethical eye on how (and why) this data is stored and used.
Caring for participants means respecting both their privacy and their experience.
Secondly, I mean “care” through consideration of participant experience all the way from the introduction to and the exiting of and moments after the experience. For me, one very simple test of whether an experience has considered this is whether I have somewhere safe to put my bag. If I’m going to be experiencing a room scale VR piece, the sense of immersion is ruined if I have a constant urge to peek outside the headset and check that no one is nicking my stuff. “Care” in this sense also means considering the viewer’s response to the content. If the story is distressing or unsettling it can take some time for the viewer to decompress afterwards, and being thrust straight back into a busy festival hall might not be appropriate. Ensuring that participants have an opportunity to reorient themselves is important.
Putting the manifesto to use
Most of the projects I have mentioned here include some of the manifesto points, but not all of them. It may be that some points will be applicable to a work, and some won’t (although I think care is fairly non-negotiable).
I do think that if we want to create immersive works that envisage a way towards a preferred future, these points are going to offer some assistance (and they’ll hopefully assist even for projects aimed at something quite different). All the manifesto points are also closely intertwined, so it’s less a checklist and more a Venn diagram.
Biidaaban: First Light
Returning to my opening point about the hot mess we’re in — I’m under no illusion that the journey forwards will be easy, but, at heart, I’m an optimist, which is perhaps why I think we can do something by imagining the future through looking for the positive. The cultural landscape here isn’t completely empty; think of films like 2013’s Her, starring Joaquin Phoenix and Scarlett Johanssen. The film presents a future city in which everyone walks to work rather than zooming around in the standard issue future vision of gleaming space pods (but, yes, intimacy is also kind of dead — I’m looking for small wins here). Biidaaban: First Light is also an intriguing example — in this VR project’s future vision, nature has reasserted supremacy over the city of Toronto, yet local indigenous tribes have since flourished, as it is they who hold the knowledge necessary for survival.
This manifesto is, purposefully, technology-agnostic. A project that enables encounter, bewilderment, agency, embodiment and care will lend itself to a range of forms, across AR, projection, installation, VR, a combination of these and more. My humble hope is that creators can use this manifesto to inform the development of amazing projects that will each connect us a little bit more with the way forward. Used together, perhaps they can help us illuminate those very necessary paths toward the future we want to create.