Interrogating Audiences: What do we mean by engagement and evaluation?

Kate Nash is a Lecturer in Media and Communications at the Institute of Communications Studies, University of Leeds. Her research is focused on emerging forms of documentary, combining production and audience study.

She has kindly provided her paper: Interrogating audiences: What do we mean by engagement and evaluation?, which she presented during i-Docs 2014, in its entirety including the accompanying presentation (see article end). We also recorded the session, so consume as you wish, this is a must see/read/hear presentation for those on both the industry and academic sides of the interactive documentary field.

Abstract

A process of re(imagining) the audience is at the heart of interactive documentary production and scholarship. Various forms of interaction and participation challenge traditional theories of film and television audience, and we have come to speak, almost routinely, of ‘new relationships’ between audiences, documentary makers and text. It is an exciting exploration that has the potential to bring the civic culture of documentary into dialogue with digital cultures and new forms of media engagement.

Re(imagining) the audience begins by calling into question the logic of mass media and interrogating the interconnected practices of production and reception. Historically relatively marginal, audience research becomes central to both documentary production and scholarship. But how can we theorise and research the interactive documentary audience? What do we mean by engagement? And against what yardstick can we measure success? These questions are significant in shaping idocs production and scholarship.

In this presentation I will talk briefly about my recent experience of idocs audience research.  Approaching interactive documentary through the lens of interactive reception sheds light on the connections between interpretation and interaction, the importance of play narrative and agency and the significance of context. Importantly the research also points to the tremendous value to be gained by both producers and researchers in collaborating to better understand what audiences do, what they experience and how they interpret interactive documentary.

 

By way of introducing her well known Hamlet on the Holodeck Janet Murray describes Captain Kathryn Janeway’s holodeck liaisons in a simulated Victorian household with the fictitious Lord Burleigh and his family. In ‘living’ her Victorian adventure she participates in an exploration of the house and its inhabitants that at times leaves her breathless with excitement. She is immersed not only physically, but also psychologically, and morally.

For Murray the holodeck provides insight into the pleasures of new media: the promise of immersion, agency and the transformation of place, time and identity. But we can also read it as a description of an ideal of audience engagement. Janeway spends long periods of time actively engaged with the psychological and moral questions at the heart of the narrative. She experiences the story on multiple levels: intellectual, physical and emotional.

But how do we begin to conceptualize, let alone study or measure, this kind of complex multi-dimensional experience? From the perspective of the media industries this question is increasingly urgent – industry academic Philip Napoli[1] has described what he sees as a move to a post-exposure media marketplace – in which eyeballs are becoming less important than being able to demonstrate that audiences are paying attention and – importantly – talking about and sharing content.

Audience responses to content have never been more visible – and measurable – reflected in a ballooning engagement industry busy counting clicks, likes and shares or analyzing the sentiment of online discussion. But what does all this activity actually mean? This is the fundamental question – what does a click or a like really say about engagement? In an interesting blog post Chartbeat CEO Tony Haille[2] suggests that more than half of the things we share online we haven’t actually read.

Engagement is becoming a ubiquitous term – but its rarely defined or interrogated and there’s little consensus about what it means. So what does this mean for idocs? Firstly it highlights the importance of being able to demonstrate not only that there is an audience, but that the audience is engaged. In doing this it matters a lot that we can articulate and defend a measure of idoc engagement that takes into account what is unique about the idoc audience.

There is opportunity here – Napoli highlights the fact that engagement can be a game changer for content creators – often content that might not rate highly in terms of exposure rates very highly in terms of engagement. There’s every reason to think that this is true of the idoc – but we need to start by understanding what idoc engagement might look like and we need to be open to the fact that it might not be reflected in a barrage of user-generated content, clicks and shares.

I spent much of last year slinking around public libraries in Tasmania, looking for people prepared to experience an idoc and tell me about it. While I didn’t explicitly set out to explore engagement, actually watching people as they interact with content and make sense of the experience, does point to some potentially fruitful avenues for exploration.

I want to start by sharing with you what I found in between the book shelves and then consider how this might intersect with concepts of engagement – if there’s a take home message from all of this it’s that there is much to be gained by working together to explore audience responses in all their complexity.

My research

I began looking at idoc audiences driven by some simple questions about practices of interpretation and their wider significance. I wanted to explore how people made sense of idocs, whether there were patterns and what role individual difference plays and ultimately what this might tell us about the ability of idocs to expand audience experience.

To do this I considered four dimensions: The first is textual analysis – if texts address an imagined user – who are they talking to and what are they saying?  Then I looked at the idoc as realized through interaction and participation. I do this using screen capture software, which allows me to go back and look at different aspects of what the audience does. Capturing the text as realized is a significant challenge – particularly because ideally we want to study media engagement in context. My library solution, like most attempts at user testing, take audiences out of context so we don’t get a sense of – for example, how consumption of idocs may intersect with other media activities.

Thirdly I interrogated the physical response of the audience – looking at forms of reaction that are not registered in clicks – such as laughing, leaning forward, leaning back, yawning. And finally I explore the individual’s reflection on their experience and interpretation through a semi-structured interview.

Bear 71

Bear 71 – by the Canadian NFB is an interesting idoc because its interface is particularly opaque.

Actually one of the participants in my research summed up quite nicely:

“One thing I noticed, because it’s such an open world – open plan – and there’s obviously not a set way in which you have to interact with it – I did notice that I looked for ways to structure the way I interacted”

While most of the interview subjects found ways to structure their interaction – significantly 7 people (out of the 23 studied) didn’t interact with the computer at all. When I explored why it turns out that there are two reasons: one is that they didn’t know that they were meant to interact or what they were meant to do and the other is that they were highly immersed in the audio narrative.

‘I saw that I was meant to be on there at some point, but I wasn’t sure how you were meant to manipulate anything or what the point of manipulating it would be’. (female, 25 Early Childhood teacher B711)

‘I didn’t even think to press the arrows and things like that and I realise now that you’ve said it that I could have done. But no, it didn’t even come into my head’. (Female 34, Administrator B7113)

More than half of the research subjects (13) described feeling some kind of tension between the voiceover and interaction with 9 reporting that they privileged the voiceover and 4 reporting that they ‘switched off’ from the voiceover in order to concentrate on the interaction.

But for many the voiceover was highly engaging. In one case a participant was afraid of clicking in case the voiceover stopped. The audience that privileged the audio was highly engaged but not in ways that translate into a measurable click frenzy.

People found much pleasure in Bear 71 sometimes in surprising ways:

It was a good documentary because the talking – it took me a little while to realize that ‘oh I should be using the mouse here’ and then the little noises that you could come up with – me being me I decided to make a little rhythm out of it for a second.

You were making a little rhythm out of the sounds?

Yeah, some of the windows when you click on them, like the coyotes had facts, it would go (makes sound) and I would do that to the music which I also like because I know those bands (Female 28 Driver B7114)

This is a fantastic example of how idocs can engage people who do not respond to traditional content. This particular woman struggles to read and has issues with attention deficit – Bear 71 engaged her not only by providing opportunities for play – in a way she enjoyed, but by giving her a degree of control over her own experience.

Play is fundamental to how individuals respond to Bear 71. Studying the pleasures of game play Aphra Kerr and colleagues describe play as a compound pleasure combining control, immersion and performance.  Transformative play – looking for ways to test the boundaries of Bear 71 – was common. People tried to ‘stand’ on the tracks – or in this case a woman talks about trying to jump on the bear:

I also tried to hop onto the bear and press the bear and see if I could have an interaction with the bear. But that didn’t seem to work. So I would have liked to have interactions and I was trying to connect with the bear … because the documentary was of the bear talking I wanted to be with the bear to see it’s world that day. But I felt like – when I was on top of the bear it felt like nothing perceptive was happening to me, I didn’t feel like anything was happening particularly (Female 45 Marine Pollution Officer B7115).

Because it pushes at boundaries transformative play can be a source of pleasure but also displeasure. So in this case getting no response from the bear actually detracts from the pleasure of play.

Similarly narrative, as I have already suggested, is an important pleasure. A significant number of participants structured their interaction around the narrative – reconciling the tension between narrative and interaction by looking for things that are mentioned in the story. When there was a match it could be highly rewarding:

‘it was only late in the story that I tweaked that I should be paying more attention to where the bear is, because there was this story about the rubbing tree and the jogger, and it was a happy coincidence … suddenly I clicked on that scene and it said rubbing tree. And I clicked on it and I think you immediately see the jogger and I thought ‘oh wow, snap’ That’s fun, I wonder if there are other spots but I couldn’t find any others that were so well matched.” (Male 39, Academic B712)

And therein lies a tension, there often wasn’t a connection – so one participant, on finding the rub tree waited for someone to come past with a dog because the voiceover had mentioned that it can be difficult to pull your dog away from a rub tree. She wanted to see the person try to pull their dog away because she thought it would be funny – but she waited and of course it didn’t happen with the result that she felt a little detached from the experience.

The ongoing narrative also created expectations of change in the ‘main character’ that remained unfulfilled. For those participants who structured their interaction around following the bear there was the eventual realization that there was no visual development to match development in the audio narrative.

Mostly I just followed around the bear. I’d occasionally click on one of the nodes to see what it was. But whenever I clicked on the bear I expected to get different things – what the bear was doing at the moment, but it just kept going back to the bear in a berry patch or the bear looking at some elk and I would have expected more than that (Male 33, Student B7112) 

So what we see is audience expectations built by the narrative (and previous experience of interactive media and documentary) setting up pleasures that were sometimes not forthcoming.

My study of the Bear 71 audience confirms much of what Kerr and colleagues found in relation to new media pleasure: It is clearly multi-dimensional with control, immersion, play and narrative useful starting points for investigating the user experience.

Asylum Exit Australia

Asylum Exit Australia is a first-person procedural simulation that positions the audience as a member of a persecuted minority, challenging them to get out of Australia before their health, money and luck run out.  The narrative is developed via some 40 ‘cut’ scenes punctuated by player interaction with non-player characters.

The goal of getting out of Australia provides a clear structure for interaction but there’s another dimension that’s critical for most users – and that is self-reflection. And it is in the connection between goal and self-reflection that Asylum fosters very strong emotional engagement.

Asylum fosters self-reflection through direct address – early in the game the audience is prompted to nominate who they would most miss – including entering their name. And you can see in the interaction that people are clicking through and then they stop … there’s a huge pause. Only 5 of the 22 participants gave a name  – but in every case engagement is very evident in the fact that people aren’t clicking.

It’s a fantastic illustration of what Adrian Miles (2014) identifies as the affective potential of interaction. By stretching the time between perception and action, moment of decision can become one of understanding.

… one of the first people I was talking to was asking me ‘what’s the most important thing’ was it friends, something or other,  I didn’t really look at the others because it struck me that it was my child and I answered the question honestly and he was like ‘do you want to tell me their name’ and being a parent I was really protective so I was like ‘no I’d rather not’ and just from that it got real personal because the rest of the story when they’re talking about  my family I just kept going back to my child so it became really personal – so as I was experiencing it I had my child in my mind. If I’d known that I might have chosen something else just so I didn’t have that extra pressure sort of but I guess it wouldn’t have been a real kind of experience.

All but three participants talked about aspects of self-structuring their interaction. People were making decisions based on their personality, situation, and very often their values. One of the ways in which Asylum challenges its audience is by forcing them to make decisions that go against their values – particularly the decision to go into debt or as one participant put it ‘do the dodgy thing’

The result is significant emotional engagement. Fear, anxiety, discomfort, stress, anger and vulnerability were terms that came up frequently in peoples’ description of the experience of engaging with Asylum. Two broke down:

You can empathise with someone and you can feel their stress in one way, but there’s a barrier, you know … things like stress and anger you can only feel first hand (Male 37, Unemployed) 

People reported feeling frustrated, but they found that frustration meaningful in the context of the simulation. Cindy Poremba recently wrote about the ability of documentary games to create a third space between reality and representation in which experience is vivified – and this is what I found in how people described Asylum.

Participants also engaged playfully to different degrees. For a small few it was the dominant form of engagement. Mostly this was a way of negotiating their relationship to the simulation. One respondent compared Asylum to an RPG game – but also said that when playing he normally tries not to get immersed as a strategy for managing emotional intensity. For two others ‘playing the game’ allowed them to engage in spite of what they saw as a gap between their identity and the game’s scenario.

One participant rejected any suggestion that he wouldn’t know exactly where his passport was. Another rejected the whole idea that he would be leaving his girlfriend in the first place.

Finally it’s worth looking at social media engagement. Asylum encourages the audience to share their experience via Facebook. None did – largely because of the experimental situation. But I asked people whether it is something that they might do in other circumstances.  Only three of the twenty two thought they would – out of commitment to the issue, largely, and a sense that their friends would also enjoy it.

Many who wouldn’t share talked about not being Facebook users. But there was another interesting phenomenon that I can only describe as the social media equivalent of not talking politics at the table.

We connect with a variety of people on Facebook – and often that results in a network comprising people with very different political views. For some not wanting to ignite debate would prevent them from sharing material like Asylum in spite of their own engagement with the issue and the content.

Again if we’re looking at engagement in terms of clicks and shares most of what makes Asylum engaging goes under the radar. We could look at the amount of time people spent with the content – 18 mins on average – arguably that gives some indication but then there were quite a few who ‘got out’ relatively quickly – while still being very involved in the experience.

What is engagement?

So what can we take away in terms of thinking about what makes an idoc engaging or successful?

Firstly, I think it’s clear that in spite of all the new ways of tracking audiences: hits, clicks, shares and so on, most people are engaging on the inside. This isn’t unique to idocs – there are now a large number of studies in various contexts showing that people are much more inclined to listen than comment.  If we judged Asylum on the basis of the number of social media shares we would miss its very significant emotional impact.

The second thing I want to suggest about idoc engagement is that connection to content is important. Asylum resonated with the audience because – the issue is very topical, many had just experienced the second series of a high profile television documentary series Go Back to Where you Came From. And at a personal level quite a few of the people I nabbed around the bookshelves were involved with asylum seekers or at least described themselves as supportive of the issue.

By way of contrast one of the participants in the Bear 71 story thought that interaction with other users would change the experience significantly. But then he quickly went on to admit that he wasn’t really all that interested in bears so there probably wouldn’t be much to talk about. Where audiences share a passion for an issue or subject it is possible for that to motivate greater engagement.

Johnathan Grey’s analysis of different levels of commitment to media content is useful for thinking about the relationships of audiences to content. The topic ‘fan’ is characterized by an emotionally intense and active involvement with an issue across platforms. But most of us, most of the time are perhaps better described as non-fans when it comes to the issues addressed by idocs. We lack any personal connection or intense involvement. We may have a view, but without holding it strongly. The challenge for idocs makers is to cater for the non-fan as well as the fan; to make space for different levels of engagement with content.

In closing, I want to make a plea to all of you – as the idocs community we should all start to think of ourselves as audience researchers. We should all be looking for opportunities to work together on understanding what makes the idocs audience unique and being able to show funders and broadcasters the kind of impact this work can have.