The last question of the UX Series is the one million dollars question: how do we understand, and measure, the impact in an i-doc?
What is the time-frame we should consider in the evaluation? Is it about the number of people exposed to our message? Is it about telling others about our documentary and issue? It is about changing attitudes? Is it about wanting to know more? Is it about wanting to be involved in a solution? Is it about considering the effect on the subjects rather than on the audiences?
The reality is that it could be all of the above, and none of these are clearly quantifiable shifts. Google Analytics offers us the possibility to track behaviours on our websites: clicks, page visits, interaction paths and time spent on content. My first interview, with Harry Davies from Google is all about this: what can we measure with the tools that we have right now. What I found fascinating is to discover how Google is trying to master both quantitative and qualitative information about user behaviors…. in a clear attempt to go beyond “what the user did” iand to start tackling “WHY the user did something”.
If we listen to Google, and to marketing people, we have the impression that digital media really allows us to “know” our users/audience… but is this true? As Kat Cizek, NFB’s Director of Highrise, points out in her interview we are still struggling to understand what impact is and whom it does affect. For example: we know that urban social policy has changed in Toronto, but can we say that it is because of the impact of One Millionth Tower? Would it make any sense to try quantify it and give it a 45% impact rate? Also, by overemphasizing the power of numbers, Kat points out that “Big data and Google are not interested on our opinions anymore, they are only interested on our behaviour, so that they can then make decisions based on our behaviour. This takes away our agency of telling our own story.”
On the other hand, our obsession with numbers and quantification is such that there is a big financial incentive to pursue audience analysis. Production money will be more available if one can prove a track record of success stories, and the overall cultural marketing trend pushes us to confuse success with traffic numbers. In the commercial world new tools are being build to be ever more efficient in tracking our users behaviours. Should we make use of such tools?
Clint Beharry, from the Harmony Institute, tell us how their new free web application, ImpactSpace, works at mapping media impact by using trend analysis, social media content spread and similar documentaries’s positioning. Is such a contextual approach able to balance Google Analytics insular approach (the analysis of the single website out of its media and social context)? As Clint says “there needs to be a tool that is focused for storytellers to understand impact beyond hard numbers and that allows them to look beyond their own product and look at the big picture. Look at what other people have done in the past, learn from their own mistakes”.
Finally, what is the value of one to one interviews and user experience surveys? Kate Nash, keynote speaker at i-Docs 2014, will present us the result of a long qualitative research she has done to evaluate the impact of three current i-docs. What does the qualitative approach offers us that gets forgotten in the quantitative logic? If you want to be part of this debate, join us at i-Docs in a few days time!