USING STORYTELLING IN EVALUATION

Written by: Jen Berger, You've Got This Core Team

Evaluation through storytelling was new to me.

With a background in qualitative data collection, I’m well versed in interview techniques and coding, but, truthfully, before the pandemic, ‘storytelling’ was a technique I had not put into practice when evaluating projects.

As with any new way of working, the idea of it made me slightly uncomfortable at first, but those initial concerns were quickly replaced by an eagerness to try something new altogether.

As a pilot, we are encouraged to test and learn. We can not discover new ways of working unless we are open to doing things differently, and storytelling became a lot less scary when I started to think of it as something we all unconsciously do every day.

We interact with people and listen to conversations to draw out meaning and items of value to make sense of a situation and the world around us, and while a global pandemic might not seem like the opportune time to change tack and wander into the unknown, local insight had led me to believe otherwise.

When COVID hit, I used our online platform Sentiment to analyse local online conversations and track trends, and this research threw up some interesting insights. For example, residents were talking about how much COVID had impacted their mental health and confidence. This insight made me think of how we could use storytelling to empower people and build their confidence while also being flexible enough to change when needed to ensure it stayed relevant during this period of uncertainty.

I knew we needed to frame storytelling to allow the participant to choose how they wanted their story told. They are the authors of their lives, and we would capture their stories in whatever way they felt most comfortable.

My understanding of the insight from Sentiment led me to believe that storytelling, rather than surveys or other quantitative methods of data collection, could be a highly effective tool for learning about the participant experience in physical activity interventions. And we hoped that framing Storytelling in a way that was fluid and shaped by the participants themselves would give the participants ownership over their story, building their confidence and capacity at a time when many people in our communities felt that they had lost control of their lives. It was also the ideal time to test and learn the effectiveness of storytelling in project evaluation as the Outdoor Offers had started.

The Outdoor Offers are insight-led and collaborative projects, run by several of our ambassador organisations, organise activities for residents living in our geographical area of focus. Most importantly, we had our fantastic storytelling team led by YGT ambassador Norma Wilburn from NWA Social and Market Research. One thing Norma said that’s stuck with me when I mentioned using storytelling to evaluate the outdoor offers was

 ‘I don’t want anyone to feel worse for telling their story, preferably they’ll feel better about themselves.’

This sentiment, I think, is really at the heart of storytelling.

With this in mind, we set about creating a storytelling framework for each of the two Outdoor Offers, which differed slightly to reflect the different demographics they were working with. Working with the organisations involved in the Outdoor Offers, we thought about outcomes and other methods of capturing the learning. We decided on a multi-pronged approach that involved videography, interviews, written narratives, photographs, focus groups, etc.

Having different methods meant that the participant had a variety of options. Storytelling is about handing control to the participant, allowing them to reflect on their experiences in whatever medium they think is best. If they did not want to partake in the storytelling evaluation, that was OK too – this was about taking a person-centred approach that moves at the participant’s pace. We were also very open to participants coming to us with ideas of how they want their stories told. To capture longitudinal stories and change as they took place, the storytelling team and a videographer made regular visits to the groups, which helped build trust with the participants.

As a result, some participants who were nervous at the start of the projects, over time, started opening-up about their experiences. One participant said about the storytelling process:

 “I felt pretty touched that you wanted to do that, that you wanted to tell people about my story because you don’t really talk about yourself – past issues and things like that.”

Another participant, talking about her story, which was captured as a video story, said:

 “I showed everyone the video. I like reading books, so I was saying to people I might write my life story. Of course, I’m just joking with them, but I never used to be like that.”

Some participants even took an interest in the storytelling process itself. Building participants’ skillsets through the evaluation were something I had hoped for. Still, I had not anticipated the participants themselves asking if they could learn about the different methods of qualitative data capture.

The storytelling team was able to run workshops on telling a story through photographs and videos and upskill some staff members in storytelling, an unintended but brilliant outcome. Teaching participants and staff about storytelling has not only helped us with our evaluation but helped us all build our insight archive, challenge systems, and create change in our place.

We’ve also shared stories with councillors, educators, and other stakeholders to inform their processes and strategies. Now, this is not to say that using storytelling for evaluation does not have its challenges. Unlike a written survey, where questions are predetermined, and we survey participants at predefined intervals, storytelling is much more fluid and holistic approach to understanding the participant journey. As such, it requires the researcher to be present throughout the process, working alongside the participant. Meaning it can be more time-intensive than other, more traditional methods of data collection. As stories are almost always unpredictable, desired outcomes might change. So, while we created a storytelling framework that listed potential outcomes based on our insight at that time, it was constantly evolving and expanding to reflect our learning.

As a result, the framework acted more as a guide than as something to strictly adhere to. Meaning we spent more time reflecting on what we had learnt and challenging ourselves to think of different approaches. But these challenges can also be brilliant opportunities. There is no reason why storytelling cannot be used alongside more traditional data collection methods, like surveys. It does not have to be all or nothing. I have found that taking a mixed-method approach to evaluation has helped us expand our insight archive in new and creative ways, but, most importantly, it has helped us understand the why behind the what. For example, it is interesting to see the percentage of inactive people in our communities, but without stories, we cannot fully appreciate their physical activity barriers. Qualitative and quantitative data collection methods are complementary, and both can be used to evaluate a project.

And the beauty of storytelling is that it is flexible. Not having a framework set in stone means that it can change to reflect the most recent learning. Thus, while there may be some specific outcomes we might expect to see, we might discover through the process of storytelling other potential outcomes that we had not considered. Indeed, the evaluation is not just a function for collecting insight; the insight also shapes the evaluation. And during a global pandemic, the power to be flexible cannot be taken for granted!

So, is storytelling for evaluation a seamless process? No. Did it give me some anxiety? At times (especially in the beginning!). Would I do it again? Absolutely!

As said earlier, this was about test and learn. It was also about being insight-led. The decision to use storytelling for evaluation was made using local insight. As far as learning goes, I have learnt so much – not only about different evaluation methods but also about our brilliant communities.