What does evaluation and impact measurement like during and after the covid-19 pandemic? Stephen Miller, Chair of CHEW and Head of Impact Evaluation at Power to Change, share his reflections.
What has changed for people from start of crisis to now?
Evaluation can be complex at the best of times, but the covid-19 pandemic has created a host of new challenges. For some colleagues, work has had to stop completely. I know several freelancers and consultants who have had contracts cancelled. I know others who have been furloughed, or their colleagues or services users have. Across a host of organisations projects and programmes have been paused, cancelled or are likely to change significantly. This creates challenges for evaluation and impact measurement, especially for projects measuring distance travelled or outcomes against a particular baseline.
Most obviously, we’re also no longer able to interview face-to-face. At Power to Change we were due to start ethnographic fieldwork, spending time on location to get under the skin of the communities we support. Plus many organisations were coming to the end of the financial year, with lots of reporting from various evaluations. Power to Change is no exception. It has felt strange to reflect backwards on a reality that may no longer exist. But it has also been a useful and cathartic exercise, not least because it has helped highlight what was working in terms of social impact that can be carried forward into the future, post-crisis.
Despite these challenges, I have also observed lots of resilience and innovation amongst evaluators and researchers. In particular I noticed an initial burst of energy in terms of people wanting to collect primary data through surveys etc immediately after the lockdown began, not just from researchers but from programme colleagues too. At Power to Change we have tried our best to coordinate this, to reduce duplication and consultation fatigue. Instead, we have been placing a strong emphasis on doing research which has mutual benefits for the participants where possible. That has included a stronger emphasis on qualitative data collection, which provides interviewees with a chance to reflect on what is happening and crystallise their learning.
One way this is happening, is that we are asking people to record themselves on their smartphones, reflecting on daily life as they and their organisations and wider communities respond to the crisis. We are also using secondary datasets a lot more, including ‘live’ data (e.g. debit and credit card transaction data), in order to remove reporting pressures and the need for primary data collection. So in the case of Power to Change, this has enabled us to understand which of our grantees are most vulnerable to going bust due to the crisis.
We’ve also been supporting our colleagues to adopt a data-informed approach, more so than before. This is taking two principle forms:
Firstly, we’ve pivoted some evaluations to adopt a more developmental evaluation approach, so the evaluators are more regularly feeding back learning and insight to delivery teams
Secondly, we’re conducting our analysis out in the open, sharing cloud-hosted documents with colleagues so they can access our analysis in real-time as it is being conducted.
What we can learn from this to take forward for the future?
I think programme and service design is likely to change as a result of this crisis. In particular I anticipate that we may see fewer projects built on linear models and assumptions, and more which are premised on more person-centered and relational approaches.
What that means for impact measurement is yet to be seen, but my guess would be that a more developmental approach will be required, where regular learning loops are built in between evaluators and delivery teams, so that the latest insight can be used to inform the next iteration of programme design.
Can you share one resource, tip, or piece of support with the group?
CHEW is partnering with a host of organisations including Inspiring Impact, SVUK, NPC, Coalition for Efficiency, UK Evaluation Society (UKES), and UnLtd - to deliver coordinated support for research and evaluation during covid-19. Inspiring Impact have launched a web-page sign posting people to relevant existing resources and online events: https://www.inspiringimpact.org/impact-support-during-covid-19/
Article adapted from a panel discussion as part of the Social Value Exchange series facilitated by SVUK. A full write-up of the session is available here.