“Agile” is a popular phrase in workplaces these days, particularly in the fields of technology and product management. This approach, however, - often characterised by being quick and iterative and following short cycles of work - is very much used in the research and evaluation space too.
One of our key principles and values at ImpactEd Evaluation is being flexible and nimble in response to partner needs, which goes hand-in-hand with an agile approach to evaluation. Dealing with the realities and complexities of doing evaluation in the education and youth sectors, we ensure our evaluations provide quick and iterative insights where needed.
Drawing on our experiences of this kind of work in the education and youth space specifically, here are some key ways in which you can ensure an agile approach:
Make the best use of existing data
Before you go out and collect lots of new data, think about what you already have available to you. What might you be able to do without collecting any new data? Often organisations we work with have loads of data sitting around that isn’t being utilised to its full potential. If resources are tight, then it might be cheaper and faster to wrangle some insight out of existing data than it is to start collecting data from scratch.
Prioritise what you want to know
When setting out your evaluation plans, it can be tempting to make a really comprehensive list of outcomes and measures you want to collect data on, however this often ends up with highly impractical and time-consuming data collection plans. Be targeted about what you want to know, and prioritise accordingly. Really think about what pieces of information will enable you to make the most valuable decisions, and prioritise these. Sometimes it's better to measure one thing reasonably well than it is to measure lots of things very superficially.
In line with this, think about whether you need breadth or depth from your dataset. Sometimes it is better to use whatever resource you have to measure one small thing quite closely (e.g. "is my intervention actually being delivered as I intend it to be" or "what effect is this having on this one outcome in this one school?") than it is to get a very blurry overall picture. In other cases, the opposite might be true, and you might want to collect shallow data on a range of things, since this might help you identify areas to commit more evaluation resources to in the future.
Consider proxies for what you want to measure but can’t
Sometimes it might not be very practical to collect data on the measure you are interested in. In these cases, try to see if there’s some kind of proxy you can use instead. For example, are there certain bits of data that you could use as a crude proxy for what you're really interested in? As long as you are clear about the assumptions you are making and any limitations, this can be really useful. As an example, you might not be able to afford a propensity score matched control group of participants from very similar socioeconomic backgrounds to your participants. But could you use public data or any national benchmarks to get a useful comparison figure?
Live dashboards for early insights
Getting the data you need is one thing. Analysing this data and getting useful insights is another. Try to set up or use systems and processes that allow for quick and early analysis. This will help you with early insights, meaning you can make timely decisions. And while there will remain a place for in-depth analysis and reporting on a less frequent basis, this kind of reporting will help you move things along in the meantime.
At ImpactEd Evaluation, we have developed a platform called the School Impact Platform, which allows for data collection from schools (e.g. demographics, school engagement data, survey data) and live reporting of findings. This means our partner organisations and schools have live insights into the progress their pupils are making. When we are working with organisations who work outside of the formal school sector, we also have experience of developing customised dashboards that organisations can use for ad hoc reporting.
Interim reporting in accessible formats
Once you have early insights, don’t wait until it’s time for your annual report to share them more widely, either internally or externally. Use early insights to feed into learning conversations, development discussions and more and make sure to do this in an accessible format. In some instances, this might be a slide deck, an infographic, or simply a one-page summary paper. While there is a place and time for longer, technical reports, a rapid approach requires more agile and easier to digest formats.
Kristy Evers is the Director of Impact Partnerships at ImpactEd and is also a ChEW Trustee.