AI in Evaluation: How to start responsibly

Ella Nation-Tellery, Impact and Learning Manager at ClientEarth

Key takeaway: Start small, be clear about where AI adds value, protect sensitive information, and always keep human judgement and accountability at the centre.

Over the past couple of years, ChEW has hosted a series of practical sessions exploring how organisations are approaching AI. Members shared their experiences experimenting with and embedding AI in their work. In this first of two blogs, we share key lessons on getting started, both from an individual and organisational perspective.

These blogs draw on experiences shared by CHEW members through our AI peer learning sessions, with contributions from Naomi Tyrell, Rocket Science and many others across the network.

Getting started as an individual 

These tips from Naomi Tyrell take you from first experiments to building tools that save you time:

1. Start small and low risk. Experiment with simple low risk tasks such as email drafting or writing LinkedIn posts. 

2. Practice your prompts. The more context and direction you give, the better the output. Try using this framework to optimise the outputs:

3. Check and challenge what AI tells you. AI can sound confident while getting things wrong or making things up (known as "hallucinations"). Treat its outputs as a first draft, not a final answer. 

• Fact check key points against trusted sources.

• Ask the AI to review its own answer for errors or inconsistencies.

• Tell it what it got wrong and ask it to revise.

4. Build your own assistant for repetitive tasks. Once comfortable with the above, you can set up a custom assistant for tasks you do regularly. You could build a "custom GPT" in ChatGPT, and other tools offer similar features. You can also share these assistants with colleagues. For research, members recommended SciSpace and Consensus as particularly useful for literature and systematic reviews. 

Getting started as an organisation

Individual experimentation is a great start, but embedding AI across an organisation needs a shared direction, clear guardrails and support for your whole team. Reflecting on their own organisation’s journey, Rocket Science shared the considerations that have helped them embed AI effectively. 

1. Anchor AI in your strategy and values. Be clear about how AI and machine learning (ML) support your mission, strategy and values. Where possible, co-design your approach with staff so it reflects how people actually work. CHEW members ITAD made integrating generative AI into its services and operations one of its five strategic priorities. Appointing AI champions and piloting specific use cases helped them build momentum and raise awareness across teams. 

2. Assess your organisation's readiness and risk appetite - take stock of your skills, systems, data and capacity, and agree how much risk you're comfortable with. A SWOT analysis can highlight internal strengths and gaps. A PEST analysis can surface external pressures such as regulation, funder expectations and public trust.

3. Agree where AI fits, and where it doesn't - a simple decision tree or set of guidelines makes sure everyone works to the same standards, especially around sensitive or personal data.  

4. Set organisational commitments and boundaries - be explicit about what you will and won't use AI for, and share this with staff, partners and funders. For example, Rocket Science has committed to keep working with freelancers, particularly in the creative industries, and not to use AI for design, analysis or report writing.

5. Keep learning and bring in expertise - AI is changing fast. Bring in external expertise to build your team's skills and to help align your policies and practices with your strategy and values.

Practical Do’s and Don’ts…

Do ✅

• Check the AI policy of your organisation or the organisation you are working with

• Check tool compliance - use tools already approved by your organisation, or the organisation you are working with to safeguard data 

• Do it yourself if there is a high risk of hallucination

• Build in checking time

• Be clear and transparent about where you have used it and to what effect

Don’t ❌

• Don’t use AI tools for work purposes unless they have been approved by your organisation and are appropriate for the type of data you’re handling

• Do not share confidential data in unsafe tools

• Don’t use AI to make decisions that have a direct or significant impact on people’s lives. Keep human judgement and accountability at the centre of these decisions

Keep people at the centre

And to wrap up, a quote from Terminator courtesy of Caroline Masundire from Rocket Science… “The future’s not set. There’s no fate but what we make for ourselves”. John Connor, 1984.

An important reminder that AI should augment, not replace the work of an evaluator. It should always be combined with human judgement, be human-driven and human-informed!