Course Overview

Artificial intelligence is transforming how organizations deliver services, make decisions, and interact with people. As a result, a new challenge is emerging: How do we evaluate AI-driven programs to understand their real-world effects?

A new frontier in evaluation is taking shape, one that brings together technical AI testing and traditional social science methods. Evaluators need new approaches to not only assess AI system performance, but to generate credible evidence about how these systems affect people, organizations, and communities.

The instructor draws from both tech sector-led AI evaluation methods and traditional program evaluation approaches to introduce participants to this rapidly evolving area of practice. You will explore cutting-edge methods for assessing system performance, identifying risks, understanding contextual factors, and measuring human-centered outcomes and impacts. Through practical examples and applied exercises, you will learn when to use different methods, what evidence they require, and what claims their findings can support. You will also begin developing an evaluation framework for your own AI-driven initiative.

What You Will Learn

Participants will gain a practical introduction to emerging approaches for evaluating AI-driven programs. By the end of the course, they will be able to:

• Understand how technical AI evaluation and traditional program evaluation methods differ from and complement one another.
• Identify evaluation approaches appropriate to different stages of AI development and deployment.
• Recognize the strengths, limitations, and evidence requirements of emerging methods.
• Assess what conclusions different evaluation approaches can reasonably support.
• Begin developing an evaluation framework for an AI-driven program, system, policy, or organizational use case.

Course Format

The Evaluating AI course consists of one 3-hour, instructor-led module, offered virtually or in person. Participants will interactively explore practical examples, discuss emerging approaches, and apply what they learn to an AI-driven use case.

Participants will leave with an initial evaluation framework they can continue developing in their work. They will also receive a certificate of completion.

Module Breakdown

The course introduces the latest methods for AI evaluation across four complementary areas of AI evaluation:

• Developmental Evaluation: Identifying and testing potential risks during AI development through approaches such as adversarial testing, human feedback, and ethical assessment.
• Product and Performance Evaluation: Assessing how AI systems function using methods such as benchmarking, trajectory evaluations, synthetic testing, digital twins, and ongoing monitoring.
• Contextual Evaluation: Examining how AI performance and risks vary across settings, populations, and use cases through narrowed red teaming, ontological framework, context and graph mapping, and qualitative assessment.
• Human-Centered Outcome and Impact Evaluation: Assessing the real-world effects of AI-driven programs using causal and quasi-experimental designs, mixed methods, and qualitative research.

Who Should Attend

Evaluators, researchers, program managers, organizational leaders, and others interested in understanding the performance, risks, and real-world effects of AI-driven programs should take this course. The material is especially relevant for professionals who want to prepare for emerging opportunities in evaluation or who work with AI-enabled services, decision-support tools, and other applications that affect people and communities.

Prerequisites

No technical or programming background is required. Basic familiarity with evaluation, research, performance measurement, or AI is helpful but not necessary.

Participants are encouraged to bring an AI-driven program, system, policy, or organizational use case they would like to explore during the course.