A Hierarchy of Program Evaluation Metrics
We're diving into researching data-based decision making for a couple of projects, which includes metrics for program evaluation. Evaluation is a well trod field --a lot of people have said a lot of things about it... but I find a lot of it contradictory or not that helpful. I especially dislike the commonly used nomenclature of "outputs" and "outcomes." First, is it even possible to find two more confusingly similar words to represent two different concepts? Second, the terms aren't even used consistently and specifically-- there's a fair amount of overlap where one authority might call something (say, attendance at a workshop) a output when someone else would call that same thing an outcome.
The idea of outcomes has also been raised to such a degree (often by funders) that nonprofits often think that it's really important to get to outcomes as downstream in their work as humanly possible. If tracking the % who completed a course or who said it made a difference to them is good, then it's even better to gather and track the long term satisfaction. And if that's good, then we all should be shooting for the holy grail: measuring what impact our programs have on people's lives and community.
I disagree with this notion. It encourages nonprofits to try to fly before they can walk. They need to start by measuring the tactical things that are close at hand and get that down before moving to more complex measures. Even more, the idea that small nonprofits should be trying to measure impact writ large is insane to me -- trying to tease out the actual impact of one organization's actions, separating it from the impact of other actions and drivers, is work for professional researchers and, often, multi-million dollar studies. There's no way that any but a huge nonprofit is -- or should be-- staffed to conduct this kind of study. When we (or, ahem, funders) encourage small nonprofits to try to measure this stuff, most often it can only result in bad research.
Well, none of us can ignore what funders think is important, or the commonly used vocabulary for such an important concept... but I can, as I'm prone to do, try to create more order and structure around it. Here's my look at a hierarchy of nonprofit-focused program evaluation metrics, and how organizations can think about their priority.
What do you think? Would you agree? What have I missed?