(613) 818-2848

At what point does “evaluation” become “not-evaluation” and should we care?

It seems to me, as a long time evaluator-type, that we should watch where we are going at the moment. There may be some “mission creep” happening, or perhaps existential choices being made, that might be good or might be bad.

We have, over the last decade or so, expanded our view from the too-restrictive summative/formative evaluation paradigm to include developmental evaluation, where you know what you expect a program to do , or at least know some things about its goals, but may have to learn how you get there and what it really looks like when you arrive. Along the way, we saw other major developments like contribution analysis and refined ideas on theory of change that helped to provide direction and support inductive inquiry in the face of complexity, but these have not been embraced so enthusiastically.

Now we see the introduction of “principles-based evaluation”, where what is evaluated is how well you are hewing to a set of principles that you believe guide your organization. The idea is that you can get where you want to go by following your principles. This is an advantage where you are having trouble defining and measuring results, as it is likely to be easier to measure your adherence to principles. This seems like perhaps another useful part of the evaluation toolbox. I notice, however, that proponents of this seem to be willing to discount the usefulness of defining goals or a theory of change connecting actions and effects, and to downright disdain logic models. If this sounds like an overstatement, I’ll come back to it.

I think this raises some questions:

1) Are we getting to where we are more concerned with establishing some kind of claimed “evaluability” than with obtaining evaluative information, even if that means being hazy on goals and progress towards them?

2) Can you, as an evaluator, lean too much toward “learning” and too much away from “evaluating”?

3) Is the urge on the part of some to emphasize values and normative over positive analysis part of this? Is there not value in being the positive, unbiased (as much as possible), eye? If we are to become normative in our work, who are we then, and what are we trying to do? And who asked us?

I don’t claim to have definitive answers to these questions; however, I have a little story that has helped me think through this.

When I raised the idea with a leading proponent of principles-based evaluation that it would be irresponsible to spend resources if you had no goals, he used the now pretty shopworn example of 3M accidentally discovering post-it notes. That example was supposed to support the value of pure, non-goal-driven research and, to invoke the idea of “innovation”. If it worked out so well for 3M, what was the problem? OK, I didn’t say it was a long story. But I think it was pretty telling. And if my characterization of this direction in evaluation has seemed to be a bit stark, I offer that exchange as evidence in my defence.

Surely public policy, programs and public-good initiatives do not simply exist as an area for pure “research”, as in “let’s do some things and see what happens; we don’t really have anything in particular that we hope will happen, just so long as we are learning something”. And 3M having a happy accident on the way to some other set of goals doesn’t support that idea (never mind the issue of 3M’s money being their own and their shareholders’, so they can do what they want with it as long as the shareholders don’t kick them out, while publicly-funded organizations are not working with their own money).

I see the usefulness of flexible and learning-based evaluation plans, that include how you do things as well as what you do, particularly where issues and their social/economic/policy environments are complex. But I cannot see doing away with stating desired outcomes or goals or reporting and reflecting on, at the very least internally, how you are doing, as best you can tell, at approaching them.

Otherwise, I think what you are doing is not evaluation. It’s something else, and I’m not sure you should be doing it with or about public resources, or at least not without being clear about it to the source of the resources.

It’s not like program planners, social-good organizations and evaluators are forced to work in a world where nothing is known and where there is no established research. If we live in a world where a great deal of research and practice already exists, why avoid theory of change and logic models even as starting points or rough guides? What about using non-evaluation research to establish theories of change? And what about past-evaluation results, i.e. meta-evaluation? In some fields in particular, there must surely be no shortage of past evaluation and research. Do we really have to pretend that we are exploring a dark room, over and over?

Should we care?

1) Yes, we should care because it is important to at least try to do the best we can do with the scarce resources of society. If “the best we can do” changes as we learn, that’s fine. But we should be aiming at what we think is best, and we should know what we mean by “aiming”. I think policy, programs and non-profit sector initiatives should be aimed using the best knowledge available as to what treatments do and do not appear to have significant effects on their targets (go ahead and smile at the naivité).

2) Yes, we should care because this has to do with quality, standards and ethics. This point came up in a recent conversation on ethics held by the Canadian Evaluation Society. It relates to the ethics of evaluators: in particular, being prepared to refuse to alter or bury findings. You have more strength to resist this kind of pressure the more your evaluation methods are solid and defensible. You probably have less to support you if your questions aren’t clear or depend on what you find, and your data is soft or anecdotal. You are on even thinner ice if what you are doing is based on your own normative framework and values.

So, as a final question, I have to ask: is it time to step back and take a close look at the evolution of evaluation and its relationship to the public trust? Or do we no longer care if programs are reaching their goals or if, in fact, they have measureable goals at all?