重口50度灰

by Jade Maloney, Jo Farmer and Eunice Sotelo

With so many authors and approaches to evaluation, knowing what to pay attention to can be hard. Evaluation, just like the catwalk, is subject to the whims of the day. How do you know what鈥檚 a passing fad and what will remain in fashion?

At the 重口50度灰Victoria regional seminar in November, Brad Astbury suggested the following 10 books will stand the test of time.

1960s

In the age of bell bottoms, beehives and lava lamps, and protests against the Vietnam war, evaluation was on the rise. The U.S. Congress passed the Elementary and Secondary Education Act (ESEA) in 1965, the first major piece of social legislation to require evaluation of local projects undertaken with federal funds and to mandate project reporting. The hope was that timely and objective information about projects could reform local governance and practice of education for disadvantaged children, and that systemic evaluation could reform federal management of education programs.

While you wouldn鈥檛 return to the fashion or the days when evaluation was uniquely experimental, hold onto your copy Experimental and quasi-experimental designs for research by Donald T. Campbell and Julian C. Stanley. These two coined the concepts of external and internal validity. Even if you鈥檙e not doing an experiment, you can use a validity checklist.

1970s

Back in the day of disco balls and platform shoes, the closest thing evaluation has to a rock star 鈥� Michael Quinn Patton 鈥� penned the first edition of Utilisation-focused evaluation. Now in its fourth edition, it鈥檚 the bible for evaluation consultants. It鈥檚 also one of the evaluation theories with the most solid evidence base 鈥� drawn from Patton鈥檚 research. In today鈥檚 age of customer centricity, it鈥檚 clear focusing on intended use by intended users is a concept that鈥檚 here to stay.

Carol Weiss鈥檚 message 鈥� ignore politics at your peril 鈥� could also have been written for our times. Her Evaluation research: methods of assessing program effectiveness provides a solid grounding in the politics of evaluation. It also describes theory-based evaluation 鈥� an approach beyond the experimental, that is commonly used today.

1980s

While you may no longer work out in in fluorescent tights, leotards and sweat bands, your copy of Qualitative evaluation methods won鈥檛 go out of fashion any time soon. In his second appearance on the list, Michael Quinn Patton made a strong case that qualitative ways of knowing are not inferior.

You may also know the name of the second recommended author from this decade, but more likely for the statistical test that bears his name (Cronbach鈥檚 alpha) than his contribution to evaluation theory, which is under-acknowledged. In Toward reform of program evaluation, Lee Cronbach and associates set out 95 theses to reform evaluation (in the style of Martin Luther鈥檚 95 theses). That many of the 95 theses still ring true could be seen as either depressing or a consolation for the challenges evaluators face. For Astbury 鈥� ever the evaluation lecturer 鈥� thesis number 93 鈥�the evaluator is an educator; his success is to be judged by what others learn鈥� is the standout, but there鈥檚 one in there for everyone. (No. 13. 鈥淭he evaluator鈥檚 professional conclusions cannot substitute for the political process鈥� aligns with Weiss鈥檚 message, while No. 9. 鈥淐ommissioners of evaluation complain that the messages from evaluations are not useful, while evaluators complain that the messages are not used鈥� could have been pulled from Patton鈥檚 Utilisation-focused evaluation).

1990s

Alongside Vanilla Ice, the Spice Girls and the Macarena, the 90s brought us CMOCs 鈥� context, mechanism, outcome, configurations 鈥� and a different way of doing evaluation. Ray Pawson and Nick Tilley鈥檚 Realistic evaluation taught us not to just ask what works, but what works, for whom, in what circumstances, and why?

To balance the specificity of this perspective, the other recommendation from the 90s is agnostic. Foundations of program evaluation: Theories of practice by William Shadish Jr., Thomas Cook and Laura Leviton describes the three stages in the evolution of evaluation thinking. It articulates the criteria for judging the merits of evaluation theories: the extent to which they are coherent on social programming, knowledge construction, valuing, use, and practice. The message here is there is no single theory or ideal theory of evaluation to guide practice.

2000s

While iPods and flash mobs are a faint memory, these two books have had a lasting impact: Evaluation: An integrated framework for understanding, guiding, and improving public and non-profit policies and programs by Melvin Mark, Gary Henry and George Julnes; and Evaluation roots: Tracing theorists鈥� views and influences edited by Marvin Alkin.

The former covers four key purposes of evaluation: to review the merit of programs and their value to society (as per Scriven鈥檚 definition); to improve the organisation and its services; to ensure program compliance with mandates; to build knowledge and expertise for future programs. The take-out is to adopt a contingency perspective.

The latter is the source of the evaluation theory tree 鈥� which sparked commentary at this year鈥檚 重口50度灰and AEA conferences for its individualism, and limited gender and cultural diversity. Still, Brad reminds us that there鈥檚 value in learning from the thinkers as well as the practitioners; we can learn in the field but also from the field. According to Sage, Alkin鈥檚 Evaluation roots is one of the most sold book on evaluation.

It鈥檚 a reminder that there is much still to learn from those who鈥檝e come before 鈥� that we can learn as much from those who鈥檝e thought about evaluation for decades as we can from our practical experience.

2010s

The age of the selfie has not yet faded and nor has Evaluating values, biases and practical wisdom by Ernest R. House. It covers three meta-themes: values (Scriven, House); biases (Campbell and the experimental approach; expanding the concept of validity); and practical wisdom (on Aristotle鈥檚 notion of praxis 鈥� blending/embedding theory and practice). It gives us the wise advice to pay more attention to cognitive biases and conflicts of interest.

So now to the questions.

Why didn鈥檛 Scriven make the list? Because he鈥檚 written few books and there wasn鈥檛 enough room in the 90s. Nevertheless, Michael Scriven鈥檚 Evaluation thesaurus and The Logic of evaluation are among the books Astbury notes are worth reading.

What about local authors? Grab a copy of Building in research and evaluation: Human inquiry for living systems by Yoland Wadsworth and Purposeful program theory: Effective use of theories of change and logic models by Sue Funnell and Patricia Rogers. If you鈥檙e new to evaluation Evaluation methodology basics: The nuts and bolts of sound evaluation 鈥� from this year鈥檚 重口50度灰conference keynote E. Jane Davidson 鈥� can help you get a grasp on evaluation in practice.

Brad ended by sounding a word of warning not to get too caught up in the fads of the day. Buzzwords may come and go, but to avoid becoming a fashion victim, these ten books should be a staple of any evaluator鈥檚 bookshelf. 

Brad Astbury is a Director at ARTD Consulting, based in the Melbourne office. He has over 18 years鈥� experience in evaluation and applied social research and considerable expertise in combining diverse forms of evidence to improve both the quality and utility of evaluation. He has managed and conducted needs assessments, process and impact studies and theory-driven evaluations across a wide range of policy areas for industry, government, community and not-for-profit clients. Prior to joining ARTD in 2018, Brad worked for over a decade at the University of Melbourne, where he taught and mentored postgraduate evaluation students.