by 重口50度灰QLD Committee Members
Evaluators in the 重口50度灰network are increasingly being challenged to apply evaluative thinking, methods and tools to innovative, emergent, place-based or otherwise complex initiatives. These initiatives often seek to achieve improvements not only in individuals and institutions, but in the systems that hold 'wicked' societal problems in place. The desired systems-level outcomes are often difficult to define, predict and measure and can change and evolve as the implementing organisations learn which strategies are most effective in reaching their goal.
In response, a recent issue of the 重口50度灰QLD regional committee's newsletter focussed on resources, methods and mindsets to support members to in evaluating complex systems change initiatives. Here are the take-outs.
Evaluating complex systems initiatives in a nutshell
Contributed by 重口50度灰QLD Committee Member Kiri Dicker
A complex system is a system in which there are multiple, interrelated parts and in which a perfect understanding of these parts does not automatically convey a perfect understanding of the whole system's behaviour. Systems are everywhere鈥攖hey can be ecological, mechanical, organisational, political or cultural. A system鈥檚 overall purpose or goal is achieved through the actions and
interactions of its components. In other words, the complexity of the system is integral to its effective functioning, which only occurs when all components are performing optimally (within themselves and in relation to each other).
It is important to remember that complex systems can rarely be described as a single system, rather that there are 鈥榮ystems within systems.鈥� These systems are open systems, meaning they interact with other systems, have permeable boundaries, and are affected by their external environment (often described as the context).
Photo by on
In the public health sector, systems have multiple programs, policies, agencies, or institutions at the national, state, or local level with the common goal of achieving better outcomes for children, youth, adults, or families. Programs that seek to improve the way that systems function are known as systems initiatives. These initiatives are inherently complex because they involve multiple interventions at different levels of a system, which are connected in ways that are often unclear and difficult to predict.
Coffman describes system initiatives as those that:
鈥nvolve multiple programs and players and feature outcomes at multiple levels (individual, family, community, and state)鈥hey tackle difficult deep-rooted problems such as gaps in services and outcomes based on race, income, culture, and language. Finally, all efforts to improve systems are long-term efforts that evolve over time in response to the political, economic, and social contexts around them.
Systems initiatives pose unique challenges for evaluators because they:
- explicitly seek to achieve outcomes at multiple levels across a defined system
- have multiple components (activities), that are interrelated and interdependent
- work with multiple actors and organisations and, therefore, have multiple components to evaluate
- are often deliberately iterative, seeking to test new ideas to figure out what works, and intend on evolving throughout the implementation timeframe, without establishing a 鈥榝ixed鈥� model.
For these reasons, systems initiatives cannot be evaluated in the same ways as projects or programs operating in simple contexts. For example, a key characteristic of complex systems is that they are constantly changing and evolving, and so they are often unpredictable. Therefore, the development of a comprehensive evaluation plan that overly specifies the evaluation鈥檚 key questions, evaluation approach, design, data collection and analysis methods, timeline, and budget may not be particularly helpful for evaluating complex initiatives or initiatives in complex environments.
Miller, John H., and Scott E. Page (1 January 2007). Complex adaptive systems: an introduction to computational models of social life.
Coffman, J, 鈥楢 framework for evaluating systems initiatives鈥�, Build Strong Foundations for our Youngest Children, 2017, pg. 3.
Ross, D., 鈥楶rocess engineering: A necessary step to a better public health system. Information Knowledge Systems Management鈥�, 2019, Vol. 8, pp. 299鈥�309.
Leischow, S., Best, A., Trochim, W., Clark, P., Gallagher, R., Marcus, S., & Matthews, E, (2008). Systems thinking to improve the public鈥檚 health. American Journal of Preventive, 35(2S), S196鈥揝203.
Ling, T, 鈥楨valuating complex and unfolding interventions in real time鈥�, Evaluation 18(1), 2012, 79鈥�91
Unleashing your systems thinking superpowers
Contributed by 重口50度灰QLD Committee Member Lew锚 Atkinson
Every system is perfectly designed to get the results it gets - Edward Demming
With bright eyes and a deep conviction, Zazie Tolmer responded to a question about her next steps after completion of the Children and Youth Area Partnerships (CYAPs) Project, the subject of her , by saying:
I am not sure really鈥ow that I have been exposed to this way of thinking, I don鈥檛 think I can ever go back to the way I was鈥�
She had unleashed her systems thinking superpowers.
Systems Thinking is an old (circa 1950s) yet newly rediscovered and higher orientation to life ( applying a Buddhist world view to traditional evaluation methodologies). It is a better, more natural, and holistic view of living systems (such as individuals, teams, and organisations) as they try to
survive and thrive in today鈥檚 dynamic environment. Systems thinking is a habit of mind. Fortunately, a lot like languages, it is easier to grasp at an early age. Indeed, the for systems thinking has a mission of imparting the .
My professional career spans more than 30 years now. I have experienced more of an 鈥渁wakening鈥� rather than an 鈥渦nleashing鈥� of my own systems thinking superpowers. My top 4 superpowers, upon which my professional practice is founded, include: seeing systems as a whole and the parts within them; seeing the 鈥渂ig picture鈥� of a system; seeing situations, experiences, viewpoints, and more, through the eyes of others; and seeing how understanding mental models can deepen our understanding of system structure. demonstrates the use of these superpowers to help scientists and researchers to identify 鈥渉idden鈥� barriers to translation of their discoveries into positive impact within our communities.
Photo by on
You are made of stuff that is as old as the planet, one third as old as the universe, though this is the first time that those atoms have been gathered together such that they think that they are you - Frank Close, Particle Physicist
Join Thinking to stay engaged.
Readers may be interested in the upcoming 重口50度灰workshop: Workshop: Systems Evaluation Theory 鈥� Practice & Implementation (Online 10 December 2020), facilitated by Ralph Renger, Lew锚 Atkinson and Brian Keogh.
Useful resources for evaluating complexity
Getting up to speed on evaluating complex systems initiatives can be overwhelming, so we have made it easy by curating this list of useful resources to get your started:
- Social impact consultancy has produced a series of highly useful frameworks for thinking about systems change and evaluating complexity. These include:
- This .
- developed by Julia Coffman for the Build Initiative is still a good read, despite being over 10 years old. It also includes advice on developing a high-level theory of change for a systems initiative.
- The recently updated '' by the British Medical Research Council also offers some good advice.
- Clear Horizon has produced a number of resources for public use that are suitable for evaluating complex initiatives, include:
- A and produced in partnership with The Australian Centre for Social Innovation and Collaboration for Impact
- The technique. This on how to develop a theory of change for a complex systems initiative.