(613) 818-2848
Menu

Logic models and complex system mapping for evaluation, performance measurement and planning

Taking time to think through how you understand your work to lead to results is one of the most valuable things you can do. With facilitated sessions, you can work through your theory of change and your logic model, that is, how your organization actuates change. You can work backwards and ask how what you have done has contributed to what you see, in the context of a complex world. Sound facilitation technique and state of the art system mapping tools help engage the room and capture the results.

I have constructed many logic and theory of change models in my evaluation and performance measurement projects, and continue to do so. There are many ways to produce a model, all the way from a program manager or evaluator drawing one up in a hurry, never to be used again, to to participatory processes involving management, staff and stakeholders in identifying and validating their intended/expected outcomes, and through an identified and agreed theory of change (or one that is accepted pending further experience) determining how their initiative can be expected to deliver results and how to tell if that expectation is in fact being borne out by evidence. As an experienced evaluator and facilitator, I support the latter model.

One serious problem, however, even with most thoughtfully constructed logic models, is that they usually leave out external influences and feedback effects, even when they are likely to be important, because they make the model “too complex”. It is good to simplify, but ignoring important influences on program success when planning a program or an evaluation is poor strategy. Failing to consider potentially important external influences and other complexities essentially places hopes for success on a best case scenario. Moreover, this may lead evaluators to fail to collect important data and to misinterpret program results.
systems dynamics - tobacco control initiative
Trying to embrace complexity by simply drawing a web of boxes and arrows like this is not helpful: it’s too complex to use and explain; will drive your audience away; and will probably come only from the mind of the evaluator or program manager, thereby easily missing important external influences and other complexities.

Fortunately, the same technology as used in social network analysis, along with more standard outcome mapping facilitation techniques, is perfect for uncovering and highlighting the patterns and structure of complex systems. Using existing research and/or facilitated sessions with experts and stakeholders, you can map out elements of cause and effect for evaluation, research and planning.

The advantage of a network analysis-based methodology is that it can build logic models and theory of change mappings that untangle complex relationships and external effects that would otherwise not be possible or practical, for policy planning, performance measurement and program evaluation.

In this simple demonstration version, the links between a program, its resources and activities, and its outcomes, as well as external factors, can be seen as parts of a system, using the network mapping technology. Moreover, the centrality of certain parts of the system can be clearly seen. At the same time, the model can be presented in a more standard form, as shown at top, if you like, once the relationships are understood and validated.

My approach is to develop a first draft mapping using existing material that can be then validated in a facilitated session with staff, experts or stakeholders, as appropriate. I bring a varied set of tools to help in illustrating the connections between elements of a system or logic model, including visuals that can easily be revised during an expert or stakeholder data validation session, using live meeting-oriented tools from sticky-note-based affinity maps and storyboards to interactive software like Kumu and others.

The data and visuals are then also used in reporting, like this interactive matrix displaying strengths of connection (read the causality as “y-axis drives x-axis”) and real-time-revisable network analysis software. The interactive matrix is used along with a network mapping, onscreen to help guide validators through potentially complex chains of influence and effect.

nodexl-live-panel-screen-shot

 

Mapping of dynamic systems is equally powerful for understanding complex webs of organizations, functions and results, such as, for example, my recent mapping of Ontario’s immunization system. This approach allowed the client to focus on key connections in the midst of a large and complex set of factors.

As the 2016 Conference Chair of the American Evaluation Association Topical Interest Group on Social Network Analysis I was very happy to give a presentation entitled ”Getting comfortable with complexity: a network analysis approach to program logic, design and evaluation”, covering this subject in detail.

My dynamic system mapping clients include the Ministry of Health and Long Term Care, Ontario, the United Way of Calgary and Area and Human Services, Alberta.
dynamic-system-mapping-clients