The Logic Model: A Tool for Incorporating Theory in Development and Evaluation

of Programs

Riki Savaya, PhD Mark Waysman, PhD

ABSTRACT. The role of program theory in developing, operating and evaluating programs is gaining increased emphasis in recent years. A clear grasp of a program’s theory can contribute to the successful perfor- mance of many important developmental and evaluative tasks along its life span. Although there is growing recognition of the importance of program theory in the development and evaluation of programs, it should be noted that this is not a simple task. Programs are often com- plex, comprising many different types of interlinking components. What is needed is a relatively simple instrument that can help the practitioner explicate and present program theory, by guiding and structuring the process. The logic model is such a tool, whose purpose is to describe and articulate program theory. Drawing upon examples from the authors’

Riki Savaya is affiliated with the Bob Shapell School of Social Work, Tel Aviv Univer- sity, Israel, and the Center for Evaluation of Human Services, Rishon Lezion, Israel. Mark Waysman is affiliated with the Center for Evaluation of Human Services, Rishon Lezion, Israel.

Address correspondence to: Dr. Riki Savaya, Bob Shapell School of Social Work, Tel Aviv University, P.O.B. 39040, Ramat Aviv 69978, Israel (E-mail: [email protected] tau.ac.il).

This article is based on a paper presented at the International Conference on Evalua- tion for Practice, University of Huddersfield, England, 12-14 July, 2000.

Administration in Social Work, Vol. 29(2) 2005 http://www.haworthpress.com/web/ASW

© 2005 by The Haworth Press, Inc. All rights reserved. Digital Object Identifier: 10.1300/J147v29n02_06 85

work with both nonprofit and governmental organizations, this paper presents potential uses of the logic model tool in explicating program theory for a variety of purposes throughout the life span of programs: for assessing the feasibility of proposed programs and their readiness for evaluation, for program development, for developing performance mon- itoring systems, and for building knowledge. [Article copies available for a fee from The Haworth Document Delivery Service: 1-800- HAWORTH. E-mail ad- dress: <[email protected]> Website: <http: //www.HaworthPress. com> © 2005 by The Haworth Press, Inc. All rights reserved.]

KEYWORDS. Logic model, program theory, program evaluation, per- formance monitoring systems

Over the last decades the literature on programs and interventions in the field of social work reflect a growing emphasis on incorporating theoretical frame- works in practice and in program evaluation (Alter & Egan, 1997; Alter & Murty, 1997; Mizrahi, 1992; Reid, 1994; Rosen, 1993). This implies a clear definition of the population, problems and outcomes that are the focus of any program, a clear presentation of the theoretical assumptions that guide the choice of intervention, and systematic assessment of effectiveness. Current professional standards for judging the quality of a program and assessing its readiness for dissemination thus require a high level of clarity in conceptual- ization and presentation of programs.

Concurrently, literature in the field of program evaluation shows a growing emphasis on the role of program theory in developing, operating and evaluat- ing programs (Bickman, 1987, 1990; Chen, 1990; Weiss, 1997). Wholey (1987) states that program theory identifies “program resources, program ac- tivities, and intended program outcomes, and specifies a chain of causal as- sumptions linking program resources, activities, intermediate outcomes, and ultimate goals” (p. 88). As such, program theory lays out the assumptions that are held regarding the transformative mechanisms that are considered to be ac- tive in creating change and growth (Lipsey, 1987; Pawson & Tilly, 1997).

Several authors have pointed to the fact that, in many cases, program theo- ries exist but are never made explicit (Cook, 2000; Leeuw, 2003). Schon (1983, 1995), for example, points out that even experienced practitioners may well be unable to transcend their concrete actions and translate their tacit knowledge into abstract concepts that may be examined and applied beyond their own setting. Although their activities may reveal an inner logic, in which the outcome of one action sets in motion a process of reflection which prompts

86 ADMINISTRATION IN SOCIAL WORK

the next action, they may well be unable to explain the connections. The knowledge they demonstrate in their actions remains implicit. The series of activities that unfolded on the basis of their tacit considerations and knowl- edge is difficult to transfer to others, so that they can learn from them, and dif- ficult to replicate elsewhere.

Thus, while there is growing recognition of the importance of program the- ory in the development and evaluation of programs, and there is often an ex- pectation that theory be addressed, this is clearly not a simple task. Programs are often complex, comprising many different types of components. Attempt- ing to integrate them all into a single coherent framework may prove daunting.

As such, what is needed is a relatively simple instrument that can help the practitioner by guiding and structuring the process. The logic model frame- work is an ideal tool for this purpose. The logic model is a diagram that describes “how the program theoretically works to achieve benefits for participants. It is the ‘If-Then’ sequence of changes that the program intends to set in motion through its inputs, activities, and outputs” (United Way of America, 1996, p. 38). The logic model captures the logical flow and linkages that exist in any program. Even in cases where the theory of a program has never been made explicit, the logic model approach can help to uncover, articulate, present and examine a program’s theory.

Although there are several ways to represent the logic model, it is usually set forth as a diagram, resembling a flow chart, that contains a series of boxes linked via connecting one-way arrows, as presented in detail, for example, in recent manuals published by the United Way (1996) and the W. K. Kellogg Foundation (2000). The basic logic model includes the following components:

Inputs: the human, financial, organizational, and community resources that need to be invested in a program so that it will be able to perform its planned activities.

Activities: what the program does with the inputs; the processes, events, and actions that are an intentional part of the program implementation.

Outputs: the direct products of program activities, usually measured in terms of the volume of work accomplished (e.g., the number of classes taught, number of group meetings held, number of pamphlets distributed, number of announcements broadcast on radio or TV) and the number of people reached (number who attended each meeting, number who received written materials, etc.).

Outcomes: the benefits or changes in the program’s target population; for example, changes in knowledge, perceptions, attitudes, behavior,

Riki Savaya and Mark Waysman 87

or status. Programs often posit a chain of outcomes that are linked to each other in a logical sequence over time, with immediate outcomes leading to intermediate outcomes, which in turn lead to long term-outcomes. For instance, it may be expected that new knowl- edge and increased skills (immediate outcomes) will lead to modi- fied behavior (intermediate outcomes), which will lead, in turn, to improved condition (long-term outcome).

Inputs or resources to the program usually appear in the first box at the left of the model, and the longer-term outcomes are presented on the far right. In between these ends, the major program activities are boxed, followed by in- tended outputs and outcomes from each activity (McLaughlin & Jordan, 1999). Figure 1 presents the basic format of a generic logic model. Actual models are usually much more complex, since programs often contain a vari- ety of activities, each of which may require its own inputs and also may lead to specific outputs, that are directed at either common or separate outcomes. Graphic presentation of this complexity may thus require more boxes to rep- resent the different components and more arrows to depict the linkages be- tween them (for examples, see Bickman, 1990; Wandersman et al., 1997; Weiss, 1997).

The logic model is a tool that practitioners and evaluators are finding increasingly useful in explicating and presenting program theory for many purposes, as witnessed by the publication of user manuals (e.g., W. K. Kellogg Foundation, 2000; United Way of America, 1996) and profes- sional papers (for example, Alter & Egan, 1997; Cooksy, Gill, & Kelly, 2001; Cozzens, 1997; Julian, 1997; McLaughlin & Jordan, 1999). The structured nature of the end product (the logic model diagram) helps to structure and focus the process of articulating the program theory by di- viding it up into discrete units (inputs, outputs, outcomes, etc.) that are connected via links that can be readily examined for logic and feasibility.

Most existing publications, however, have focused on a limited range of uses. In this paper, we demonstrate potential uses of the logic model tool in explicating program theory for a variety of purposes, via an inte-

88 ADMINISTRATION IN SOCIAL WORK

Inputs Activities

Situation: Target Population & Needs

Outputs Initial Outcomes

Intermediate Outcomes

Long-Term Outcomes

FIGURE 1. Basic generic logic model.

grative framework based on the life span of programs (see Figure 2). The paper consists of a series of case studies, taken from our work, that illus- trate the different uses of logic models at different points in time through- out the life cycle of programs.

PROGRAM INCEPTION STAGE: USING LOGIC MODELS FOR ASSESSING PROGRAM FEASIBILITY

AND READINESS FOR EVALUATION

Purpose

A clear and concise presentation of program theory can be extremely useful prior to program implementation and prior even to the detailed planning of programs, at the early stage, when the idea for a program is first being consid- ered. When programs are first being conceived and examined, feasibility and evaluability assessment help us to determine (a) whether there is a good chance that the program can, in fact, be implemented with the given resources, (b) whether the program has a reasonable chance of achieving its intended out- comes, and (c) whether the program has been presented in a sufficiently clear and detailed manner to allow for the planning of its evaluation. A proper as- sessment can prevent investment of resources on programs that, with some forethought, could have been seen in advance to be impractical or ineffective. For this purpose, program planners need to clarify several things: What does the program intend to achieve (outcomes)? For whom (target population)? Via which activities? and With what resources? The resources available must be sufficient to implement all the planned activities, these activities must be ap- propriate to the target population (age-appropriate, culturally sensitive, etc.), and they must also logically have a reasonable chance of leading to the desired outcomes. In the following example, the authors demonstrate the use of the

Riki Savaya and Mark Waysman 89

Program Inception Program Modification Program Operation

LM for assessing program feasibility and its readiness for evaluation

LM for program development

LM for developing performance monitoring system

Program Dissemination

LM for knowledge building

FIGURE 2. Uses of the logic model (LM) along the life span of a program.

logic model to assess the evaluability of a program, prior to its commence- ment.

Context

The Special Projects (SP) department within the Planning and Research di- vision of a large Israeli government agency funds pilot testing of innovative new social welfare programs that are subsequently to be disseminated by other public agencies. One of the conditions for funding any program is that its initi- ators and managers agree for it to be accompanied by external evaluation. As each program is approved for funding, qualified evaluators are approached and asked to submit evaluation proposals.

Example

In this case, we were sent a copy of the program proposal that a government agency had submitted to the SP department for running a program geared to helping a specific group of parents, who had been separated from their chil- dren. The program’s aim was to help these parents prepare for resuming con- tact with their children and to help them develop their parenting skills. We were asked to study the document and submit a proposal of our own for evalu- ation of this program that had been approved for funding and was about to commence operation.

Review of this agency’s proposal raised a number of questions about the program’s readiness for both implementation and evaluation. We therefore decided that prior to preparing a formal evaluation proposal, we would attempt to systematically examine its feasibility and evaluability (Wholey, 1987) by assessing the program theory with the help of a logic model. Using the compo- nents of the logic model described above (target populations, problems, re- sources, activities, outputs, outcomes), we read through the proposal and marked any statement that could be classified as pertaining to any one of them. We then assembled all the statements in each category and tried to see if we could arrive at a clear picture of each component. Finally, we examined whether there appeared to be logical linkages connecting these components to each other. By the end of this process, we had some serious concerns that this program, at least as it appeared in writing, may not be ready for implementa- tion or for evaluation: Some of the activities appeared to be very difficult to implement and the intervention itself did not seem to have a reasonable chance of leading to its stated outcomes.

For example, the program’s target population was defined in different ways throughout the proposal: “mothers or fathers of children up to age 17, who

90 ADMINISTRATION IN SOCIAL WORK

have not lived with them for at least six months”; “any father who has at least one child under the age of 17 or whose wife is pregnant”; “any father about to return home [no specification of children’s ages].” It is apparent that the target population was defined differently in different places, such that it was not clear to us whether the program was intended for fathers only or for both fa- thers and mothers; whether a minimum period of separation was required; and whether or not the program was limited to actual parents or was also to include parents-to-be.

The program’s planned activities were not specified in the proposal, be- yond a general statement that they will include 12 weekly group meetings, each lasting two hours. It was, therefore, not possible to infer from the pro- posal what experiences these parents were going to undergo that were ex- pected to lead to the desired outcomes, which, in turn, were also not specified beyond general statements about preparing the parents for their return home and enhancing their parenting skills. Figure 3 illustrates how the logic model helped us to identify and clarify the problems in this program proposal and to present them in a succinct and clear visual schema.

While the reader may suspect that we have singled out here an extreme ex- ample of poor or vague program planning, from our experience, this is unfor- tunately not the case. Programs are usually planned by practitioners, who have little experience in planning and developing new interventions and are not usually aware of the importance of explicating the program theory when de- veloping a program proposal. Without the aid of a simple and structured tool for reviewing programs, such as the logic model, it is easy to “get lost” and end up with a plan that is full of holes or contradictions.

Although we were unable in this case to specify a clear logic model, our at- tempt to do so guided us in converting a long, diffuse proposal into a clear, concise, structured and focused document that highlighted the lacunae in the theory underlying this program. This document was sent to the SP department, along with our reply, in which we declined their offer to submit an evaluation

Riki Savaya and Mark Waysman 91

Situation Inputs Activities Outputs Outcomes

different and contradictory definitions of the target population: no specification of needs

OK group meetings contents unspecified

− 12 weekly meeetings, each lasting two hours

general, non-specific statements, not operationally defined

FIGURE 3. Example of logic model to help assess program feasibility and readiness of evaluation.

proposal at this point in time, since it appeared to us that the program was not yet ready for evaluation. Instead, we recommended that the agency should work on clarifying and specifying the program theory prior to its implementa- tion and prior to planning any evaluation.

Although we do not know whether this recommendation was accepted, in- terestingly, the SP department later decided to contract with evaluation con- sultants to help them assess program proposals, prior to their final approval. Previously, evaluators had been included in the process only after approval of funding, at which point it can be very difficult to make major changes in a pro- gram.

PROGRAM MODIFICATION STAGE: USING LOGIC MODELS FOR IMPROVEMENT

OF EXISTING PROGRAMS

Purpose

In many cases, initial program planning does not include a clear specifica- tion of all program components and a systematic examination of the logical linkages between them. Although this may not necessarily be a problem, there may be cases where difficulties become apparent after a program has been in operation for a while. Logic models can also be beneficial in these cases, where they may focus formative efforts at program improvement. In the fol- lowing example, we constructed and elaborated the logic model of an existing program with management and staff of the Department of Social Welfare in a large Israeli city, in a collaborative process that lasted almost a year, based on the findings of an implementation evaluation.

Context

The Welfare Department established a network of Family Aid Centers (FACs), aimed at improving services to those segments of the population that had not been considered a high priority in the past. In particular: (a) clients who had applied only for material or instrumental assistance and (b) those for whom traditional approaches (such as psychotherapy and counseling) had been exhausted but who were still in need of other forms of assistance. These neighborhood-based FACs were to provide instrumental and material help, in a fast, efficient and courteous manner, without requiring an in-depth psychosocial assessment and without the prior prerequisite, that receipt of instrumental aid would be dependent on the client’s willingness to participate in counseling.

92 ADMINISTRATION IN SOCIAL WORK

The FACs were also supposed to develop new and innovative services for these groups and, in particular, were to promote the use of advocacy strategies to help these clients exercise their rights, within both the social welfare system and other public agencies.

Example

After the FACs had been operating in six neighborhoods within the city for a period of about two years, it was decided to evaluate their implementation and generate recommendations for enhancing the model prior to expanding the program to additional sites throughout the city. The evaluation had four stages:

1. to uncover and elucidate the FACs’ existing conceptual model (mission, policies, and practices);

2. to assess the implementation and effectiveness of this model; 3. to identify gaps between the conceptual model and its implementation

and results in practice; and 4. to facilitate development, operationalization and consolidation of a re-

vised model geared to narrow the above gaps.

Construction of a logic model was the tool that we used in the fourth stage. First, we presented our findings from the previous stages, regarding the FACs’ existing conceptual model and its implementation, which revealed disagree- ments and lack of clarity among stakeholders regarding program components, such as target populations and required inputs, as well as discrepancies be- tween the program plan and its implementation in practice.

Based on these findings, we initiated an interactive, participatory process involving ourselves and two separate groups within the department (adminis- tration and program staff). Over a period of eight months, we held a total of 15 meetings, each lasting approximately three hours, during which we utilized the logic model tool to develop and refine the FAC program. The components of the logic model guided and structured this group process. Each meeting was devoted to defining and elaborating one component of the logic model, which in this case began with the program’s “mission.” Only after the two groups had reached agreement on the written formulation of a component did we progress to the next one. Since each component in the logic model stems logically from the previous component, the process flowed in a systematic and organized manner.

For example, the program’s intended outcomes stemmed clearly from the activities that are to be implemented by the FACs, which in turn were targeted

Riki Savaya and Mark Waysman 93

at addressing specific client problems. One problem that the group agreed should be addressed by the FACs was difficulties that some clients have in re- alizing their rights for services and benefits that they are entitled to receive from government agencies. Several types of activities were planned to address this problem, including provision of information about rights and entitlements, representing clients before governmental agencies, physical accompaniment of clients to agencies, and development of advocacy campaigns to promote utilization and expansion of rights. These activities were expected to lead to an increase in utilization of benefits and rights among clients that failed to do so in the past.

The discussions were often very heated, suggesting a high degree of in- volvement and interest on the part of group members. The end result of this group process was a written program plan that differed from the original con- ception in several ways: It was more explicit, more detailed and specific, more operational, more consensual and, most of all, much tighter–all components were clearly and logically connected and integrated into a coherent program theory.

This product served a number of purposes within the organization and also between the organization and external agencies. First, the existence of a clear and consensual logic model facilitated communication, cooperation, and shared understanding among different levels of management and staff within the Welfare Department and also promoted greater clarity in role division (see Coffman, 2000). It also specified the inputs that each FAC needed and was en- titled to for implementing the planned activities and who was responsible for allocating these resources. Previously, there had been considerable confusion and disagreement regarding key elements in the program that created tension and frustration among the stakeholders, limited the scope of activities, and prevented the FACs from realizing their full potential to help their target cli- ents.

Furthermore, the existence of a clearly articulated document presenting the program model helped the Welfare Department to present the program to ex- ternal agencies, such as government ministries and foundations, in order to raise funds for running and evaluating the program.

Finally, this program was developed in a large multicultural city, and was to be implemented in a variety of different neighborhoods, each with different sociodemographic characteristics: different ethnic groups, with different reli- gious affiliations and levels of religiosity (from secular to ultra-orthodox), as well as immigrants from many different countries. As such, it was clear that any model developed might have to be adapted to suit the characteristics of the local residents. The existence of the logic model helped to make decisions

94 ADMINISTRATION IN SOCIAL WORK

about the components and elements that were core to the program versus those that could be adapted (for more details on this application, see Hacsi, 2000).

PROGRAM OPERATION STAGE: USING LOGIC MODELS

FOR DEVELOPING PERFORMANCE MONITORING SYSTEMS

Purpose

Performance monitoring (PM) has been widely promoted as an effective way to ensure a focus on program results and accountability, to determine what publicly funded programs are actually doing, and to provide for control over expenditures (Perrin, 1998). Performance monitoring systems are cur- rently being implemented, and sometimes even built into legislative require- ments by governments in developed countries throughout the world (e.g., the United States 1993 Government Performance and Results Act; see Wholey, 1997). In the private and nonprofit sectors, performance monitoring is also gaining popularity as a means for keeping programs on track and improving the quality of program implementation and results by providing ongoing peri- odic measurement of key indicators (Newcomer, 1997).

Along with this, critics have claimed that performance monitoring is often implemented inappropriately, focusing on trivial indicators, providing data that are irrelevant to stakeholders (and especially program staff) and not focus- ing sufficiently on outcomes (Bernstein, 1999; Greene, 1999; Winston, 1999). Many of these shortcomings may be prevented, when the measures are se- lected and developed after the program theory has been formulated in collabo- ration with the system’s intended users.

Context

The example that we present here stems from our work with the New Israel Fund (NIF), a philanthropic foundation that helps to fund local NGOs active in achieving social change. The NIF also offers its grantees two additional ser- vices via its capacity-building branch, SHATIL: (1) technical assistance and training in a number of areas (such as organizational development and re- source development); and (2) help in establishing and maintaining coalitions of NGOs to promote common goals. While SHATIL is an extension of the NIF, it also receives independent funding from other sources, some of whom have begun to require greater accountability and, in particular, assessment of the results of SHATIL’s work. The director of SHATIL approached us to help

Riki Savaya and Mark Waysman 95

the organization develop a PM system, with the specific requirement that the development process be in accord with SHATIL’s organizational culture, which tends to be more egalitarian and less hierarchical, thus involving a high degree of staff participation in many processes. When the decision to develop the PM system was presented to staff, with the understanding that they would be actively involved in the development process, they saw it also as an oppor- tunity to obtain a set of tools that would help them to learn from their accumu- lated experience in a systematic manner so as to improve practice.

Example

We present here the work that we did in developing a PM system for one of the SHATIL teams: consultants in organizational development (OD). In ap- proaching this task, we decided that the first step should be to develop a clear and comprehensive picture of their work. The rationale was that the PM sys- tem should be based on a set of indicators that truly reflect the core aspects of this team’s work–the program theory that stems from the practice wisdom they have developed over the years–and not necessarily the indicators that are most sa- lient or easiest to measure. We thought that the ideal way to arrive at this picture was with the aid of a logic model.

In developing the model, we decided to engage in a group dialogue with the team, not only because management had requested this, but also because we believed that in order to promote future utilization of the system, it should have broad support from within and reflect a consensus that the system is rele- vant and useful. The information to be provided should be generally perceived as bearing the potential to help the OD consultants learn from their experience and derive ideas for improving the quality of the consultation that they provide their clientele.

Although the group process with this team proved to be considerably more difficult and more prolonged than we had anticipated, it was clearly justified. It required the consultants to explicate and formulate their activi- ties, their working assumptions and the outcomes that they strive to achieve with their clients. As often occurs in other participatory processes with peo- ple that are invested in their work, here, too, the process involved a number of very heated discussions. Consensus was, however, reached eventually, leading to a clear model depicting all the components of their work. For ex- ample, it included a detailed statement of six major outcome dimensions, formulated as areas of organizational health: It was conceptualized that any NGO that functions well in all of these dimensions can be deemed to be healthy and not in need of help from SHATIL. These outcomes cover all the

96 ADMINISTRATION IN SOCIAL WORK

main target areas to which SHATIL OD consultants direct their interven- tions, such as establishing “a clear and coherent organizational identity,” “a clear and functional organizational structure,” or “an organizational culture that promotes growth and effectiveness.” Each outcome dimen- sion was further broken down into several more specific outcomes. For example, the outcome dimension “existence of a clear and coherent orga- nizational identity” comprises the following three specific outcomes: “the organization has a clear and coherent vision and mission,” “the orga- nization’s target populations are clearly defined,”