ADDRESSING THE ATTRIBUTION QUESTION IN EXTENSION

Satish Verma

Specialist (Program and Staff Development)

Professor of Extension Education

Louisiana Cooperative Extension Service

LSU Agricultural Center

Baton Rouge, LA 70894-5100

225/388-6194

225/388-2478 (fax)

sverma@agctr.lsu.edu

Michael Burnett, Professor

School of Vocational Education

Louisiana State University

Baton Rouge, Louisiana

225/388-5748

225/388-5755 (fax)

vocbur@lsu.edu

Abstract

Program developers and evaluators need to address the important program accountability question of attribution of outcomes. This discussion is a beginning. Starting with some basics – the meaning of program, approaches to program theory development, and the nature of attribution – three types of attribution are suggested. An agricultural research verification and extension program provides background for the recommendation to use causation attribution in a controlled program environment, and associative attribution in a confounded program environment. Implications for evaluation design and methodology are discussed.

Paper presented at the Annual Conference of the American Evaluation Association

Orlando, Florida, November 3-6, 1999

ADDRESSING THE ATTRIBUTION QUESTION IN EXTENSION

Extension systems in the public sector face increasing accountability demands to justify allocation of funds and to demonstrate that effective, need-based programs are in place. External stakeholders want to know what difference extension education programs make in the lives of people for whom they are intended. In the United States, for example, the federal extension system and a number of state extension systems prepare performance-based budgets and report progress/outcomes of their programs against predetermined goals. The expectation is that these programs will bring about behavior changes in individuals, socio-economic benefits for families, desirable environmental consequences for communities, and justifiable returns on tax-supported investments.

An important question for program developers and evaluators is whether the observed outcome of a program is attributable to it. Finding an answer to this question is neither easy nor straightforward because (a) non-program variables in the program environment influence outcomes, (b) identifying and measuring all relevant program variables may not be feasible or possible, and (c) practical program and study considerations mitigate outcomes. Exogenous variables such as non-extension information sources, the weather, the agricultural/community infrastructure, and agricultural/consumer market forces, can have beneficial or adverse effects on program outcomes. Program variables such as delivery methods, contextual factors, and audience (consumer) characteristics also have a bearing on outcomes. While certain program variables could be manipulated in the design and implementation of the program, it may not be possible to do so for other variables. Furthermore, resource constraints, program design requirements and difficulties, and audience considerations are practical implementation and study barriers for program developers and evaluators. Answering the attribution question is complex and challenging, yet interesting, important, and necessary.

The purpose of this paper is to present a rationale and implications of the attribution question, and discuss a program evaluation strategy.

Program, Program Theory, and Attribution

Program

Wholey (1987) defines “program” as a set of resources and activities directed toward one or more common goals. An administrative interpretation of the U.S. Government Performance and Results Act (GPRA, 1993) for the U.S. Cooperative Extension System defines an “extension program” as a series of learning experiences designed to bring about desired behavior changes in target clientele (Cooperative State Research, Education and Extension Service, 1997). Boyle (1981, p. 14) maintains that “a (major) program is not an isolated workshop, event, or activity…not a variety of different educational offerings available cafeteria style...not individualized response to the continuous and urgent requests from individuals for information.” These notions suggest that a program should be coherent, organized, and focused on accomplishing stated goals.

Approaches to Program Theory Development in Evaluation

Chen (1990) posits that most writers in the field of program theory conceptualize it as describing the program and how it works. For example, Bickman (1987, p.5): “the construction of a plausible and sensible model of how a program is supposed to work.”; Lipsey (1987, p. 7): “a set of propositions regarding what goes on in the black box during the transformation of input into output; that is, how, via treatment inputs, a bad situation is transformed into a better one.”; Wholey (1987, p. 78): “program resources, activities, and intended program outcomes, and…a chain of causal assumptions linking program resources, activities, intermediate outcomes, and ultimate goals.” Chen suggests that there is a prescriptive aspect of program theory, namely what the program should accomplish. The prescriptive aspect of program theory construction is based on the values of the program developer in that it prescribes what actions should be taken, how they should be organized (treatment and implementation), and what the outcome criteria should be.

Patton (1997) conceptualizes three major approaches to theory development. The deductive approach uses experimental design methodologies to construct models of the causative relationship between program treatments and outcomes. In the inductive approach, the evaluator does field work to generate theory, for example, as part of an early evaluability assessment process, or in conjunction with a literature review. Patton states, “The product of the inductive approach is an empirically derived theoretical model of the relationship between program activities and outcomes framed in terms of important contextual factors.” (p. 221). The user-focused approach involves working with intended users to extract and specify their implicit theory of action about a program. The full chain of events that links inputs to activities, activities to immediate outputs, immediate outputs to intermediate outcomes, and intermediate outcomes to ultimate goals constitutes a program's theory of action. In contrast to deductive and inductive approaches which view programs as replicas of larger phenomena, theories of action are specific to a particular program.

Bennett (1979) conceptualizes a relationship between the “chain of events” in a program and the “levels of evidence” needed for evaluation of extension programs . This chain of events enables one to construct the program's theory of action:

a. Inputs (resources) must be assembled to get the program started.

b. Activities are undertaken with available resources.

c. Program participants engage in program activities.

d. Participants react to what they experience.

e. As a result of what they experience, changes in knowledge, skills, attitudes, and aspirations occur (if the program is effective).

f. Behavior and practice changes follow knowledge and attitude changes.

g. Ultimate impacts result, both intended and unintended.

Attribution

Is attribution the right question for extension? We start with this rhetorical question because if we define attribution solely as causality the question defies a definitive answer. How can we in Extension, working in a practical, real-life setting with people, families and communities, satisfy the strict criterion of causation, metaphorically defined by Scriven as the relation between mosquitoes and mosquito bites (Scriven, 1991)? It stands to reason that the notion of attribution needs to be broadened beyond strict causality.

In a sense, causality is implicit in a program's theory of action since means are linked to ends in an hierarchical relationship. The linkages suggest a logical chain or progression in the program, in that each program event is the outcome of the successful attainment of the preceding event and, in turn, is a precondition to attainment of the next higher event. In practice, it would appear that the meaning of attribution should be moderated to include on an attribution continuum, causality at one end, inference at the other, and association in between.

Relating this continuum to a program's theory of action is instructive. Obviously, cause-effect linkages are transparent in the series of links between events. However, the apparent temporal logic of immediate to intermediate to ultimate outcomes cannot be rigid in a real life application of the program. In practice, the components, links and stages of a program are highly interdependent and dynamically interrelated (Patton, 1997). For example, a positive attitude toward learning helps students learn better, but more knowledge can also change negative attitudes. This suggests associative attribution in that there is an association or relationship but one is not certain about the direction of the relationship. By inferential attribution is meant a value-based, judgmental perception of the program developer/evaluator describing what happened in a program, then inferring reasons and providing supporting explanation for the program's role in the outcome.

Determining Attribution in Extension: Design

and Methodological Considerations

Can experimental and quasi-experimental designs be used in evaluating extension work to establish causality? To a limited extent, this is possible by (a) using trend data to examine the history of, for example, a community's performance before an initiative has begun and after it has been implemented, and (b) using contrived situations in which one or more variables of interest are controlled in conventional experimental designs such as time series and comparison groups. Some obvious reasons why extension programs targeted to individuals, families or communities do not lend themselves easily to experimental design are the smallness of change, time duration required to achieve impact, number of cases available to study, high cost, and the ethics of withholding education from a specific population.

It would appear that associative attribution and inferential attribution are more suited to extension program evaluation. General evaluation techniques in these cases include:

a. Combining multiple sources of evidence, including primary and secondary data sources, and groups that influence program performance and outcome so as to achieve a confluence of results.

b. Using multiple tools such as qualitative and quantitative methods, and suitable analytical techniques to strengthen attribution.

c. Using the theory of action approach to show how a program's logic falls into place and whether the program was “on track” with this logic.

d. Systematically examining factors other than the program treatment that might influence outcomes.

e. Documenting early outcomes that lead to longer-term results to show how means leads to desired ends. This also helps to improve the program as it is being conducted.

Elucidating the Attribution Question: An Example

An example of an ongoing extension program of a state extension service in the United States illustrates some considerations in addressing the attribution question. In general, the following steps are visualized in planning the evaluation of a program focused on this question.

a. Involving relevant stakeholders in developing the program's theory of action because they are the intended users of results. Bringing them in at this stage (a) gets legitimation and some guarantee that evaluation findings will be used, (b) clarifies the attribution question, i.e., causal, associative or inferential, (c) provides stakeholder expectations in this regard, and (d) guides the program developer/evaluator to frame one or more evaluation questions, and consider design and methodology.

b. Deciding level(s) of impact to evaluate and the type(s) of attribution to be determined. This decision will suggest evaluation design and methodology – instrumentation, data collection, statistical techniques.

Example: Soybean Research Verification and Extension Program

Background. Soybean is a major commodity in the state, contributing $252 million to the annual economy. In 1996, there were 5,697 producers, who produced 35.9 million bushels on 1 million acres. The state average yield is about 26 bushels/acre.

Commodity research and extension programs over the years have shown that yield is increased when farmers adopt approved practices, tested in research settings and under field conditions, and disseminated through an information system. An intensive, holistic approach was introduced in the soybean extension program in 1994. A set of cultural practices – improved varieties, tillage (seedbed preparation), planting, fertilizing, pest management, irrigation – recommended by soybean researchers was verified under field conditions. Farmers who agreed to stay in the program for two growing seasons and follow the recommended technology and management system were chosen by county agents as program cooperators. Under close technical supervision of county agents and subject-matter specialists, cooperating farmers learned how to manage their production operation in a systematic, attentive manner by following and adopting the set of cultural practices recommended for their individual operations. Over a four-year period, data on yield, cost, and the relative influence on yield and cost of each practice were gathered and analyzed on 38 verification fields (one field per farmer per county). Average soybean yield from these fields was recorded at 11 bushels per acre more than the state average, and the combined fixed and variable production cost per bushel was $3.78, well below the current price of $6.40 per bushel.

County agents and specialists expected cooperating farmers to extend the technology and financial management system from the small verification fields to their whole farm. Field days and producer meetings using cooperating farmers as spokespersons to share the results they achieved were organized in the counties as a means of encouraging other farmers.

The Program's Theory of Action and Matching Levels of Evidence. How can we evaluate the outcome of this program to answer the attribution question? We need first to construct the program's theory of action. Figure 1 shows the theory of action and matching levels of evidence for the program.

The program theorizes that resources such as faculty time, technology/management systems guidance, and education materials are provided by the state extension service through weekly visits to cooperators during the crop's growing season, and organized production meetings and field days using cooperator fields as demonstrations for other farmers. Cooperators on their part provide production inputs, and incorporate technical advice of extension faculty into their operations. They develop a positive regard for these interactions, learn the value of attention to detail and timely management of the production system, as well as any new technology, and share the results of their experience with other farmers in field days, production meetings, and informal contacts. Cooperators practice the recommended technology system on a small scale in field verification plots and eventually extend the system to the entire farm operation. As a result, soybean yields and, potentially, incomes are increased on the farms of cooperators. Other farmers begin to evaluate the technology system on a small scale and also experience increases in yield and income.

Evidence for levels 1-4 (inputs, activities, farmer participation, and reactions) can be gathered from program records and attributed with a high degree of certainty to the extension program because there is a close association among extension faculty, program activities, and cooperators. Levels 5-7 indicate the outcomes of the program. In general, this evidence is more difficult to plan for and gather, and requires greater resources. The task becomes more complex, time-consuming, and difficult when we want to attribute observed effects or outcomes to the program and its activities. Furthermore, program effects that are not planned for or seen by the evaluator, may not have had sufficient time to appear, or may be hard to measure, complicate the task. This suggests that the total outcome of a program is likely to be underestimated.

It is generally agreed that Extension is a major player in the information dissemination system since it is a prime source from which research information in the public sector originates. But it is not the only source. Increasingly, the Natural Resources Conservation Service is a major source with advice on irrigation, soil conservation, etc. Also, Experiment Stations and the Agricultural Research Service are public sector sources. Other players – agricultural consultants, dealers, the print and broadcast media, and friends and neighbors – have a significant role in providing information to farmers. Adoption-diffusion research in agriculture has shown that a variety of personal, social, economic, and technological factors are related to knowledge, skills and attitude changes, and adoption of technological innovations. The inference that economic and social benefits occur as a result of behavior and practice changes envisioned in a program's theory of action is confounded by environmental factors. In this kind of dynamic environment, therefore, program evaluation has to be focused on a key question, and the evaluation design has to be flexible and include variables which have an influence on or help provide answers to the question.

In our example, the evaluation question could be focused on (a) determining the effects on farmers cooperating in the research verification program, or (b) other farmers participating in the broader extension program. In either case, one could determine impact at Level 5, 6 or 7.

In the first question, since cooperating farmers constitute a homogeneous group with whom extension faculty have worked intensively over two growing seasons, Level 7 effect can be examined. Records of cultural practices recommended by extension faculty and followed by farmers are available. Yield and cost data are also available. Income received by each farmer, and return to cost can be calculated for each field plot. Additional income can be determined by extrapolating field plot results to the total operation. It would appear that the economic benefit to cooperating farmers is totally, or partially, attributable to the research verification and extension program. The basis for this assertion is that the program is highly structured through the chain of events identified in the program's theory of action. Verifying linkages between successive events will strengthen this assertion, and test the theory.

The focus of the second question is farmers who have not participated in the research verification program but are involved in the extension phase. They are a broader and more challenging group for studying attribution. Their contact with extension faculty is limited to field days and production meetings, informal contacts with cooperators, and routine extension education activities and materials. Learning and adopting recommended practices to ultimately impact production and income will depend on a number of factors in their personal situations, the farming and education context, and successful experiences. Evaluating outcomes in this typical extension scenario will require that program and non-program (exogenous) variables be identified, and appropriate statistical techniques applied. In this situation, the evaluator would probably focus on determining associative or inferential attribution.

Summary and Conclusions

We began this discussion with the question – “How do we determine attribution of outcome to a program?” Placing the question in the context of a program evaluation strategy, we stated that it was necessary for program developers and evaluators to:

a) understand that a program is a coherent, organized set of resources and activities focused on one or more common goals and not isolated, disconnected events;

b) construct prior to implementation of a program, the program’s theory of action specifying the “chain of program events” and the “kinds of evidence” appropriate for each event in the chain;

c) decide which program event or events to evaluate, the type(s) of attribution to be determined, and the appropriate evaluation design and methodology to use.

Discussions of the question in the above light, and the example of an agricultural research and extension program which was given to illustrate this approach, lead to several propositions:

a. Attributing outcome to a program with a high level of certainty is confounded by non-program influences and those program variables which cannot be or are not considered by the evaluation. As such, causation attribution in extension programming is likely to be rare and restricted to programs and program environments that are homogeneous and tightly controlled, with one or a small number of treatment variables.

b. A majority of extension programs lend themselves to two other types of attribution – associative and inferential. In these cases, the evaluator cannot be certain that a particular program results in some definite outcome, but the connection between program and outcome could be described using suitable techniques such as multiple sources of evidence, multiple tools, analytical techniques, and anecdotal material.

c. A caveat that undergirds program evaluation efforts and the attribution issue should be pointed out. Ostensibly, program effects that are not planned for or discerned by the evaluator, that may not have sufficient time to become transparent, or that may be hard, even impossible, to measure complicates the task of evaluation. It would appear, therefore, that the total effect of a program is likely to be invariably underestimated.

These propositions and the accompanying discussion are offered to program developers and evaluators as food for thought and empirical research as we try to ourselves understand and promote understanding of this issue among educators and other stakeholders in the extension system.

References

Bennett, C. F. (1979). Analyzing impacts of extension programs. Washington, D.C.: United States Department of Agriculture

Bickman, L. (1987). The functions of program theory. In L. Bickman (Ed.) Using program theory in evaluation (pp 5-18). New Directions for Program Evaluation (33), American Evaluation Association, San Francisco: Josey Bass Inc.

Boyle, P. (1981). Planning better programs. New York, NY: McGraw-Hill.

Chen, H. (1990). Issues in constructing program theory. In L. Bickman (Ed.) Advances in Program Theory (pp 7-18). New Directions for Program Evaluation (47), American Evaluation Association, San Francisco: Josey Bass.

Cooperative State Research, Education and Extension Service. (1997). Government performance and results act. Washington, D.C.: United States Department of Agriculture.

Lipsey, M. (1987). Theory as method: Small theories of treatments. Paper presented at the National Center for Health Services Research Conference: Strengthening Causal Interpretations of Non-Experimental Data, Tucson, Arizona.

Patton, M. Q. (1997). Utilization-focused evaluation. Thousand Oaks, CA: Sage.

Scriven, M. (1991). Evaluation thesaurus. Newbury Park, CA: Sage.

Wholey, J. S. (1979). Evaluation: Promise and performance. Washington, D.C.: The Urban Institute.