10/26/99

Marilyn Grantham, Accountability and Reporting Specialist

Some writers and thinkers in the evaluation profession are calling this era "the age of accountability" because of the growing demand from public and nonprofit funding sources for a strict accounting of how funds are spent and what is accomplished with them. Because of their unique collaborative funding base, cooperative Extension organizations, in particular, must respond to a number of funders—federal, state, and county governments, as well as grantors and contractors. This situation raises several issues. For one, the demand for accountability is coming at a time when organizational resources in general are stable at best and no additional funds are being provided specifically for accountability. So responding to accountability demands and preparing and reporting data, including the outcomes and impacts of educational programs, takes organizational resources that otherwise would be invested elsewhere, such as in program development and evaluation. Secondly, accountability enhances the need for program evaluation in order to provide reliable and valid information about outcomes and impacts. The end result is a growing need for more program evaluations at a time when staff time and additional money are becoming scarcer.

Some background on this study:

In late 1996, Robert Shumer, a faculty member in the College of Education and Human Development, and Richard Krueger, Program Evaluation Leader, approached the University of Minnesota Extension Service’s dean and director with a proposal to conduct a series of "hands on" non-credit staff development workshops on outcome evaluation. Some funds were designated and the author of this paper volunteered to assist Professors Shumer and Krueger with the workshop series.

The target audience was faculty and academic staff who had received competitive Extension internal grants during 1995 and 1996—the first two years that this approach for providing "start-up" funding for new and innovative programs had been used. This audience was chosen because one of the specifications associated with receiving an internal grant was that the recipient(s) would evaluate their work and report the results to the members of the review committee, Extension administrators, and their colleagues.

Members of the target audience were sent personalized letters of invitation from the dean and director, inviting them to participate in the outcome evaluation workshop series. Grantees were told in the invitation letter that the purpose of the workshop series was to help them design and conduct an evaluation of their program.

Each workshop in the series of four was planned for five hours per day on campus and spaced about two months apart so that participants would have time to put into effect what they’d learned before coming back for the next phase. Training materials and lunches were provided and travel expenses for staff off campus were covered from the grant funds.

About 30-32 internal grant recipients indicated an interest in the workshop series. About 24-25 of them came to the initial workshop. When the second workshop was held about two months later, the number of participants dwindled to about two-thirds of the initial group. The number of participants dropped substantially again for the third workshop. About a third of the initial number of participants were in three or four of the workshops. However, those participants who persisted indicated that they were highly pleased with the learning process and the outcomes in terms of designing and implementing evaluation plans in an oral evaluation at the end of the final workshop.

Nevertheless, this experience raised a number of issues in the author’s mind about Extension faculty and staff attitudes toward and perceptions of program evaluation, as well as how to successfully design and deliver staff development that would provide the necessary skills and competencies to do program evaluations. It was this situation that "set the stage" for conducting a series of five focus groups with Extension faculty and staff in order to explore their attitudes and perceptions about program evaluation.

Research methodology:

The names of faculty and staff members invited to participate in the focus groups were taken from the original list of faculty and staff who had received Extension internal grants during 1995 and 1996. The invitees were those persons on the list who had not responded to the invitation to participate in the outcome evaluation workshop series. Subtracting faculty and staff members who had come to one or more of the workshops resulted in 55 persons who were contacted about participating in a focus group. A total of 35 faculty and staff members participated in one of five focus groups. Twenty-two of them were Extension educators who participated in one of three groups held via teleconferences. The remaining 13 persons participated in one of two face-to-face focus groups held on campus. Six of these persons are faculty members in academic departments; the remaining seven persons are academic professional staff on campus.

The purpose of the focus groups was to explore Extension faculty and staff attitudes toward program evaluation and their perceptions of their individual responsibility for evaluating programs, the benefits and the barriers in evaluating programs, and the support and recognition provided by Extension for this kind of activity. The questions asked in all five focus groups evolved from themes that emerged out of the evaluative discussion held in the final Outcome Evaluation Workshop in October 1997. This discussion had been videotaped and the tape observed for themes, then an edited version was produced as the evaluation report for the workshop series.

The questions asked participants in all five focus groups were

1. When you hear the words "program evaluation," what immediately comes to your mind?
2. When you are asked to evaluate a program or a project, what do you think that means? What are you expected to do?
3. What are the benefits to evaluating Extension programs?
4. What are the barriers to evaluating Extension programs?
5. What have you done about evaluating the program for which you received an Extension internal grant?
6. What would have helped you with this effort or any other program evaluation?
7. How should Extension recognize and reward evaluation work?

Issues emerging from the answers to these questions:

Evaluation issues:

• Difficulty measuring outcomes and impacts, particularly in the short-term when Extension programs often won’t show such results for many years.
• Difficulty in separating Extension’s role in achieving outcomes from other influences on program participants’ attitudes or behaviors.
• What is "success" in terms of program outcomes?
• Assessing "unintended consequences" of programs.
• Lack of knowledge/connection with evaluation studies done elsewhere—models that might be adapted for use in Minnesota.
• Tendency to focus on numbers and other "input" information that is more easily gathered, rather than on the "so what" question.
• Need for new approaches to program evaluation that are "culturally" appropriate to new audiences, e.g., that will work when language and other cultural barriers are involved, actively engage program participants and other stakeholders.
• Credibility of internal evaluation work versus hiring an external evaluator.
• Evaluating collaborative work—who does it, who gets the credit?

Personal professional issues:

• Lack of evaluation knowledge and skills, especially of simple, easy to use, evaluation techniques.
• Lack of time to do evaluations, especially when "crisis" programming occurs.
• Dealing with the political realities of responding to clients with "clout," even when the request is trivial.
• Fear about the consequences of identifying lack of success in a program—both to Extension administrators and outside funders and decision-makers.
• Evaluating collaborative efforts—who gets the credit? Is it sometimes politically risky to claim credit when collaborating with external partners?
• The volume of work done by campus-based faculty that isn’t part of an Extension "program" per se, e.g., consulting, advising industry people, etc.
• Lack of recognition of Extension work, including evaluation, in campus-based faculty promotion and tenure decisions.
• Skepticism about whether evaluations really matter—is the information used and useful for anyone other than the staff involved in the program?
• Is there value/payback for doing evaluations?
• Preference for having mentors and/or internal or external consultants provide technical assistance and aid in designing and conducting program evaluations, rather than more formal, "traditional" kinds of general staff development on program evaluation.
• Possibility of creating "affinity" groups—people doing similar programs who would share ideas and techniques for evaluating their programs. Could use listservs and chat rooms instead of meeting face-to-face.

Organizational issues:

• Lack of commitment on the part of Extension administration for genuinely supporting program evaluation—in terms of staff development and funding for evaluations.
• Lack of guidelines--unclear expectations of staff and failure by Extension administation to articulate the purpose and need for program evaluation, e.g., which programs should be evaluated? How will the information gathered be used in meaningful ways to secure funding, etc.?
• Lack of clarity on the purpose(s) of evaluation—how much of it is for program improvement, how much for satisfying funders and decision-makers (accountability)? How much is for "public relations" and/or program marketing purposes? How much is for individual and organizational learning about what works and what doesn’t?
• Lack of recognition, use of evaluation results in meaningful ways; lack of consequences for not evaluating programs.
• Lack of centralized program planning, program priority and goal-setting.
• Lack of clarity on what is a "program."
• Lack of mechanisms for sharing evaluation results internally, learning from each other’s work, sharing ideas.
• The "disconnect" between campus and field—lack of thorough knowledge of what programming is happening—and should be evaluated comprehensively.
• Lack of assistance in designing and conducting evaluations—inability to hire external evaluation expertise due to lack of funding.

Summary and recommendations:

The preceding issues indicate a definite need to re-think the approach to program evaluation in the University of Minnesota Extension Service. While a number of the focus group participants stated that they perceived program evaluation to be an important part of their educator role, there was a general concern that organizational commitment, especially in terms of funding, clarity of purpose and use of evaluation results, was lacking. As a result, many participants thought that at present there are neither rewards or consequences related to evaluation, beyond their own satisfaction in learning how they might improve their programs or success in gaining additional grant funds for program support. The perceptions and attitudes of campus-based faculty were generally even less favorable than those of Extension educators.

Extension administration should address these issues by

• Clarifying organization commitment to program evaluation and making a greater effort to assist in providing technical assistance and funding.
• Clarifying the purposes and expectations of faculty and staff for doing program evaluation and providing guidelines for determining which programs should be evaluated.
• Revitalizing a program planning process that emphasizes incorporating evaluation up front.
• Prioritizing major programs and evaluation efforts.
• Recognizing and sharing evaluation efforts across the organization.
• Providing more opportunities for faculty and staff to discuss their work and share ideas for improving it, including efficient, effective approaches to evaluation that are appropriate to the work.
• Emphasizing providing more specific technical assistance and other resources to faculty and staff when they are designing programs and writing grant proposals, than formal staff development.
• Identifying faculty and staff who are doing exemplary evaluation work and rewarding them by providing assistance with their work so that they have time to mentor others who are interested in learning from them.
• Identifying evaluation research and models from other states and organizations other than Extension that might be readily adapted for use in Minnesota.