Evaluating Collaborations: Challenges and Methods

Ellen Taylor-Powell and Boyd Rossing 

NEVER DOUBT THAT A SMALL GROUP OF THOUGHTFUL, COMMITTED CITIZENS CAN CHANGE THE WORLD. INDEED, IT IS THE ONLY THING THAT EVER HAS.

Margaret Mead

Background 

This paper and manual in progress grew out of comments and requests from county agents in Wisconsin who increasingly find themselves involved in collaborative work playing unfamiliar roles in interactions with new players. They are finding that their common evaluation practices, most built on a model of individual behavioral change, don't seem to work. They have raised a myriad of questions that indicate the complexity and uniqueness of collaborative programming and the potential tensions: 

• How can I measure outcomes when it is so long term and dependent on so many factors?
• How do we establish realistic outcomes when other collaborators are involved?
• How can I highlight my contribution but also share credit?
• What can I report to the county board when they want to know what impact I'm having?
• It seems that I'm on every collaborative there is. Is a collaborative always appropriate and do I always need to be involved?
• How can I get my partners enthused about evaluation?
• Is evaluating the process of the collaborative effort good enough or do I need to measure the outcomes of our work?

To begin addressing these questions, we drew upon case examples from each of the four program areas in University of Wisconsin Cooperative Extension, conducted a series of workshops with family living agents across the state and searched the literature. We have been encouraged to find a rich and growing research base on collaboratives and evaluating community-based initiatives. This paper and the manual in process are somewhat unique in that they take an'insider' perspective - vs. external, professional evaluator perspective. The aim is to provide practical help that will assist county agents and their partners in evaluating their own collaboratives.

First, some common language. What do we mean by a collaborative? What are we talking about?

The terms collaborative, coalition, and partnership have become such common parlance that there is great variety and sometimes confusion in the way these terms are used. A common view is emerging, however, that suggests a hierarchy. As one progresses from networking to collaboration, inter-party relationships become more sophisticated, complex and potentially effective for solving multidimensional problems (Table 1). Identifying distinctions among these types may help keep expectations realistic and ensure that appropriate evaluation questions are asked. This work focuses on collaboratives, treating them as the highest form of shared resource relationship. With some adaptation, many of the ideas can be applied to other forms as well.

Table 1. Types of shared relationships

Why are we seeing so much interest in collaboratives?

The conditions in communities and the roles of local people and organizations in confronting problems and achieving desired outcomes are changing. More and more local communities are re-acquiring the ultimate responsibility for determining the quality of life of local residents. At the same time the issues facing local communities are growing ever more complicated. People also expect a broader range and quality of services. As challenges mount while resources dwindle new inter-sectoral relationships are emerging eg. public/private/citizen partnerships, community collaborations, etc. (Banovetz, J., 1994). In addition efforts to address community concerns in more comprehensive and integrated ways (in human services, environmental management, etc.) have merged as recognition mounts of the inadequacy of current fragmented agency efforts. Complex problems are calling for specialized knowledge and skills, but at the same time increasing use is being made of local people s knowledge through expanded citizen participation. Collaborative working relationships may be mandated by funders or arise spontaneously.

What are the challenges for evaluation?

Collaboratives involve multiple actors across multiple systems or sectors, are flexible and evolving, have broad and imprecise goals and seek changes in an array of domains (individuals, groups, organizations, systems, communities and policies). Conventional Extension programs, on the other hand, are easily identifiable with specific objectives that program personnel set, expected behavioral changes are identified (e.g., learning changes or practices to be adopted), the program is delivered, and before-after measures are largely used to indicate change(s) resulting from the program and thus Extension's performance. In discreet programs, developed and implemented by an agency, the agency's role is clear -- a 'package' that Extension delivers. In collaboratives, the Extension agent's role is not always clear, or changes over time, expected outcomes are often broad and imprecise. There are multiple actors and agendas. The implications for evaluation are numerous (Table 2). Qualitative evaluation methods are suggested but there are few models to apply (Kagan, 1991).

 Table 2. Nature of collaborations and implications for evaluation

What do we know about collaboratives? -- Phases of Collaborations

A growing body of evidence indicates that collaboratives develop in phases. Each phase needs to be nurtured and allowed to develop fully for the collaborative to be successful. Goodman, Wheeler and Lee (1995) discuss the consequences of ambitious agendas and inadequate timelines that preempt a collaborative from developing sufficiently in each phase. Florin, Mitchell and Stevenson (1993) stress the importance of understanding phases and the key characteristics and activities within each phase in order to determine appropriate technical assistance and training needs. In a review of 18 carefully elected studies on collaboratives, Mattesich and Monsey (1992) found 19 factors influencing the success of collaboration including those related to environment, membership characteristics, process/structure, communications, purpose and resources. While no two collaboratives are alike, research is identifying critical events/activities/processes within each phase that appear critical to success and thus, we can identify corresponding evaluation criteria (Table 3).

Table 3. Phases of collaboration with illustrative key tasks and evaluation

Not unlike other program initiatives, the development and use of a logic model - a roadmap that depicts where the collaborative wants to go and the steps along the way - seems useful. However, creating the logic model which may involve a fairly lengthy process of discussion and negotiation during the formation phase takes longer than in single agency of single person programming. A logic model depicts the way in which the collaborative is intended to work, the linkages between and among the phases (inputs to outputs to outcomes/impact) and the critical events or milestones. It then becomes possible to identify markers and measures for monitoring performance. A simplified graphic representation is shown in Fig1.

What are the evaluation questions?

 The questions being raised by our Extension faculty are as varied as the actors and stakeholders involved. A collaborative comes into being because a set of actors agree to explore and define mutual interests and then work together to achieve mutual goals. Often each of these actors represents a larger organization or constituency. While these questions are varied, they seem to cluster into four sets which can be categorized into evaluation types (Table 4):

 Table 4. Questions and evaluation type

Feasibility questions. Feasibility questions usually occur prior to getting involved. With a rising number of collaboratives and given Extension's visibility in the county as a neutral information source, links to University, and experience and skill in process and content, agents are asked over and over to be part of collaborations. When is it in the agent's (agency's) interest and when is it okay to say "no"? Some groups and/or communities are just not ready for collaborative work, some problems do not warrant a collaborative effort and sometimes extension's mission does not fit that of a collaborative. Also, research indicates that the level of organizational and/or community "readiness" to undertake collaborative work is critical to success including such factors as awareness of need for an integrated approach, resource availability, flexibility in organizational structure and communication, history of collaborative work, favorable political and social climate (Mattesich and Monsey, 1992; Casey, 1995; Blumenkrantz, 1992; Melaville and Blank, 1991). It may be important to first help establish the community's capacity for change as well as the potential of a collaborative to foster change (see Chrislip and Larson, 1994).

At any time during the life of a collaborative, feasibility issues may also be a concern. For example, when the collaborative is considering a new initiative or project, questions such as the following come into play: Do we have the resources, the commitment, the capacities to carry out the given activity/program?

These issues - whether before getting involved in a collaborative or at any time when working in a collaborative - correspond fairly well with feasibility evaluation, context evaluation, and front-end analysis.

Process questions. The term process is used here to mean the continual development/evolution of the collaboration (refer to Fig 1.). This links fairly well to Scheirer's (1994) definition of process evaluation as a generic term to document or measure a complex chain of events from start-up to impact. It encompasses formative evaluation, evaluability assessment, application of program theory, implementation analysis, and monitoring outcomes. Given the evolutionary nature of collaboratives, process issues take on great importance. Implementation is ongoing, not predetermined or known. Collaborations are flexible, changing as local circumstances change. Having empirical information on process measures, such as involvement of people and actions taken, helps bridge the long span between the formation of the collaborative and final impacts. It is also essential for ensuring the successful development of the collaborative. Collecting data on milestones, critical events, process indicators and monitoring outcomes is fundamental.

Outcome questions. In this accountability era, outcomes seem to engage most of our time and attention. Yet, there are no standards of success or expected outcomes for collaboratives. We find that outcomes may occur in a number of strata (Table 5) consistent with ecological and systems thinking (see McLeroy et al, 1988). As well as outcomes for clients or individuals (e.g., improved health, educational attainment), there may be outcomes for members of the collaborative (develop skills and new capacities). At the next level, we may be interested in outcomes for families (economic self-sufficiency, healthy relationships) or other groups such as the county office or the interdepartmental workgroup (resource sharing, integrated program delivery). Often, we are looking for outcomes in terms of services provided (new or modified services, e.g., mentoring program is added, parent aides included) and for systems whereby agencies or related organizations work together in new ways, interact differently, share resources and provide services in an integrated fashion. Ultimately, there are outcomes for communities and perhaps policy changes (e.g., environmental regulations, drinking laws).

Table 5. Collaborative outcomes

 Debate goes on whether collaboratives should be accountable for outcomes in people, services, systems or communities. Is it good enough that the collaborative made a new service available? Or is the role of the collaborative to ensure that these services are used and positively benefit the target clientele group to create improved community conditions? In many communities, the expectation is that successful collaborative initiatives will be institutionalized demanding concerted and long-term efforts. Institutionalization may be operationalized as (1) creation of indigenous competence to address continuing issues; (2) creation of organizational and community supports to sustain programs; (3) continuation of the collaborative effort after termination of funding. And in some cases, empowerment is the intended final outcome with the emphasis on building local capacities to take control of their own development.

 Answers to such questions depend upon the scope and focus of the collaborative. It may be that the collaborative only expects outcomes for its members - greater sharing of resources and cost-efficiencies (e.g., sharing resources to expand urban gardening) versus a collective outcome that bears on community well-being (e.g., reduced crime). The critical point is to fit the evaluation to the collaborative: It is inappropriate to measure community-level outcomes when the collaborative was only concerned with delivery of services; or the collaboratives implementation was aborted or changed.

Self-Interest questions. We have broken out 'self-interest' issues since it comes up so often in discussions with extension faculty. Wrapped around every collaborative is the concern of self-interest whether it be the (a) individual and (b) the individual's agency. Most agencies require evidence of personnel performance, so members of collaboratives often have to account for their personal performance, despite the fact that they are not sole players or in control.

Yet, self-interests are problematic in collaboratives since the very nature of the collaborative is not the 'individual' or 'me' but is the 'team', 'us' -- 'shared'. Some even think that focusing on the individual undermines the ability of the collaborative to function as a true team. Nevertheless, most of us will want to (need to) keep track of our individual contributions and achievements in collaborative work. And so may our partners. Successful collaboratives keep self-interests in the forefront and even renegotiate them as members and self-interests change (Winer and Ray, 1994). Adding to the complexity of this issue is the fact that the role and resource contributions of different members change over the course of the collaborative's development cycle. Members move in and out of various roles or play several roles at once. From just one Wisconsin case example, the agent's role changed in the following ways:

• Facilitator
• grant writing
• leadership in survey development and analysis for initial assessment
• public relations
• program development education

 A systems approach to evaluating collaboratives

 While we clustered the evaluation questions into 4 separate types of evaluations - linking to the conventional terminology and distinctions in the field of evaluation - we find that these labels place artificial boxes on what needs to be a fluid, integrated, holistic process. It is probable that distinct, separate studies will continue to be conducted to answer specific stakeholder concerns. However, looking at the growing research on collaboratives and taking an ecological approach to evaluation that corresponds to such a view of collaboration (see for example, Goodman et al, 1996) provides a more holistic, integrated approach. In this model, evaluation serves as a mechanism that engages and sustains participation of system units and that guides the change process. Specific questions that might deal with feasibility, process, outcomes and/or self-interests are interconnected. The emphasis is on ensuring the successful movement through all phases of the collaborative's development. This involves the triangulation of assessment (methods, sources, time, location) over the course of a collaborative's life -- not separate and distinct evaluations.

By taking the phases as the organizing framework, we can lay the evaluation questions being raised over the process of collaborative development (Table 6). Particular evaluation questions may be asked at particular phases of the collaborative's evolution but there is a strong emphasis on continual learning and feedback to ensure successful movement through the phases.

 Table 6. Systems approach to collaborative evaluation

Thus, process evaluation takes on paramount importance, running throughout the entire lifespan of the collaborative. Other questions may be answered by discreet, time referenced studies to inform particular decisions or information needs -- for example, an evaluation of the planning process, functioning of committee meetings, assessment of skill development or implementation of a new service -- but there is ongoing data collection to monitor movement through the stages.

A variety of data collection methods and tools exist which can be adapted by local collaboratives. The Forecast System as developed by Goodman and Wandersman (1994) for alcohol and drug abuse programming provides one such example. It consists of markers -- milestones or road signs -- indicating progress and measures that provide data for judging if the marker has been attained. Typical methods include analysis of meeting minutes, phone logs, staff calendars, and other internal documentation such as records of participation at meetings, activity and contact records.

A number of other tools exist - some from the published literature, some we've created and others that we've found being used by local collaboratives. For example, community/organizational readiness profile, meeting effectiveness inventory, project insight form, group/committee climate survey, member satisfaction survey, needs assessment checklist, plan quality index, action logs, periodic reflections/member reviews, diaries, logs, journalling, sociograms, goal attainment scaling, etc.

While we have emphasized the importance of evaluating process in collaborations, backing away from measuring final outcomes and impacts because of methodological difficulties, is irresponsible. We must tackle the thorny issues of measurement and judge collaborative results in a credible and responsible manner. The challenge lies on two fronts: (1) the development of indicators and measures appropriate for higher level units of analysis (i.e., organizations, systems, communities) and (2) the lengthy time horizon to final outcomes. Given the evolutionary and long-term nature of most collaboratives, documenting milestones along the way -- the process or intermediate outcomes-- remains essential. Selecting these outcomes and indicators of achievement will be done through negotiation and collaboration. No standards apply.

 Again, a variety of tools exist to help us collect information to document outcomes: 

• surveys of one form or another: member, beneficiaries,community survey, key informant survey, key leader survey
• interviews: individual and group (e.g., Long's 1,000 MILE JOURNEY); with beneficiaries; with other community residents
• case studies: policy analysis; beneficiaries; collaborative
• indicators: community level trend data
• scales (e.g., level of institutionalization scale, Goodman et al., 1993) to measure permanence of coalition initiative
• expert review: invite 'outsiders' to review the collaborative's accomplishments; perhaps, 'experts' involved with a similar collaborative in another location or invite key stakeholders to be the experts
• action research: collaborative, functions as community researchers and co-learners, to build a complete story of the collaborative's work and impacts.
• control community comparisons

Conclusions

We have made an earnest effort to rethink evaluation in the context of collaborative programming in which extension is involved. To do this, we have drawn on existing evaluation frameworks and reoriented them to fit the characteristics of collaboration. Caution is suggested in using words or labels, familiar in evaluation, that may inadvertently direct our thinking. Conceptualizing the collaborative as a fluid, evolving system helps ground the evaluation. Participatory approaches seem appropriate that integrate evaluation within the structure and process of the collaborative. Evaluation becomes part of the collaborative process itself. Triangulation of sources, methods, time and location is critical in order to weave together multiple pieces of information to understand behaviors, patterns of organization and changes in social norms. 

Process appears to be key. Funders and agencies may be focused on outcomes and results in this era of accountability and GPRA (Government Performance and Results Act), but without attention to process, we are unlikely to achieve the desired outcomes -- and the collaborative movement will end up as another 'bandwagon'. Encouraging is the fact that process and intermediate outcomes seem to satisfy funders who are increasingly aware of the complex, long-term nature of social change.

With experience in group facilitation and evaluation procedures, Extension personnel are critically placed to help collaboratives evaluate themselves, thus, furthering the local initiatives movement. We must, however, be willing to feel discomfort and not fall back into familiar ways. It is also incumbent that we link with the larger research community and work collaboratively as evaluators.

References

Banovetz, J. 1994. The challenges of community government. IN J. Banovetz, D. Dolan, J. Swain (eds), Managing Small Cites and Counties. Washington, D.C.: International City/Council Management Association.

Bergstrom, A., et al. Collaboration Framework...addressing community capacity. National Network for Collaboration.

Blumenkrantz, D.G., 1992. Fulfilling the promise of children's services. San Francisco: Jossey-Bass.

Butterfoss, F.D., R.M. Goodman, A. Wandersman, R. Valois, M.

Chinman. 1996. The Plan Quality Index. In. Fetterman et al., (Eds). Empowerment Evaluation: Knowledge and Tools for Self-Assessment & Accountability. Thousands Oaks: Sage.

Casey, Annie E. Foundation. 1995. The Path of most resistance. Baltimore, Md.

Chrislip, D. and C. Larson. 1994. Collaborative Leadership. San Francisco: Jossey-Bass.

Clark, Richard W. 1992. Building Coalitions. Columbus, OH: The Ohio State University. 

Connell, James P., Anne C. Kubisch, Lisbeth B. Schorr and Carol H. Weiss. 1995. New Approaches to Evaluating Community Initiatives. Washington, D.C.: The Aspen Institute.

Florin, P., R. Mitchell, J. Stevenson. 1993. Identifying training and technical assistance needs in community coalitions: a developmental approach. Health Education Research. 8:3, 417-432.

Goodman, R.M., A.Wandersman, M. Chinman, P. Imm, E. Morrissey. 1996. An ecological assessment of community-based interventions for prevention and health promotion: Approaches to measuring community coalitions. American Journal of Community Psychology. 24:1, 33-61.

Goodman, R.m., and A. Wandersman. 1994. FORECAST: A formative approach to evaluating community coalitions and community-based initiatives. Journal of Community Psychology, Monograph Series CSAP Special Issue. Brandon, Vermont: Clinical Psychology Publishing Co., Inc.

Goodman, R.M., McLeroy, K.R., Steckler, A., & Hoyle, R.H. 1993. Development of level of institutionalization (LoIn) scales for health promotion programs. Health Education Quarterly, 20, 161-178.

Gray, Barbara. 1989. Collaborating: Finding common ground for multiparty problems. San Francisco: Jossey-Bass. 

Mattesich, Paul W. and Barbara Monsey. 1992. Collaboration: What Makes It Work? St. Paul, MN: Amherst H. Wilder Foundation.

McLeroy, K.R., D. Bibeau, A. Steckler, K. Glanz. 1988. An ecological perspective on health promotion programs. Health Education Quarterly. 15:4, 351-377.

Melaville, A.I., M.J. Blank. 1991. What it takes. Washington, D.C.: Education and Human Services Consortium.  

Miller, Lorna Clancy, Boyd E. Rossing and Sara M. Steele, Partnerships: Shared Leadership Among Stakeholders. 1990. Madison, WI: University of Wisconsin.

Scheirer, M.A., 1994. Designing and using process evaluation. In J. Wholey et al (eds), .Handbook of Practical Program Evaluation. San Francisco: Jossey-Bass.

Winer, Michael and Karen Ray. 1994. Collaboration Handbook: Creating, Sustaining, and Enjoying the Journey. St. Paul, MN: Amherst H. Wilder Foundation.

Young, Nancy, Sid Gardner, Soraya Coley, Lisbeth Schorr and Charles Bruner. 1994. Making a Difference: Moving to Outcome-Based Accountability for Comprehensive Service Reforms. Falls Church, VA: National Center For Service Integration.

Return to AEA EEE-TIG Front Page

Send comments to ANR at David.Underwood@ucop.edu