Partnership Implementation in the MSP Program

Jennifer Scherer, Ph.D.
COSMOS Corporation

This research is part of the Math and Science Partnership Program Evaluation (MSP-PE), supported by Contract No. 0456995 from the National Science Foundation. The MSP-PE is led by COSMOS Corporation, with Robert K. Yin of COSMOS serving as Principal Investigator (PI) and Jennifer Scherer serving as one of three Co-Principal Investigators. Additional Co-Principal Investigators and their collaborating institutions (including discipline departments and math centers) are Patricia Moyer-Packenham of George Mason University and Kenneth Wong of Brown University. Other collaborating institutions include Vanderbilt University and The McKenzie Group. The latter organization is coordinating advisory board activities, assuring the autonomy and integrity of the external peer review work. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.

The present draft is based on materials, information, and data that were available to the authors as of December 2004.

1. WHY PARTNERSHIPS?

1.1 INSTITUTIONAL PARTNERSHIPS IN MATH AND SCIENCE EDUCATION

« back to contents

The Math and Science Partnership (MSP) Program at the National Science Foundation (NSF) promotes the development, implementation, and sustainability of exemplary partnerships to advance high-quality math and science education. The MSP Program anticipates that the partnerships will be instrumental in improving K-12 student achievement, as well as reducing achievement gaps among diverse student populations differentiated by race/ethnicity, socioeconomic status, gender, or disability, a strategy advocated by Haycock et al. (1992). The importance of being partnership driven with science, technology, engineering, and math (STEM) faculty engagement is apparent not only from the name of the program, but also in the National Science Foundation's (NSF's) decision to include it as one of the five "key features" of the program.1

A required partnership in the MSP Program is between an institution of higher education (IHE) or eligible nonprofit organization (or consortium of such institutions or organizations) and one or more local education agencies (LEAs) that may also include a state educational agency or one or more businesses.2, 3 The MSP Program also distinguishes between core and non-core partners. Core partners share responsibility and accountability for the MSP grant. All core partner organizations are required to provide evidence of their commitment to undergo the coordinated institutional change necessary to sustain the partnership effort beyond the funding period. A non-core or supporting partner is not required to commit to the institutional change necessary to sustain grant activities beyond the funding period, but is an important stakeholder in K-12 math and science education.

At the same time, successful partnership building requires a significant amount of time, money, and human effort—all of which may be considered precious resources. Why partnerships are needed to improve math and science education is therefore a question worth asking. One response is based on the belief that coordinated and aligned actions and policies are needed to make such improvements in student achievement—starting with widespread agreement over the goals for student learning, based on rigorous content and performance standards for K-12 students and teachers, who in turn receive their training from IHEs (Raizen et al., 1997). Even more basic, Goodlad (1990) measured the magnitude of the problem of achieving meaningful reform and purported the value of systemic approaches, an approach that NSF, among others, has operationalized through partnerships programs, such as MSP.

Because of the complexity of trying to advance K-12 math and science education, which is influenced by many different institutions, there is a need for coordination and alignment ultimately among the institutions (not just the formal K-12 system and most certainly not just what takes place in a K-12 classroom):

College admissions criteria that serve as a highly motivating force for precollegiate schooling (Callan, 1998);
Teacher preparation and professional development offerings by IHEs that affect the quality and quantity of a student's teachers; and
A host of policies implemented by state departments of education regarding student promotion, course requirements, assessments, and curriculum, as well as teacher certification rules (Teitel, 1993).

Partnerships are needed to create coordination and alignment across these institutions, as well as within K-12 systems that traditionally have been "loosely-coupled" (Weick, 1976). Partnerships also can provide continuity of focus, align curricula and assessments, create desired normative climates, and instill accountability (Elmore, 2000). Also, the Annenberg Foundation's "Challenge" gifts, which began in 1993, have helped build strong coalitions among businesses, foundations, universities, and grassroots community groups to muster greater public will and support for public school reform (The Annenberg Foundation, 2002). Previous research suggests that collaborations between IHEs and K-12 systems, far from taking place within a congenial framework, may even evoke the clashing of two cultures (Committee on SMTP, 2001; Conf. Bd. of the Math. Sci., 2001; Goodlad, 1993; and Goodlad & Sirotnik, 1988). The MSPs may have led to insights into the nature of such clashes, if any, and how to overcome them. Some of the participating IHEs might even have grappled with the historic role of schools of education (Clifford and Guthrie, 1988; Tierney, 2001; Timpane and White, 1998), and the evolving role of "professional development schools" (Clark, 1999; Committee on SMTP, 2001; Holmes Group, 1990; Pritchard and Ancess, 1999; Rice, 2002).

For math and science education, the partnerships also may be functionally critical due to the dynamic nature of science, marked by the central notion of "scientific progress." Because such progress generates new information, needs, and questions, K-12 partnering with IHEs is essential for the transference of state-of-the-art knowledge and the development of enhanced or refined teaching practices to K-12 teachers and administrators. Scientific knowledge and progress also can create new demands for math knowledge (e.g., recent emergence of computational biology; earlier emergence of computer science) and requires the partnerships to be responsive to these needs. For any given MSP, the complexity of the structure and functionality of the partnership can lead to variable implementation start-up and progress. Thus, the progress of implementation is likely to vary across the partnerships.

1.2 SUBSTUDY OBJECTIVES

« back to contents

The substudy is one of several under the Math and Science Partnership-Program Evaluation (MSP-PE). Like the other substudies, the partnership substudy will tentatively occur in stages (MSP-PE, Yin, Wong, Moyer-Packenham, and Scherer, 2005). The primary program-level questions to be addressed will include:

What kinds of partnerships work under what kind of situations?
1. What is their size?
2. What is their leadership?
3. What is the tolerable geographical relationship or distance?
4. What are the benefits/costs for regional partners?
5. What are the tolerable cultural differences?
What kinds of partnerships were enacted and how have they changed?
What difference does it make to be a core versus a non-core partner?
Are there better start-up strategies (planning meetings, agreement on vision/mission, etc.)?

Some of the project-levelquestions that will be addressed in the first stage of the evaluation include:

To what degree was there a pre-existing relationship among the partners?
During the start-up phase of the partnership, what kinds of partnerships existed at each MSP?
1. How many partners were involved with the partnership and who were they?
2. Geographically speaking, where were the partnerships located?
What has occurred in terms of operational set-up?
1. What types of formal and informal relationships did the partnerships have?
What activities are taking place at each MSP?
Is the MSP evaluating its partnership?
1. What instruments are being used to measure the partnership?
2. How is it using the results of the partnership evaluation?
3. How many MSPs are evaluating their partnerships?
Are there any contextual conditions or rival explanations that help to explain the partnership?

Data also are currently being collected through a management information system (MIS). In the subsequent years of the substudy, MSP-MIS data and MSPs' partnership data will be collected and analyzed to confirm or expand through information collected during the initial stages of the original data collection effort. For example, using data collected through the MIS, specific partnership activities will be described in conjunction with the type of partnership. The MIS further contains several open-ended type of questions that can be analyzed such as new practices or policies and lessons learned. These survey items will be analyzed in a subsequent phase of this study. Appendix A lists the type of queries related to partnership that are captured through the MSP-MIS.

The substudy will be coordinated with a related Research, Evaluation, and Technical Assistance award (RETA) conducted by Gordon Kingsley, Ph.D. (Principal Investigator of Alternative Approaches to Evaluating STEM Education Partnerships: A Review of Evaluation Methods of an Interorganizational Model, NSF-MSP Award number 0231904), to minimize duplication and to maximize mutually beneficial data collection efforts. Kingsley's research is described in more detail on page 8.

2. A FRAMEWORK FOR EVALUATING MSP PROGRAM'S PARTNERSHIPS

2.1 DEVELOPING AN EVALUATION FRAMEWORK

« back to contents

Previous research has extensively examined institutional partnerships in math and science education, as well as in related fields. Any assessment of the MSP's partnerships should be based on a framework derived from, or at least recognizing, this extensive research.

To prepare for an assessment of MSP partnerships and to examine the start-up and implementation phase, the upcoming sections provide an overview of the relevant literature on partnerships, beginning with the basic and most noted definitions of partnerships, integrated with a discussion about the development of an evaluation framework. The research appears to support (or be consistent with) a general framework as depicted in Exhibit 1.

One of the first important things to understand about partnerships is how the partners involved in the partnership process are defining it. Partnerships are complex and can be defined in a myriad of ways, often with subtle differences of demarcation. Some definitions of partnerships closely resemble other typesof associations, and the distinctions can be murky. Terminology associated with partnerships also is an issue. In the literature, for example, the terms collaborative partnerships, community-based interventions, and community coalitions overlap and generally "describe interventions with similar characteristics" (Wandersman and Florin, 2003). The authors go on to hypothesize that the overarching purpose of partnerships and coalitions alike is described as trying "to bring about changes through processes of collaboration, collaborative planning, and implementation across different agencies and community sectors."

To complicate matters further, many partnerships and associations are categorized into various schemas to classify their levels of interaction or participation. Much of the general research presents detailed categorizations that characterize coordination, cooperation, and collaboration (and sometimes additional terms) on a continuum. On these continuums, short-term, non-binding, limited engagements fall at one end of the spectrum (such as coordination or cooperation); and on the other end are long-term, binding or formalized agreements with financial commitments, staff and resource allocations, and decisionmaking mechanisms (such as collaboration). Numerous terms are used to describe the middle ground between these two distinctions (Himmelman, 1996; Hord, 1986; Intriligator, 1992). In this view, collaboration is "a definition of mutual relationships and goals; a jointly developed structure and shared responsibility; mutual authority and accountability for success; and sharing of resources and rewards," which also requires a formal commitment (Mattesich and Monsey, 1992). With regard to the MSPs, Kingsley (2005) has stated that it may not be possible to "classify partnerships as distinctive types." Because of the range of activities the partners undertake, they may describe or define what they are doing in a range of ways.

Berkowitz (2001) makes a distinction between community coalitions and other forms of community-based cooperative activities. Community coalitions "place most emphasis on: a) representation from multiple community sectors; b) attention to multiple community issues; c) active local citizen participation; and d) bottom-up planning and decisionmaking." He contrasts coalitions with other forms of collaboratives that are "often agency-driven and single-issue or both." He adds that, "terminology is a concern.... Some use 'coalition' and 'collaborative' (or 'collaborative partnership,' and sometimes 'consortium') as rough synonyms, though a 'collaboration' may involve more agency dominance and control."

Wolff (2001) refers to prior work by Florin and Chavis (1990) distinguishing between agency-based initiatives and community-based: "In the agency-based coalition, the intervention comes from the professionals and institutions in the community, and citizens are secondary players.4 In community-based coalitions... community members identify the issues, analyze the problems, select the interventions, and deliver the interventions and the evaluation." Wolff further cites Himmelman (1996) who distinguishes between collaborative betterment and collaborative empowerment. The empowerment model is designed to increase the community's capacity to set priorities and control resources.

These various distinctions are useful in examining the literature as it pertains to the various forms and activities of the MSP partnerships, but defining exactly what is a partnership is compounded by the different views authors hold about partnerships. For example, Pimmel defines partnership as: "A relationship between individuals or groups that is characterized by mutual cooperation and responsibility, as for the achievement of a specified goal" (Pimmel, 2004).

Butterfoss et al. (1993), define the ideal partnership as: "A dynamic relationship among diverse actors, based on mutually agreed upon objectives, pursued through a shared understanding of the most rational division of labor based on the respective comparative advantages of each partner. Partnership encompasses mutual influence, with a careful balance between synergy and respective autonomy, which incorporates mutual respect, equal participation in decisionmaking, mutual accountability, and transparency." This is a functional definition of partnerships in that it allows for a complementarity among partners in relation to a common goal.

Gouvis Roman et al. (2002) define a partnership as "a linkage between community organizations and government agencies formed for the purpose of reducing a defined social problem or improving the conditions of the community."5 Jehl et al., discuss the importance of partnerships between schools and "community builders" such as community development corporations, neighborhood-based organizations, faith-based groups, settlement houses, and others in including education reform as part of "their agenda to develop the community's social, physical, economic, and political infrastructure." They reported that these relationships can be difficult to sustain. The schools are driven by the demands of accountability and do not understand how the community can assist; and the partners do not understand the "magnitude of the challenge" of trying to improve academic achievement.

The literature also reveals that some believe an overarching definition of "partnership" is impossible to develop and that partnerships should be cast in terms of typologies or taxonomy. For example, through his NSF MSP grant, Kingsley (2005) has been working on the identification of critical factors for evaluating STEM partnerships. Kingsley asked a panel of experts to define partnerships. Their definitions fell into five major categories: 1) student-focused reform partnershipsthat framed partnerships as "a reform movement that brings together the resources of a community and educational institutions to strengthen teacher performance and close the achievement gap of students in low-performing schools;" 2) process partnerships that "focus on the way in which organizations and institutions interact with one another...great emphasis upon trust and mutuality of communication and resource exchanges;" 3) institutional partnershipsthat focus on the "structural alignment of organizations that are brought to bear upon the challenges of STEM education...greater emphasis was also placed on the complementarity of strategic goals across organizations;" 4) professional partnershipsthat "focus on the shared sense of professional practice and emphasizes the partnership as a manifestation or, in some cases, a catalyst for a learning community linking K-12 and institutions of higher education in math and science;" and 5) partnerships are institutionswho hold "the partnership as a distinctive institution that then serves as a clearinghouse and point of connection among institutions with a stake in STEM education."

Many partnership constructs derive from the world of business and management. In defining collaborations, the business literature distinguishes between transactions and alliances. AriÒo et al. (1999) define an alliance as a formal agreement between two or more business organizations to pursue a set of private and common goals. Ghere (2001) characterizes public-private partnerships as "long-term ventures that involve numerous financing and responsibility-sharing agreements that can be aptly discussed in terms of choices that arise in the contexts of interdependence and imperfect information." Linder, (2000) states that public-private partnerships further require that each partner acquire, adopt, or adapt characteristics and points of view of the other partner(s) "that once defined and stabilized the identities of their counterparts."

Savas (2000) believes that public-private partnerships can be thought of in three different ways. First, this type of partnership can be used to describe an arrangement in which the partners form a union to produce and delivers goods and services. Second, it can be used to describe "complex, multipartner, privatized, infrastructure projects." Third, it can refer to corporations exceeding their usual marketplace role and becoming involved in schools, etc. Kettl hypothesizes that private partnerships with the government bring three key issues to the forefront. First, when the government partners with a private supplier of services, the government is essentially relying on "private partners to do public work," which has resulted in a new form of public management. Second, this new form of management is far from uniform because of the "highly variable" relationships, which require unique and directed management approaches. Third, the role of the government has changed in that it is now "less the producer of goods and services and more the supervisor of proxies who do the actual work" (Kettl, 1993 and 1988; Mosher, 1980; Salamon, 1981).

Oliver (1990) makes a distinction between partnerships and interorganizational relationships (IORs). He defines IORs as "the relatively enduring transactions, flows, and linkages that occur among or between an organization and one or more organizations in its environment." Oliver further describes six contingences of relationship formation (or reasons why organizations choose to enter into relationships with one another): 1) necessity; 2) asymmetry; 3) reciprocity; 4) efficiency; 5) stability; and 6) legitimacy.

These various definitions have in common the concept of collaboration among different entities towards a shared goal, where the outcomes of the partnerships as a whole are greater than the sum of what the individual partners contribute (Brinkerhoff, 2002).

2.2 ROLE OF PRE-EXISTING PARTNERSHIPS, IF ANY

« back to contents Assesing
Partnerships: An Example of Sequential Phases: Pre-Existing Partnerships
(If Any)

Pre-existing Relationships. When pre-existing partnerships or relationships exist, they can provide a foundation of familiarity, shared interest, mutual commitment, and trust, which therefore may accelerate the rate of implementation of new grants while facilitating start-up and also the partnership process. In many instances, successful partnerships and relationships are sustained from grant to grant. In these cases, those lasting beyond the grant period usually have a combination of individual and institutional support focused on developing and sustaining the relationship (Phillips, et al., 2004).

The general consensus in the literature is that to be successful, partners should either have a pre-existing relationship or be able to devote time during the initial planning phases getting to know one another and building a relationship. However, sometimes the funding agency does not view this step as necessary or does not believe that a grant can allot the time to do so (Seifer and Krauel, 2003).

Rackham et al. (1996) specified three key areas that are consistently correlated with successful partnerships: 1) impact; 2) intimacy; and 3) vision.6 Of these three, intimacy is relevant to pre-existing partnerships because it denotes the level of closeness of the participants in the partnership and how well they relate to each other. Pre-existing partnerships would have developed and attained a certain level of intimacy that impacts the partnership as a whole.

Alternatively, pre-existing partnerships or relationships may inhibit the pace of implementation due to previously established institutional patterns and behaviors. For example, partners may not agree or fully support a new partnership's new leadership (assuming that it is the same), new operational components, new overall vision, or new direction. Further, partners may perceive that they are not being dealt with fairly or are not equally represented at the table, compared to their earlier roles. Partners also may feel strain with regard to their own internal economic and financial priorities that may not align with that of the overall partnership. Given these historically ingrained patterns and sentiments, the partnership may suffer because these issues carry forward to the new undertaking.

Of the MSP Program comprehensive (cohorts I and II), targeted (cohorts I, II, and III), and institute (cohorts II and III) awards, some partnerships and/or individuals within the partnerships have received prior NSF funding.

Since not all of the MSP Program's grantees had pre-existing partnership(s), the MSP Program did allow for the development of partnerships that seemingly would not have existed without it. It remains unclear at this juncture in the evaluation whether these partnerships formed solely to apply for the grant award or formed for another reason. It would be reasonable to speculate that the partners would have had prior interactions, knowledge of one another, and shown a willingness to commit to a joint endeavor. Moreover, all MSP partnerships have engaged in important partnering activities in preparing their MSP proposals. From this standpoint, the timing of the partnerships' startup predates the eventual MSP award, receipt of which in turn might be construed as the first significant partnering accomplishment.

Community Readiness. When examining the context of the pre-existing relationship it is important to consider not only the partnership's readiness but the community's as well. For the MSP partnerships, readiness among the K-12 districts and their communities—parents, businesses, and community organizations—would have been a relevant condition. Community readiness is generally described as a critical prerequisite to assessing a partnership's success (Birkby, 2003; Drug Strategies, 2001; Goodman, et al., 1996; Wolff, 2001). Goodman et al. (1996) used a social ecological approach to examine community-based interventions—interventions consisting of applying multiple strategies across multiple levels of a community. They found that the selection of effective interventions was in part dependent on the community's readiness to address the concern.

Wolff (2001) considers the state of the community prior to the creation of the coalition to be critical. Factors that increase readiness include: motivation from within the community, positive prior history with collaborative efforts, and existing leadership. Factors that can impede readiness include unresolved turf wars and a condition that he describes as being "overcoalitioned." Drug Strategies (2001) found that it is helpful, especially in the initial stages of the coalition, to have what they term a critical mass of members with social capital, namely: ready access to businesses, funding sources, media, and other major community institutions.

Gouvis Roman et al. (2002) identify a number of positive and negative influences on a community's readiness to participate in a community justice partnership. These include the capacity of the organizational partners; prior history of collaborations in the community; the existence of politics or turf wars; funding history; partnership over-saturation; and the community's willingness to evolve and change.

2.3 MSP START-UP OF MSP PARTNERSHIP

« back to contents Assesing
Partnerships: An Example of Sequential Phases: Start-Up of MSP Partnership

Start-up of the partnerships actually began during the application phase of the grant (prior to award). In terms of start-up activities for partnerships, the research literature identifies several guidelines for forming effective partnerships.

Of these dimensions, the following important components will be reviewed:

Composition and size of the partnership;
Size of the region covered by the partnership; and
Goals, objectives, values, and mission of the partnership.

Composition and Size of Partnership. There appears to be consensus that inclusive and diverse memberships are key principles to building a successful partnership (Birkby, 2003; Drug Strategies, 2001; Kumpfer and Chavez, 2000; Wolff, 1997). Inclusive in this context means that all members who endorse the coalition's mission should be able to join, and the two power extremes must be invited—the most powerful as well as the least powerful (Wolff, 1997). Birkby adds that successful coalitions have ongoing recruitment efforts and pay attention to the retention of existing members.

Kumpfer and Chavez (2000) report that when the main purpose of the partnership is to establish and maintain broad-based support for the effort, it is important to have partners represent all segments of the community so that the range of community norms can be addressed.

In examining community justice partnerships, Gouvis Roman et al. (2002) write, "Because community justice initiatives aim to articulate the voice of the community and improve quality of life for everyone that uses or provides resources to the community, the range of stakeholders is broad."

As of the award period 2003-2004, the MSPs were composed of a diverse range of partners representing IHEs, LEAs, and the private and public sectors. As a requirement of the grant, the partnerships must include at least one IHE or other eligible nonprofit and one or more LEAs. Partnerships may further include state education agencies (SEAs) and for-profit entities. As mentioned earlier in this report, partners agree to a certain level of commitment in terms of responsibility and accountability at the outset of the grant (termed core or non-core partner). Exhibit 2 provides an illustrative example from the North Cascades and Olympic Science Partnership of the types of partners commonly aligned for this program. This particular partnership, for instance, is composed of five IHEs (all of which are core), 28 LEAs (all of which are core), three SEAs, and one foundation.

As of the award period 2003-2004, the MSPs partnered with a number of prominent businesses. Businesses are important partners in MSPs, both for their engagement in the work of the partnerships themselves and for their perspectives on STEM preparation of the workforce. MSPs have partnered with a number of corporations and businesses, as well as with local Chambers of Commerce.

The following are illustrative examples of partnerships that existed between MSPs and business:

Appalachian Mathematics and Science Partnership: Kentucky Science and Technology Corporation;
East Alabama Partnership for the Improvement of Mathematics Education: Blue Cross/Blue Shield;
Mathematics and Science Partnership of Greater Philadelphia: WHYY, Inc.;
Merck Institute for Science Education: Merck & Company, Inc.;
Partnership for Student Success in Science: Agilent Technologies and Synopsys, Inc.;
Project Pathways: A Math and Science Partnership Program for Arizona Targeted Project Track: Intel Corporation;
Puerto Rico Math and Science Partnership: Texas Instruments, Inc. and the Ford Motor Company;
SUNY-Brockport and Rochester City: Texas Instruments, Inc. and Xerox Corporation;
Teachers and Scientists Collaborating: Progress Energy and GlaxoSmithKline; and
Vermont Mathematics Partnership: IBM Corporation.

In addition, multiple MSPs, including The Milwaukee Mathematics Partnership and the El Paso Math and Science Partnership, involved Chambers of Commerce as partners.

All of the comprehensive (cohorts I and II), targeted (cohorts I, II, and III), and institute (cohorts II and III) awards' partnerships have one Principal Investigator7 and multiple co-Principal Investigators.8 Exhibit 3 lists the Principal Investigators and their institutional affiliations.

All but eight of the Principal Investigators are affiliated with an IHE. The eight Principal Investigators not associated with an IHE come from a variety of places including a science center, school district, national-level association, and county level education service center. The majority of the co-Principal Investigators is associated with an IHE.9

A subsequent phase of the substudy will examine if the organizational location of the Principal Investigator has an impact on the partnership.

Depending on a partnership's goals and objectives, SEAs can be a critical component of the partnership equation. These state-level agencies are involved and ultimately responsible for the roll-out of new state assessments—therefore they know the content and the timing of these events—and they help craft state policies. Six of the partnerships include SEAs as partners (some have multiple SEA partners).

The following are among others serving as SEAs partners:

Georgia Department of Education;
North Carolina Department of Public Instruction;
North Carolina Science, Mathematics, and Technology Education Center;
Vermont Department of Education;
New York State Education Department; and
Washington State Educational Service Districts.

As of the award period 2003-2004, in terms of the median number of partners within the partnerships in the comprehensive (cohorts I and II), targeted (cohorts I, II, and III), and institute (cohorts II and III) awards, Exhibit 4 shows that comprehensives tend to have the highest number of IHEs and districts involved (2.5 and 12 respectively), while the targeted and institutes are slightly lower, but approximately equal across the same categories. All three types of awards had a median of one "other" partner. The institute shows 0.5 due to the fact that some of the partnerships did not include partners in the category of "other."

The size of the partnerships varies dramatically. The smallest partnership has only two partners and the largest has 64 partners. The size of region covered, or the geographic dispersion, is significant in some cases as well. For example, as Exhibit 5 illustrates, only seven partnerships have all of the partners located in the same county; 31 are in more than one county; and 10 are in more than one state. The most geographically expansive partnership has partners located in four cities in four states.

Approximately one-half of the district partners were located in an urban setting; the majority of the remaining district partners were located in less densely populated settings (Silverstein, et al., 2005). In reviewing a random sample of six MSP grants, all of the partnerships, except for one, enacted the partnership with the partners originally proposed.10 In the one partnership that differed from the original set of partners proposed (The Mathematics and Science Partnership of Greater Philadelphia), it enacted one district-level partner that was not proposed and did not enact one district-level partner that was proposed. See Appendix B for a comparison of the proposed versus enacted partners for the six partnerships.

Information contained in this section is summarized in tabular format in Appendix C.

Agreeing On Goals, Mission, and Values. Birkby (2003) uses the term "intentionality" to describe this dimension: "Clear plans, attainable goals, measurable objectives, and community ownershipare critical to the success of the coalition." Kumpfer and Chavez report that each of five model partnerships that significantly reduced substance abuse had a comprehensive vision, covering all segments of the community and all aspects of community life, and a wide sharing of this vision, agreed upon by groups and citizens across the community.

Wolff (2001) speaks of "intentionality," which he describes as a common shared vision and mission. Intentionality also refers to "members' faith in themselves to tackle whatever issues come along." Wolff (1997) distinguishes between the self-interests of the various constituencies of a partnership and the shared mission and goals of the partnership, "Walking the tight rope between these agendas is critical to coalition success." Harms et al., (2001) describe a shared vision and mission as the foundation on which coalition actions are built.

All of the partnerships developed a set of goals and objectives and report to NSF their progress in meeting these (See Appendix D for a complete list of the partnerships' goals and objectives). In general, most of the goals express a high-level expectation and the objectives delineate how the goal will be achieved. For example, the Consortium for Achievement in Mathematics and Science has a primary goal (in italics) to: provide intensive, sustainable, systemic reform in four urban school districts with the vision that all middle school students will understand and be able to apply key concepts in math and science.

To achieve this goal they created the following objectives:

Implement challenging instructional programs;
Build professional capacity in schools, the university, Educational Testing Service, and Merck Institute for Science Education;
Develop leadership among teachers, administrators, and university faculty;
Develop a student-centered learning climate in every classroom; and
Build parent and community support.

As would be expected, some of the partnerships have revised their goals and objectives, or the timeline associated with them, based on their experiences during the first year of the grant.

2.4 ONGOING OPERATIONS

« back to contents Assesing
Partnerships: An Example of Sequential Phases: Organizational Structure/and Ongoing Partnership

Once the partnership has been established there are several preliminary operational activities that should take place. Jasuja et al. (2005) define operational processes as "factors related to ongoing operational functioning of the coalition and including staff role, leadership, decisionmaking, communication, conflict, costs and benefits of participation, organizational climate, and capacity building." Among other things, the partnership needs to come to agreement on the development of an overall organizational structure, create an action plan and operational guidelines, identify key leadership, seek resources (staffing, facilities, etc.) to support the partnership, and create a mechanism to communicate effectively.

Developing An Organizational Structure. To be successful and sustainable in the long term, partnerships need not only have an organizational identity, but also a structure or planning mechanism and the necessary resources to perform their tasks effectively (Birkby, 2003; Gouvis Roman, et al., 2002; Holland, 2001; Metzler, 2003; Wolff, 1997; Wolff, 2001). For university-community partnerships, it is important that there be formal governance partnership structures, including community-campus advisory groups, and, at the university level, integration of partnership activities into the curriculum and direct learning experience of the students (Holland, 2001).

Smock (1999), in reviewing partnerships developed during the 1990s, found structure to be an essential component of a successful partnership. Some organizations believed that to be inclusive and democratic, a partnership should keep formal structure to a minimum. Partnerships without structure experienced difficulties in start-up, recruitment, implementation, accountability, and communication. Moreover, rather than being inclusive, these partnerships were overwhelmed by a vocal minority.

Brinkerhoff (2002) finds that each partner's organizational identity and its unique strengths and resources form the foundation for partnership. It is important that each partner keep its unique organizational structure and goals. Organizational identity of each partner is key to the partnership's success. If it is lost, the partner loses its capacity to maximize its unique contribution. She further advocates that the partnership itself should develop an identity as it is:

"The glue that holds the partners together and forms the basis for legitimacy and values identification of major stakeholders. Partnership identity entails an identifiable organization culture, complete with processes and mechanisms reflective of the partnership's underlying values; a unique and identifiable mission, with associated comparative advantages and value-added; and a set of constituencies that go beyond the constituencies of individual partner organizations."

Maintaining identity is not easy. Researchers must learn to approach problems from the perspective of community members, while the community needs to understand the research perspective (Metzler, 2003). However, this does not mean that partners should lose their organizational identity, culture, or perspective.

Gouvis Roman et al. (2002) find that partnerships are more likely to succeed if "partnership structures support multiple organizational contacts with clear lines of communication across organizations, as well as equal decisionmaking among community organizations and government agencies....Success appears likely to be achieved when both horizontal integration (among community organizations) and vertical integration (between community organizations and traditional power holders) are strong."

Goodman et al. (1996) describe this stage of coalition development as the stage where turf issues are resolved. Unless these issues are addressed, the coalition will not have the ability to build capacity for action, the next stage of coalition readiness.

HUD's Office of University Partnerships (HUD, 2002) identifies five principal ways that universities can organize to work with community organizations: 1) centralized at top level of the administration with a university-wide commitment to the partnership; 2) decentralized to a specific department level; 3) located within interdisciplinary schools or centers; 4) organized by a separate nonprofit organization, jointly controlled by the university and the community partners; and 5) run in collaboration with other colleges and universities.

Setting Up and Implementing an Action Plan and Operational Guidelines. Developing an action plan and operational guidelines are important steps that cannot occur without prior readiness of the partners (Goodman, et al., 1996). Wolff (2001) emphasizes that a prerequisite to a good action plan is having goals and objectives that are "concrete, attainable, and, ultimately, measurable." He found that many coalitions fail to implement actions directed at bringing about the desired changes in the community. Rather, they become involved in internal readiness activities such as in-service trainings and the like. All MSP awards are required to develop a strategic plan and an implementation plan.

As of the award period 2003-2004, the partnerships in the comprehensive (cohorts I and II), targeted (cohorts I, II, and III), and institute (cohorts II and III) awards have many formal agreements in place. These include articles of incorporation, memoranda of understanding, subcontract agreements, letters of support, and guiding principles.

As but one example of how a partnership structured itself, The Consortium for Achievement in Mathematics and Science established a multi-tiered organizational system that allows for partner collaboration and communication. This includes a consortium management and oversight committee and a consortium planning and implementation team.

Qualities of the Partnering Relationship and Maintaining the Partnership. "Perhaps the greatest and unending challenge facing partnerships is the level of time and energy it takes to launch and maintain an effective partnership relationship" (Holland, et al., 2003).

The following partnership characteristics, culled from the literature, are associated with quality partnerships:

Mutuality and trust;
Leadership;
Obtaining financial and other resources; and
Collaboration and mechanisms of communication.

Mutuality and Trust. The concept of trust between organizations is difficult to define. Traditionally trust has been defined (and measured) as a dimension of interpersonal relationships. More recently, trust is being studied as an aspect of the relationship between organizations and the public. In a paper on trust in organizations, the Institute for Public Relations at the University of Florida (Paine, 2003) discusses trust as a multi-dimensional concept:

"Multi-Level: Trust results from interactions that span co-worker, team, organizational, and inter-organizational alliances.

Culturally-rooted: Trust is closely tied to the norms, values, and beliefs of the organizational culture.

Communication-based: Trust is the outcome of communication behaviors, such as providing accurate information, giving explanations for decisions, and demonstrating sincere and appropriate openness.

Dynamic: Trust is constantly changing as it cycles through phases of building, destabilization, and dissolving.

Multi-dimensional: Trust consists of multiple factors at the cognitive, emotional and behavioral levels, all of which affect an individual's perceptions of trust."

Paine discusses mutuality in the context of relationships between the organization and the public (or, in the case of partnerships, the community). She speaks of the importance to control mutuality: "For the most stable, positive relationship, organizations and publics must have some degree of control over each other." In other words, control of one party over the other decreases both trust and mutuality.

AriÒo (1999) points out that trust requires the presence of an element of risk and a "degree of vulnerability;" there must be an exposure to potential loss or harm. "Trust does not involve blind faith, nor is it particularly relevant in contexts involving certainty."

To Brinkerhoff (2002), mutuality "encompasses the spirit of partnership principles." It implies "horizontal, as opposed to hierarchical, coordination and accountability" and "equality in decisionmaking, as opposed to the domination of one or more partners." She lists the following indicators for mutuality: 1) equality in decisionmaking; 2) resource exchange—limited not just to financial resources but extending to managerial and technical skills, contacts, information and the like; 3) reciprocal, as opposed to hierarchical, accountability; 4) transparency in areas of common concern; 5) mutual respect based on the acceptance and recognition of each partner's contribution; and 6) degree of partner representation and participation in partnership activities.

Metzler (2003) studied partnerships between researchers and the community in three urban settings (New York, Seattle, and Detroit). She found that in these urban settings, the "legacy of racism" posed a particular challenge to the principle of mutuality. Researchers and agency heads were usually Caucasians with advanced degrees, while community partners were often minorities with less formal education. Historical discrepancies in power and trust had to be overcome before the community could move from "community-as-advisor to the community-as-consultant to community-as-full-partner in all phases of research." She adds, "Trust, patience, commitment and willingness to compromise were necessary for growing trust and building partnerships."

Brownson et al. (2001) stress that part of the participatory process in community partnerships is looking at differences in culture, not only race and ethnicity, but also organizational culture. As an example, he cites, "A university...may be a hierarchical organization that may seem overly bureaucratic to community volunteers." Researchers and community agencies may also have a different view of the same problem.

In the business and management literature, mutual trust is defined as the shared belief that individuals can depend on each other to achieve a common purpose. In an alliance, where the purpose is to get results that exceed what a transaction can do, mutual trust also means one can depend on the others to adapt as necessary. This involves more than keeping promises, because it entails changes that cannot be planned in advance (Lewis, 2000). Lewis lists eight conditions that must be present for trust to occur: 1) mutual need creates the opportunity; 2) interpersonal relationships make the connections; 3) joint leaders work closely together; 4) shared objectives guide performance; 5) safeguards encourage sharing of information and other resources; 6) commitment creates enthusiasm; 7) adaptable organizations support alignment; and 8) continuity sustains understanding.

During the coming phases of this study, mutuality and trust will be examined through the collection of primary data from the partnerships.

Leadership. Kumpfer and Chavez (2000) stress the importance of having a strong core of committed partners at the outset, or start-up phase. Even though coalitions are based on the notion of broad-based ownership and power sharing, several studies found that strong leadership is important to overall success (Birkby, 2003; Drug Strategies, 2001; Metzler, et al., 2003). Birkby adds that successful coalitions develop leadership among their members, rather than relying on a single charismatic individual.

Metzler finds that "the presence of a partnership champion (and the presence of champions within the partnership organizations)," is another characteristic of an effective partnership. She defines champions as:

"Entrepreneurial individuals who advocate on behalf of the partnership and the partnership approach within their home organizations, within the partnership as a whole, and externally. Championing capacity not only entails communication, negotiating and organizational skills, but also perceived legitimacy among partners and stakeholders."

Wolff (2001) speaks of collaborative leadership—leaders who share power. "They focus on facilitation and process rather than decisionmaking. They are flexible rather than controlling, decentralized rather than centralized, inclusive rather than exclusive, proactive rather than reactive, and they focus on process and product rather than product only. At the core, collaborative leaders need to be risk takers."

Gouvis Roman et al. (2002) use the term transformational leadership to describe the concept of effective leadership. Transformational leadership "generates awareness and acceptance of the purposes and the mission of the group as [the leaders] stir [community partners] to look beyond their own self-interest for the good of the [community]." Their research concludes that transformational leadership is "based in trust and communication; it can be expressed by the following leadership skills: developing leadership and effective followership, building interconnectedness, mobilizing and empowering the informal community, and articulating the community voice."

As discussed in Section 2.3 above, the majority of the MSP Principal Investigators (or grant leaders) are associated with an IHE. Twenty-three of the partnerships have the Principal Investigator and at least one co-Principal Investigator located at the same IHE. Three of the partnerships that have Principal Investigators affiliated with an institution other than an IHE also have co-Principal Investigators working with them at the same organization. The co-Principal Investigators are quite geographically dispersed in some instances.

Some of the partnerships have presented organization charts suggesting highly-structured leadership. For example, Exhibit 6 presents the organizational chart of the Partnership of Student Success in Science grant's leadership organizational chart. This organizational chart clearly illustrates the grant leadership and staffing of the critical partnering entities and their primary roles. It depicts collaboration with an advisory board and input from administrators. It further shows where the key staff members are located institutionally and the amount or percentage of time to be committed to the effort.

Obtaining Financial and Other Resources. The source of funding and the way funding is used are cited by a number of studies as important elements to consider. In lessons learned from successful partnerships, Kumpfer and Chavez (2000) cite the importance of a partnership having a fiscal agent, such as a university, a community-based organization, or a public agency. They state, "One key to later operational success is how the partnership, its board, and its fiscal agent interpret and divide their responsibilities for important functions such as the hiring and firing of staff or deciding on strategic directions." They also found that when partnerships with substantial new funds are formed an initial challenge that must be recognized and addressed is the partners' understandable focus about how much funding each partner will receive. Metzler et al. (2003) found that the ability of the partnerships to secure financial and other resources was critical to their development. Drug Strategies (2001) recommends having diversified funding sources since relying too heavily on one source can mean that the coalition will fall apart if the original funding source withdraws or ends support.

Gouvis Roman et al. (2002) conclude, "To act as a capable partner, an organization must have some asset to bring to the partnership. This can be a tangible resource, such as money, supplies, or time, or an intangible resource, such as generating participation or having a strong understanding of community problems." They identify three types of resources: 1) human; 2) financial; and 3) technological, stating that to be a competent partner, an organization need not have each of these resources.

HUD's Office of University Partnerships (HUD, 2002) found the provision of human and financial resources to be an important benefit to community organizations participating in university-initiated partnerships. Human resources included individuals from one partner helping as staff or board members of the other. A review of the program reports that "Resources—especially human assistance—flow both ways. Colleges and universities gain from community development partnerships when the neighborhood comes into the university as well as when faculty, staff, and students go out into the neighborhood. Community residents and staff and board members [of community development corporations] contribute as classroom speakers, as panel participants, as student and faculty mentors, and as members of advisory committees or joint task forces."

As discussed in Section 2.2 above, many of the partnerships leveraged the existing partnership arrangement to obtain their current MSP grant in addition to awards from other sources. In the coming phases of this study, this issue and sustainability among MSP grantees will be explored through primary data collection.

Collaboration and Mechanisms of Communication. Dimensions of collaboration and communication are combined here since effective communication (listening and providing feedback) appears to be an essential component of effective collaboration.

Gajda (2004) identifies the following principles of collaboration derived from observed facts about the development of strategic alliances between organizations:

Collaboration is an imperative: "there is an ever increasing need for individuals, educational authorities, community networks and business groups to come together to address the complex issues that confront our society today;"
Collaboration is known by many names, including: join ventures, partnerships, networks, partnerships, coalitions, consortiums, councils, task forces, and groups;
Collaboration is a journey not a destination;
With collaboration, the personal is as important as the procedural; and
Collaboration develops in stages.

Maurana et al. (2001) identify principles of good partnerships that appear in effect to be principles of collaboration, including concepts of trust, leadership, and communication. Metzler et al. (2003) present principles of collaboration that guided collaboration between academic researchers and three urban communities. These principles provided criteria by which to gauge the actual collaboration in the research effort.

Collaboration also includes ability to resolve conflicts within the collaborative framework (Birkby, 2003; Drug Strategies, 2001; Kumpfer and Chavez, 2000). Birkby describes the success of a coalition on its ability to "create an environment where conflict can be surfaced and handled effectively and efficiently, rather than avoided."

As of the award period 2003-2004, the partnerships are communicating through a variety of mechanisms. These include:

Establishment of advisory boards, steering committees, or advisory councils;
Regular meetings, conference calls, or electronic communications;
Development of grant coordinator type-positions;
Creation of management and communication plans;
Development of an organization chart; and
Development of forms (e.g., reporting forms, logs, etc.).

The nature of the pre-existing partnerships may further contribute to lines and modes of communication being more readily established.

2.5 ACTIVITIES

« back to contents Assesing
Partnerships: An Example of Sequential Phases: Math and Science Education Activities

Once the partnership is established, the partners are positioned to begin undertaking activities and implementing proposed programs. These activities may be either external to the partnership or internal. An example of an external activity would be to implement a professional development module for teachers or provide training and technical assistance. An example of an internal partnership activity would be to conduct a process evaluation of the partnership and then use those data to improve the workings of the partnership or to conduct an evaluation of the grant's proposed outcomes to monitor and track results.

As of the award period 2003-2004, the MSP partnerships have undertaken a wide range of activities that can be categorized into the following areas (see Appendix E for a list of partnerships' activities):

Provided professional development through workshops, training, seminars, etc.;
Developed curriculum for professional development;
Recruited teachers, teacher leaders, professionals changing careers, or mentors (focused on diversity);
Developed K-12 curriculum for math and science and developed (enhanced or modified) courses for pre-service teachers;
Encouraged enrollment in challenging courses;
Developed and implemented a new Master's degree program;
Developed certification program;
Aligned school curriculum or university instruction with state standards;
Collected data to assess various components of the grant or the partnership;
Published journal paper or wrote article for publication in other venues (e.g., newspaper, magazine, etc.); and
Increased awareness about the grant through the development of a Web site or convening meetings.

Note that not listed in the above, are all of the organizational grant-specific activities the partnerships undertook such as authored a strategic plan, developed a communication system, identified the appropriate staff, or convened monthly grant team meetings.

By far the greatest number of activities occurred in the area of professional development through workshops, trainings, and seminars. The activities include identify and hiring individuals to provide the professional development; creating and designing professional development modules; coordination of professional development activities; opening a professional development center; and evaluating the sessions.

The partnerships also have engaged in a variety of recruiting activities to increase the level of diversity among teachers, to increase the numbers of qualified math and science teachers, and to fill newly created positions such as teacher leaders and mentors. These activities include recruiting potential teachers from different venues of the education continuum and community such as through high schools, colleges and universities; community-based mentoring programs; recruiting career-changing professionals; recruiting individuals with specific backgrounds such as science, math, engineering, and technology; and recruiting teachers into specific education programs.

Some of the partnerships conducted activities that involved designing, developing, restructuring, or revising curriculum or courses at either the K-12 level or the university level. Components of the professional development activities discussed above included the training of teachers to use the revised curriculum. Part of this transformation process included the adoption and implementation of standards-based curriculum that aligned with state standards, state expectations for math and science, other tests, and ensured readiness for college expectations. As of the award period 2003-2004, five partnerships either developed or are in the process of helping enroll teachers in a Master's degree program geared toward teaching math or science. For example, available programs include Master of Arts in Teaching Mathematics, Master of Arts in Teaching Science, Master of Science in Science Education, Master's for training in-service science teachers, Master of Integrated Science Education, Master of Chemistry Education, Master's degree with a specialization in math or science for middle school teachers, and one specifically for math teaching fellows. Other partnerships have developed certification or licensure programs.

Requesting Training and Technical Assistance. Training and technical assistance are viewed as critical to the success of partnerships (Birkby, 2003; Mitchell, et al., 2002; Wolff, 2001). Emphasizing the need to develop effective technical assistance systems to support implementation activities, Mitchell et al. (2002), write that a technical assistance system can provide the intermediate support needed to develop an organizational infrastructure, design appropriate interventions, and engage the community. In 2004, Mitchell et al. reported on the outcomes of a statewide technical assistance project to 41 partnerships. The researchers found that coalitions with greater initial capacity were more likely to identify technical assistance needs and request expert assistance. However, data obtained from key informants did not indicate a relationship between levels of technical assistance and observed outcomes.

Walker et al. (1999) reviewed community partnerships for cultural participation that were sponsored by the Lila Wallace-Reader's Digest Fund. They found that technical assistance was needed by the local foundations as well as by their grantee/partners. The foundations needed assistance in areas such as evaluation, performance measurement, and related data collection. Local community organizations (the partner/grantees) often needed technical assistance in a number of areas, including how to complete the grant application process.

HUD's Office of University Partnerships (HUD, 2002b) identifies a number of approaches universities have used to provide technical assistance to partners. The technical assistance may be provided directly by faculty and students, by a specific program within the university, or by an umbrella technical assistance organization. Technical assistance provided through these approaches addressed organizational development, articulation of mission and goals, creation of an organizational structure, strategic planning, financial management, and grantwriting.

None of the MSP partnerships reported needing or requesting formal technical assistance from NSF outside of the NSF-sponsored grantee conferences or the online Web-based forums offered through MSPnet.

Assesing
Partnerships: An Example of Sequential Phases: Math and Science Education Activities + Partnership and Education Outcomes

Monitoring Results. Closely tied to the importance of agreeing on the desired outcomes of the partnership, is the need to have a plan to measure performance (Drug Strategies, 2001; Metzler, 2003). Metzler found that effectiveness included maintaining a continuous assessment presence whether through internal or outside evaluators, and reviewing findings.

It is essential that there be a theoretical link between the activities of the coalition and the outcomes (Connell, et al., 1995). Without such a link it will be difficult to attribute observed changes to the partnership.

The Institute of Medicine conducted a study, jointly funded by The Robert Wood Johnson Foundation and the Office of the Assistant Secretary for Health, U.S. Department of Health and Human Services, to examine how performance monitoring can be used to improve the public's health through considering the roles played by the many stakeholders working towards improving community-wide health. Although the emphasis of the study was on developing health indicators for specific public health concerns (such as breast and cervical cancer, violence, elder health, and depression), the study also examined the collaboration among various stakeholders. Findings show that "bringing together diverse groups that influence community health is critical to building support and acceptance of performance monitoring" (The Robert Wood Johnson Foundation, 1998).

Building coalitions takes time; achieving the desired outcomes takes even longer. The importance of patience, time, and persistence, and taking a long-range view is repeated in most studies (Brownson, et al., 2001; Metzler, 2003; Wolff, 1997). Brownson sums up the key role of time as follows:

"The participatory research processes are complex and time consuming. In some cases, practitioners and researchers may attempt to measure an outcome before sufficient intervention exposure has occurred, making the demonstration of a significant effect virtually impossible. Processes of trust building and gaining mutual respect can be very time intensive."

In planning to evaluate partnerships, it also is important to explore reasons for avoiding partners or partnership relationships. Some of the primary reasons include:

Different purposes for pursuing similar objectives;
Different means of operating;
Poor history of previous interactions;
Political subtleties;
Lack of support from senior management;
Lack of quick fixes; and
Concerns about cost of getting started and of continuing maintenance (NSF/Pimmel, 2004).

2.6 MEASURING OUTCOMES AND DEVELOPING AN EVALUATION FRAMEWORK

« back to contents Assesing
Partnerships: An Example of Sequential Phases: Partnership and Education Outcomes

To develop an evaluation framework for the MSP partnerships, it is important to start with the process itself—a process that is complex, challenging, dynamic, and ever-changing. Systematic examination of these processes will help to determine, within a relatively short time frame: 1) the basic characteristics of the partnerships; 2) how partners work with each other; 3) the dimensions of the partnering relationship; and 4) immediate effects of the partnership on the partners. In the long term, it will be essential to determine the effect of the partnership on the intended outcomes (such as changes in student achievement). However, it will be difficult to determine the extent to which partnerships actually work if the only outcome studied is distal in nature, and if no theoretical link has been established between the partnership and its long-term outcomes. In fact, much of the evidence about partnerships' contributions to overall performance is, with the exception of a few private sector alliances where increased efficiencies have been documented and quantified anecdotal (Shah and Singh, 2001).

To attribute distal outcomes to the work of the partnership, it is important to have documented the partnership start-up process, identified key elements of the partnering relationship, and assessed the immediate effects of the partnership on major stakeholders: the members of the partnership, the partnership itself, and the targeted community. In the case of a university-K-12 partnership, the members are: 1) the researchers, faculty, and administrators at the university as wells as the students (who may be termed "service learners"); 2) the K-12 teachers, administrators, and students; 3) other partnering organizations (such as SEAs or nonprofits) and community advocates; and 4) the members of the targeted community.

Measuring Partnership Processes: Assessment Methods and Instruments. In addition to looking at the overall outcomes, it is essential to assess the functioning partnership itself. Schulz et al. (2003) report "evaluators interested in evaluating partnerships find few assessment instruments available to them."

As of the award period 2003-2004, some MSP awards began to look at partnership evaluation at the outset, while others showed evidence of wanting to conduct an evaluation at some point in the future. In subsequent years of the partnerships as it matures, it will be important to examine the role of STEM discipline faculty engagement in the partnership and how they have contributed to achieving the goals of the partnership. Of the MSP Program comprehensive (cohorts I and II) and targeted (cohorts I, II, and III), three partnerships are planning to evaluate their partnership in terms of characteristics of an effective partnership, efficacy, and cultural changes. These include:

Milwaukee Mathematics Partnerships: Sharing Leadership for Student Success;
Promoting Reflective Inquiry in Mathematics Education; and
Math and Science Partnership in New York.

The Milwaukee Mathematics Partnerships: Sharing Leadership for Student Success grant is measuring the degree to which a true effective partnership was established and identify the defining attributes of such a partnership. The Promoting Reflective Inquiry in Mathematics Education grant is planning to evaluate the efficacy of the partnership. One of the primary components of the Math and Science Partnership in New York grant's evaluation will be to evaluate cultural changes including reward systems, district priorities and policies, IHE priorities and policies, and lines and type of communication and participation.

Three additional partnerships have the instruments to do an evaluation of the partnership. These include:

Cleveland Math and Science Partnership;
Consortium for Achievement in Mathematics and Science; and
Partnership for Student Success in Science.

The Victorian Health Promotion Foundation developed the partnership evaluation instrument for the Cleveland Math and Science Partnership and the initial results appear in the evaluation report. The Consortium for Achievement in Mathematics and Science evaluation questions included: 1) To what extent is the Consortium using existing resources and lessons from previous initiatives to their advantage? 2) How efficiently and effectively do partners work together? and 3) To what extent are the resources and capacities of the Consortium partners adequate for carrying out Consortium goals with quality? The Partnership for Student Success in Science partnership evaluation will include components that evaluate building a functional and healthy relationship.

One other partnership, SUPER STEM Education, will provide lessons learned documentation of what works and information about how to construct such a partnership to a wide audience of policy makers and university and school leaders. The North Cascades and Olympic Sciences Partnership team read and discussed "Effective School-College Partnerships, A Key to Education Renewal and Instructional Improvement" (Education, Summer 2001:732-736) to increase their understanding of partnerships and asses their prior interactions against the described criteria to identify strengths and areas for improvement. The Standards Mapped Graduate Education and Mentoring partnership reported that its advisory board would provide comment on the general progress and direction of the partnership.

In evaluating partnerships some effective instruments do exist and some have been adapted and modified for evaluation purposes. While Exhibit 1 in this document shows the overarching evaluation framework for this examination of partnerships, there are a significant number of instruments and frameworks that could be used to augment or supplement the proposed framework. Some of these could be used in later phases to supplement or augment the work on this topic. Methods include: in-person interviews with partners, partner surveys, and observations. Other researchers used similar methods. As shown in Exhibit 7, most of the instruments discussed below cover more than one dimension of working with partners. For example, Brinkerhoff (2002) presents a detailed list of assessment targets and methods (see Exhibit 8).

To assess the working of community coalitions, the John S. and James L. Knight Foundation and The Robert Wood Johnson Foundation developed a number of survey instruments: 1) a nine-question expert advisory panel instrument; 2) an 18-question coalition mail survey; 3) an approximately 45-minute telephone survey for coalition leaders; 4) a 20-minute telephone survey of key informants (e.g., non-coalition community leaders); and 5) an in-depth site visit guide (Drug Strategies, 2001). These instruments are comprehensive in nature and vary in topical areas covered and length.

A community organizational assessment tool developed by the Citizens Involvement Training Program at the University of Massachusetts- Amherst, as a mechanism to facilitate organizational discussion and development, may be relevant to partnership development (Bright, 1998).

The Nonprofit Management Education Center of the University of Wisconsin Extension has developed a Strategic Alliances Assessment Tool that may be relevant to assessment of strategic planning by partnerships (Lewis, 1998). The tool is based in part on the above-referenced community organizational assessment tool (Bright, 1998) and a checklist of nonprofit indicators developed in 1998 by the United Way of Minneapolis Area.

Hays et al. (2000) studied the relationship between the structure of substance abuse prevention coalitions and community impact. Below are the measures they used to assess the following constructs: sectorial representation and member diversity.

Sectorial Representation. The members of each of 28 Illinois coalitions were asked to identify the community sector they represented from among 17 different sectors. Sectorial representation was measured as the total number of unique community sectors represented on a given coalition.
Member Diversity. On the assumption that diversity usually means the inclusion of non-white members, member diversity was measured as the percentage of non-white members in a coalition.

Butterfoss et al. (1996) developed a Plan Quality Index. The instrument examines respondents' assessment of the components of a committee plan: goals and objectives, scope, and community resources. It also includes a pre-implementation checklist.

Gardner (1995) developed a 29-item Community-Based Self-Assessment Instrument that measures nine dimensions of a community organization's progress toward responding to policy changes. The nine dimensions are: 1) collaboration with other agencies; 2) internal agreement on values and mission; 3) diversity and inclusiveness; 4) organizational priorities; 5) budgets and resources; 6) staff and leadership development; 7) commitment to outcomes and accountability; 8) response to policy changes; and 9) future role of the organization.

Harms et al. (2001) developed a community readiness instrument and a sustainability assessment worksheet for Washington State's coalition for children's oral health. Bell-Elkins (2002) developed an instrument to assess principles of partnership in a community-campus partnership.

Butterfoss et al. (1996) developed a 129-item self-administered survey to measure the effectiveness of committees of community coalitions for prevention and health promotion. The instrument was derived from existing instruments and tested for reliability (all but one had high internal consistency). Committee characteristics covered by the survey were: leadership roles, staff-committee relationships, organizational climate, decisionmaking processes, community linkages, member satisfaction, member participation patterns, and members' costs and benefits.

Van Houten et al. (2000) developed a number of instruments to assess the establishment and effectiveness of state and minority health networks. Relevant to the study are: 1) a list of key study questions and corresponding subquestions and 2) sections of an interview guide for directors of established offices of minority health and for minority health contact persons.

Goodman et al. (1996) developed a meeting effectiveness inventory asking respondents to rate the meeting's agenda, leadership, decisionmaking, and value.

Measuring Key Indicators. As mentioned earlier in this report, mutuality and trust are critical elements of a successful partnership and therefore should be part of the measurement process. Below are measures that specifically focus on mutuality and trust, leadership, and collaboration.

Mutuality and Trust. The U.S. Department of Education's Institute of Education Sciences holds that a key item in a study analyzing outcome data is that that the measures are valid; that is, they accurately measure the true outcomes that the activity was designed to affect (U.S. Department of Education, Institute of Education Sciences, 2003). Metzler et al. (2003) searched for validated instruments to measure trust between community and research partners, and were unable to find any. The following instruments measure trust from a management perspective. It may be possible to adapt them to the trust in the MSP partnerships.

Paine (2003) presents a trust measurement questionnaire intended to answer the following three questions: 1) have the organization's programs and activities changed what people know, think, or feel about the organization, and how they act; 2) have the actions of the organization had an impact on how constituents trust the organization; and 3) can the organization document that its communication efforts have increased this trust? The instrument covers: mutuality, commitment, satisfaction, communal relationships, and exchange relationships. Communal relationships are those in which both parties provide benefits to each other; in exchange relationships, one party gives benefits to the other, because the other party has done so in the past or is expected to do so in the future. According to Paine, "communal relationships are essential to developing and enhancing trust in an organization."

Glaeser et al. (2000) combined two experiments and a survey to measure trust and trustworthiness, which they define as two key components of social capital.

Gillespie (2003) developed a behavioral trust inventory based in part on existing measures of trustworthiness, disposition to trust, trust in the team, common values, common goals, interdependence, risk in the relationship, relationship effectiveness, overall trust, strength of the relationship, and satisfaction with performance.

Leadership. Hays et al. (2000) studied the relationship between the structure of substance abuse prevention coalitions and community impact. They measured leadership effectiveness through a six-item instrument assessing members' perceptions of the extent to which the coalition leader directs the group towards collaborative goal achievement. Each item was measured on a five-point Likert scale. Goodman et al., developed another instrument, a key leader survey (1996).

Collaboration and Communication. Gajda (2004) has developed an assessment tool, the Strategic Alliance Formative Assessment Rubric, based on the above-mentioned principles of collaboration. The tool can be used to help partnerships measure the relative strength of their partnership over time.

Hays et al. also developed a measure of collaboration. Members were asked how frequently they engaged in six increasingly complex collaborative activities with other partners. Responses were measured on a five-point Likert scale.

Researchers at the Amherst H. Wilder Foundation in St. Paul developed the Wilder Collaboration Factors Inventory that assesses collaboration's strengths and weaknesses (Mattessich, 2001).

Gardner (2000) has created instruments to examine collaborative values in California partnerships for substance abuse prevention.

The National Network for Collaboration (Borden and Perkins, 1999) developed a collaboration progress chart. The chart allows partnership members to rate the partnership as the following factors: goals, communicating, sustainability, research and evaluation, political climate, resources, catalysts, policies and reputations, history, connectedness, leadership, community development, and understanding the community. A definition of each of these factors is part of the instrument.

The Cooperative State Research, Education, and Extension Service, U.S. Department of Agriculture has created five national networks to marshal faculty and program resources to respond to the economic, social, and human stresses faced by children, youth, and families (Bergstrom, et al., 1995). These national networks created a collaboration framework to address community capacity. The framework is designed as a planning tool to develop and sustain collaboration, as well as a diagnostic tool to evaluate ongoing development and progress.

In order to measure partnerships, Kingsley and O'Neil (2004) developed a three-staged partnership logic model. Stage one, partnership preconditions, examines the embeddedness of the partnership. Kingsley and O'Neil define embeddedness as the number and types of relationships that organizations have with one another prior to the development of the partnership. Stage one further explores the strategic needs or the types of resources and need confronting organizations and whether there is a congruence or complementarity in these needs. Stage two, partnering activities, looks at partnership formation (including aspects such as agreements, goals, resource allocation, etc.) and partnership operations or the actual behaviors in which the partners engage. Stage three, partnership outcomes, examines both process and performance outcomes. Kingsley and O'Neil define process outcomes as "the qualitative and quantitative assessments that measure whether the partnership achieved the goals and duties of operation" and performance outcomes as "assess such improvements as in the working environments of the organizations, transfer of knowledge between organizations, or increased ability to quickly innovate" (Kingsley and O'Neil, 2004). The MSP Program realized that rich information existed about partnership in other arenas beyond education and therefore funded a RETA (see page 8) to inform this specific knowledge base.

Evaluating Long-Term Outcomes. Even though this report does not focus on measuring the long-term outcomes of MSP partnerships, it is relevant to briefly discuss some of the challenges associated with such measurement and evaluation. It is precisely these challenges that make it important to document and measure the establishment and working of partnerships, and their immediate effect on the capacity of all participants to address the targeted issues. In a report to congressional committees, the U.S. General Accounting Office states that having collaborative partnerships is one of the key indicators of evaluation capacity (U.S. General Accounting Office, 2003).

Evaluations of the long-term effect of partnerships on targeted issues show a mixed record. Birkby (2003) reviewed literature on effectiveness of coalitions and identified a number of studies that were not able to conclusively demonstrate effectiveness of major initiatives. An evaluation of The Robert Wood Johnson Fighting Back program targeting drug use concluded, "Coalitions are expensive to maintain and may not lend themselves to effective or well-implemented strategies" (Halfors, et al., 2002, as cited in Birkby). Yin et al. (1997) evaluated the CSAP Community Partnership Program and found that only eight of the 24 communities studied showed statistically significant results lower than comparison communities on at least one of six outcomes examined.

On the other hand, Birkby's review of the literature did identify a number of successful partnership collaborations. Berkowitz (2001), as reported by Birkby, finds that coalitions have achieved positive outcomes in the following areas: disability advocacy, education, health clinics, access to prenatal care, housing for the mentally ill, and physical exercise.

Wandersman and Florin (2003) identify successful outcomes for an arson prevention coalition in Detroit and the CINCH initiative in Norfolk, Virginia. Stevenson and Mitchell (2002)—as discussed by Wandersman and Florin—review the results of studies examining the effect of collaborative efforts targeting substance abuse and find that collaborative strategies targeting policy change appeared to be the most effective.

Birkby identifies the following reasons why it may be so difficult to evaluate the long-term effectiveness of partnerships and coalitions:

Coalitions are not well defined. Unique characteristics of each coalition make it difficult to replicate the initiative.
Extraneous variables can influence outcomes. Moreover, extraneous variables differ from community to community. They include policy changes, new government initiatives, and population changes. All of these can interact with each other as well as with the community initiative.
It is difficult to match the community with the partnership initiative with a similar community without such an initiative. Without such comparisons, however, it is difficult to attribute changes to the partnership.
It is difficult to draw conclusions across coalitions. They often differ in intended outcomes, or worse yet do not have the same access to good baseline data.
The long-term effects may not be measurable until years later. Many coalitions either do not measure intermediate outcomes or do not have well-articulated theory to link intermediate and long-term outcomes.
Coalitions may change in essential components. Political pressure or pressure from funding sources may change the coalitions structure or functioning.
Coalitions are multi-layered and complex. The complex nature of the coalition does not lend itself well to traditional evaluations.

Kaftarian and Yin (1997) discuss the methodological challenges of evaluating the outcomes of community-based partnerships, specifically partnerships for substance abuse prevention. When interventions target individuals, it may be possible to randomly assign some individuals to the interventions and the others to a control group. This, however, is not feasible when the intervention targets an entire community system: its institutions, norms, behaviors, attitudes, and policies. In the latter case, the community itself is the unit of analysis, not the individuals; in these instances, individuals when studied are seen as subunits, nested within the overall unit of analysis. Furthermore, the open systems nature of the partnerships and the complex nature of communities make it very difficult to ascribe change. In a special journal edition on this topic, Kaftarian and Yin present several approaches used to overcome these challenges. Although none of these were, or could be, experimental or quasi-experimental designs, they did each explore alternate explanations (or rival hypotheses) for the observed changes. Two of these methods included cross-community analysis in which the partnership community was matched with another community with similar demographic characteristics (Yin, et al., 1997).

Kubisch et al. (1998) describe features of comprehensive community initiatives for children and families that make them difficult to evaluate:

"Horizontal complexity. They work across multiple sectors (social, economic, physical, political, and others) simultaneously and aim for synergy among them.
Vertical complexity. They aim for change at the individual, family, community, organizational, and systems levels.
Community building. They aim for strengthened community capacity, enhanced social capital, an empowered neighborhood, and similar outcomes.
Contextual issues. They aim to incorporate external political, economic and other conditions into their framework, even though they may have little power to affect them.
Community responsiveness and flexibility over time. They are designed to be community-specific and to evolve in response to the dynamics of the neighborhood and the lessons being learned by the initiative.
Community saturation. They aim to reach all members of a community, and therefore individual residents cannot be randomly assigned to treatment and control groups for the purposes of assessing the [comprehensive community initiative] impact; finding equivalent comparison communities is also not feasible."

In a similar vein, Wandersman and Florin (2003) found fewer than expected community interventions (including but not limited to partnerships) that show the desired results. They recommend that future initiatives include "greater articulation of theory, increased sensitivity or measures, improved (or different) methods or designs, and expanded use of best practices."

Key outcomes of the partnership will be documented in subsequent phases of this substudy.

2.7 CONTEXTUAL CONDITIONS

« back to contents Assesing
Partnerships: An Example of Sequential Phases: Contextual Conditions and Rival Explanations

Examining contextual conditions (including partner and community capacity) and rival explanations are a critical component of the partnership evaluation. Two key contextual conditions include partnership capacity and community capacity.

Partnership Capacity. "Constructing effective partnerships among diverse organizations is hard work" (Reich, 2000). The reason organizations are willing to engage in partnership efforts is because of the firm belief that they create value. "To assure a sustainable collaboration, the value created must be useful to society, and value must flow to all core partners" (Reich, 2002).

Brinkerhoff (2002) terms a partnership's relationship outcomes as "value-added," which she describes as follows:

"Value-added may include qualitative or quantitative synergistic outcomes of the program itself (i.e., aspects of program performance that relate to advantages beyond what the actors could have independently produced), linkages with programs and actors, enhanced capacity and influence of individual partners, and other multiple effects such as program extensions and replication, new programs etc....Another element of the effectiveness and outcomes of the partnership is the extent to which individual partners meet their own objectives through the partnership."

She adds that value added is challenging to measure because it is difficult to attribute the observed changes to the partnership. She suggests that evaluating these effects of the partnering relationship is usually perception- or consensus-based, and is closely related to partner satisfaction.

Successful partnerships provide positive organizational returns to all partners and change and strengthen the partnership itself, thereby increasing trust and mutual respect among partners and more sharing of resources and project ownership (HUD, 2002). This section examines the enhanced capacity of the partnership, the partners, and the targeted community.

HUD's Office of University Partnerships (HUD, 2002) identifies four types of partnerships, the most effective of which is the capacity building model. The other three models are: 1) the paternalistic/theory testing model in which the university poses the questions and uses the community to test its theories; 2) the professional/expertise model in which either party can ask the questions but the university provides the answers; and 3) the resource model which is similar to the professional/expertise model but is somewhat more community-focused in that the university is available to help as needed "but does not set the agenda." The empowerment/capacity building model, on the other hand, "emphasizes the building of the power and capacity of local community organizations and residents to formulate and carry out their own planning, research, and implementation. College and university personnel work alongside, gaining as well from the collaboration (building their capacity to work with and learn from the community)."

Holland (2001) describes an effective academic institution as one that is "committed to direct interaction with external constituencies and communities through mutually beneficial exchange, exploration, and application of knowledge, expertise, and information. These interactions enrich and expand the learning and discovery functions of the academic institution while also enhancing community capacity." An important benefit of university engagement in partnership activities is the service learning of students (Greenberg, et al., 2003).

Rothwell (2004) defines organizational learning as " how organizations gain experience, reflect upon it and even anticipate new experience as a means of gaining competitive advantage." According to Peter Senge, author of The Fifth Discipline: The Art and Practice of the Learning Organization(as quoted by Rothwell), "The organizations that will truly excel in the future will be the organizations that discover how to tap people's commitment and capacity to learn at all levels of the organization." Senge's learning organization possesses five characteristics that encourage learning:

Systems thinking—the ability to see the whole as well as the parts;
Personal mastery—individuals become committed to their own lifelong learning;
Mental models—images that influence how individuals understand the world and how they take action;
Building shared vision—the ability to create a compelling vision that excites others to action and builds enthusiasm for the organization's goals and strategic objectives; and
Team learning—the ability of individuals to work and learn effectively together in groups.

Gouvis Roman et al. (2002) conducted an extensive literature review and conducted focus groups with community justice partnerships to determine those characteristics of community organizations that are important to the development of capacity to partner for community justice initiatives. They identified three key organizational characteristics: 1) leadership; 2) resources; and 3) orientation.

Community Capacity. Chaskin (2001), having reviewed the many definitions of community capacity, finds that each of these various definitions "places a different relative emphasis on various dimensions of community capacity. Some focus largely on organizations, others on affective connections and shared values, and still others on processes of participation and engagement." He concludes that taken together, the various definitions agree on the following aspects of community capacity. Community capacity includes: 1) the existence of resources (ranging from the skills of individuals to the strength of organizations to financial capital); 2) networks of relationships; 3) leadership; and 4) some type of mechanism for community members to engage in collective action and problem solving.

When discussing networks of relationships, Chaskin refers to social capital. The term "social capital" appears throughout the literature on community capacity, drawing on the seminal work on social capital by Bourdieu (1983), Coleman (1988), and Putnam (1993). Putnam (1996) defines social capital as the "Networks, norms, and trust that enable participants to act together more effectively to pursue shared objectives."

Boris (1999) in an examination of the role of nonprofit organizations reports on "the central role that formal and informal nonprofit organizations play in creating the glue that holds communities together and the avenues they provide for civic participation." He writes, "Nonprofit organizations, regardless of origin, create networks and relationships that connect people to each other and to institutions quite apart from the organization's primary purposes. Those relationships build social capital, the cooperative networks that permit individuals to work together for mutual goals."

A distinction can be made between horizontal social capital that essentially links community members with their peers, and vertical social capital that links community members with those with political, economic, and other power (Grootaert, 2003).

2.8 MEASURING PARTNER AND COMMUNITY CAPACITY

« back to contents

Partner Capacity. Many of the measures listed earlier in this document, if administered at different points in time, can be used to measure increased capacity of the partnership and its partner organizations. Rothwell (2004) has developed a set of self-assessment questions to measure institutional learning.

Community Capacity. Bartle (2003) proposes analyzing the strength, power, or capacity of a community by measuring change in the following features of the community: altruism, common values, communal services, communications, confidence, political and administrative context, information, intervention, leadership, networking, organization, political powers, skills, and wealth. He recommends that community members (not just those in power) assess whether there has been an increase in any of these dimensions. However, to prevent bias, he recommends the collection of complementary data (such as the number and type of communal services). This includes facilitator handouts designed for participatory measurement of the strength of each of the above dimensions. The first measure provides an estimate of strength. Both measures examine the current status and ask participants for a retrospective assessment of change over the past 12 months and the previous five years.

Gardner (1995) developed a collaborative assessment of capacity. The instrument is designed as a guide for county-level youth and family collaboratives. It covers 10 elements of collaborative capacity: governance and accountability, outcomes, financing, non-financial resources, community and parent ownership, staff and leadership development, program strategies, policy agenda development, organizational coherence, and addressing the equity issue.

Putnam (Hudson and Chapman, 2002) proposed a social capital questionnaire as a supplement to the 2002 Census Bureau's Current Population Survey (CPS).

Grootaert et al. (2003) have developed an instrument to measure social capital of communities in underdeveloped countries. Nevertheless, with some revision, some of the questions may be applicable to the MSPs. As Grootaert points out, the content and phrasing of questions will be not appropriate in all countries, and locally important questions may need to be added. The Social Capital Questionnaire collects data on six dimensions: 1) groups and networks; 2) trust and solidarity; 3) collective action and cooperation; 4) information and communication; 5) social cohesion and inclusion; and 6) empowerment and political action.

Healy (2003) reviewed the international literature to identify measures of social capital (quite a few instruments exist to examine social capital in developing countries). He concludes that "a single measure approach to social capital based on, for example, numbers of associations, membership rates or generalized trust offers a very limited means for measuring the extent of social capital." He includes examples and selections of questions on social capital from a number of international surveys. He recommends that the measurement of social capital be approached at a number of levels:

Standardized questions on trust, civic engagement, social support networks, etc., in large-scale household surveys;
Surveys of observed or reported human behavior;
Specific and contextual questions on relationships, attitudes, and behavior in community or organizational-specific surveys neighborhood, enterprise or school;
Case-study, qualitative, or action-based research, which seeks to explore the meaning and interpretation of social interaction in a particular situation or context; and
Randomized social experiments that seek to combine measurement with active policy intervention and "laboratory-simulated" conditions.

Bjornslov and Tinggaard Svendson (2003) examined existing measurement systems, and identified four dominant operational features of social capital measures: 1) the trust radius of a population as measured by the percentage of a population believing that people can be trusted; 2) the density of voluntary organizations in a given area, as measured by the number of organizations in which an average resident participates; 3) community members' perceptions of honesty and corruption; and 4) measures of economic freedom. They conclude that one may need to divide social capital into two dimensions: one dimension in which social capital refers to honesty and trust in both individuals and institutions and another dimension that refers to civic participation.

Gouvis Roman and Moore (2004) used the following data sources to measure social capital in several District of Columbia neighborhoods: secondary data on organizations in the community—including the National Center for Charitable Statistics database (http://nccs.urban.org), and interviews with representatives of community organizations.

Bullen and Onyx (1998) present a social capital instrument and practitioners guide used to measure social capital in five communities in New South Wales, Australia.

2.9 RIVAL EXPLANATIONS

« back to contents

During the primary data collection phase of this substudy, rival explanations will be examined in relation to the partnership. One of the initial areas that will be explored will be the impact of pre-existing partnerships on the current one. For example, did the partnership have parallel requirements and support a similar partnering effort? Did the pre-existing partnership contribute to accelerating or impeding the progress of implementation of the current grant?

2.10 LOOKING AHEAD

« back to contents

In looking ahead, the question of Why partnerships? becomes increasingly important. Overall, it will be critical to understand whether and how partnerships with STEM discipline faculty engagement helped to define and accomplish the MSP Program's math and science education goals and objectives. Have the partnerships created something greater than the sum of what the individual partners could have done?

The partnership as a whole brings more skills, resources, and diversity. A single organization also may be more vulnerable to unanticipated circumstances such as turnover or internal crises. In this situation, the partnership composed of multiple partners will have more resources to sustain itself through unexpected challenges. In principle, K-16 (vertical integration) ensures continuity of math and science at any given level and cross-district (horizontal) promotes collaboration that spans these areas.

REFERENCES

« back to contents

The Annenberg Foundation, "The Annenberg Challenge, Lessons and Reflections on Public School Reform," March 13, 2002, p. 57. http://www.annenbergfoundation.org/other/other_show.htm?doc_id=212527

AriÒo, A., J. de la Torre, and P. Smith Ring, "The Role of Trust in Inter-Organizational Alliances: Relational Quality and Partner Behavior," 1999, http://www.anderson.ucla.edu/research/ciber/wponline/pdf/99-22.pdf.

The Aspen Institute, "Measures for Community Research. Roundtable for Comprehensive Community Initiatives," undated, http://www.aspenmeasures.org/index.html.

Bartle, P., "Measuring the Strengthening of Communities for Community Development Participatory Methods of Measuring Empowerment," trainer's notes, Seattle Community Network, Seattle, WA, 2003, http://www.scn.org/cmp.

Baum, H.S., "How should we Evaluate Community Initiatives?" Journal of the American Planning Association, 2001, 67(2):147-158.

Bell-Elkins, J., "Case Study of a Successful Community-Campus Partnership: Changing the Environment through Collaboration," doctorial dissertation, University of Massachusetts, Boston, 2002, http://depts.Washington.edu/ccph/pdf_files/friendly%20principles2.pdf.

Bergstrom, A.R., T. Clark, T. Hogue, J. Iyechad et al., "Collaboration Framework—Addressing Community Capacity," National Network for Collaboration, Fargo, ND, 1995, http://crs.uvm.edu/nnco/collab/framework.html.

Berkowitz, B., "Studying the Outcomes of Community-Based Coalitions," American Journal of Community Psychology, included in Birkby's literature review, 2001, 29(2):213-227.

Birkby, B., "Community Coalitions: Questions, Controversy & Context," Prevention Evaluation Perspectives, 2003, (1)1, http://