Skip to main content


Welcome, the Hub connects all projects

MSP Conferences


Breakout Session: Findings from the MSP Management Information System (MIS)

Teacher Quality, Quantity and Diversity
2006 Math and Science Partnership Learning Network Conference
January 30 - 31, 2006
Washington, DC


CONTENTS
Using 2003-04 and 2004-05 school year data from the MSP Management Information System (MIS), Westat Research Associates Robyn Bell and Holly Bozeman describe how Comprehensive, Targeted and Institute Partnerships have addressed teacher quality, quantity and diversity. Specifically, they discuss project activities being implemented to enhance the teacher workforce.

Comprehensive Partnerships

Targeted and Comprehensive Partnerships
Robyn Bell
Research Associate, Westat
The MSP Management Information System (MIS)

Institute Partnerships
Holly Bozeman
Research Associate, Westat
The MSP Management Information System (MIS)

Group Discussion/Questions


An Important Caveat

The data and summaries in this document draw on selected preliminary findings from the MSP Management Information System (MIS) for the 2003-04 and 2004-05 school years. Because Westat has not yet validated the 2004-05 school year data included in this document, these data are subject to change.
Final reports on the data will be made available from Westat and will be posted on MSPnet.




TARGETED AND COMPREHENSIVE PARTNERSHIPS

Session Introduction

Robyn Bell
Research Associate, Westat
The MSP Management Information System (MIS)

In September of 2004, the National Science Foundation initiated the MSP Management Information System, which is a web-based data collection system. The MIS is designed to obtain annual information from each MSP-funded project that can be used to assess the implementation and impact of the overall MSP program and to monitor the progress of individual MSP awards.

During this breakout session we will use data from the MSP Management Information System (MIS) to describe how Comprehensive, Targeted, and Institute Partnerships have addressed teacher quality, quantity and diversity.

This is a small group, and you are encouraged to ask any questions you may have as we present these findings. The questions and insights that those of you working on the projects have can help us to think about future analyses and look at validation from the MSP demographic.

We are going to be discussing a lot of figures today, and it is important to note that the 2004-05 data that we are going to cover have not yet been validated, so they are subject to change at this point. We are going to begin our validation process in the very near future. At the end of the session, we are going to hand out some packets that detail our teacher quality, quantity and diversity findings, and we will let you look over those and ask us any questions that you might have.

We're going to start today by looking at the findings from the Comprehensive and Targeted Partnerships. Holly Bozeman is then going to discuss findings from the Institute surveys from last year.

Robyn Bell
Research Associate, Westat
The MSP Management Information System (MIS)

I'm going to start by talking about each of the Comprehensive and Targeted Partnership surveys to give you an overview of the different teacher quality, quantity and diversity elements that are found in each of the four types of survey data.

TQQD Elements in MIS Surveys for Comprehensive & Targeted Partnerships
  • Annual Survey for Partnership Projects
    TQQD activities in place during the previous school year
  • Annual IHE Participant Survey
    IHE faculty participation in preservice and inservice activities
  • Annual IHE Survey
    Demographics of students enrolled in MSP- supported preservice courses
  • Annual K-12 District Survey
    Demographics of teachers and students in participating K-12 schools

The first is the Annual Survey for Comprehensive and Targeted Partnerships. It is completed by the PI of the project or someone designated by the PI. Each project is asked to identify the different teacher quality, quantity and diversity activities that they have done in the previous year. There are three different types of activities: preservice recruitment, preservice preparation, and inservice retention and enhancement. Those of you familiar with this portion of the survey know that respondents select from a predetermined list of activities and tell us which activities are either in place or under development. They also have the opportunity to list other activities that may not be included in the survey.

The Annual IHE Participant Survey is completed by all faculty and administrators involved as the higher education partners, and it asks whether or not they participated in any preservice or inservice activities during the previous year. For participants with more than forty hours, we ask specifically which preservice and inservice activities they were involved in.

The Annual IHE Survey is completed by all core and supporting higher education partners, and through this survey we collect information about preservice courses in higher education that are revised or enhanced with MSP support. We also ask, for the courses that were offered in the previous year, the demographic information on the students who were enrolled in those courses.


The Annual IHE Survey is completed by all core and supporting higher education partners, and through this survey we collect information about preservice courses in higher education that are revised or enhanced with MSP support. We also ask, for the courses that were offered in the previous year, the demographic information on the students who were enrolled in those courses.

Finally, there is the Annual K-12 District Survey, completed by all core and supporting district partners. For this survey, with all of the schools that have a significant amount of participation, we collect information on the demographics of the teachers at that school and the teachers that participated for thirty hours or more in their MSP. We also collect information on the students enrolled at that school.

Participant Question: Who Is Included?

Q:
Are you reporting only those that would have "significant participation," as identified according to your metric?

A:
When we look today at the demographics of the teachers and the students at the schools, we have only collected information from the schools that have a significant amount of participation. Otherwise, most of the information today is coming from the Annual Survey for Partnership Projects, which is completed by all Comprehensive and Targeted Partnerships.

By looking at all of these surveys together, we can start to tell you what is being done to enhance teacher quality, quantity and diversity, who is involved in developing these activities and delivering them, and also who is ultimately benefiting from the teacher quality, quantity and diversity activities. As you'll see in the table above, not all of the projects are doing preservice recruitment and preservice preparation activities, but in 2004-05 there were a higher number of projects conducting these activities than there were in 2003-2004. And as you can see, all of the projects were conducting inservice retention/enhancement activities.

TQQD Activities Conducted in 2003-04 and 2004-05
TQQD Activity Type Number of projects conducting TQQD activities
TQQD activities
2003-04 (n=34) 2004-05 (n=40)
Preservice recruitment 25 (73.5%) 32 (80.0%)
Preservice preparation 23 (67.6%) 30 (75.0%)
Inservice retention/
enhancement
34 (100%) 40 (100%)

We are not going to take a look at all of these surveys in depth today, but this should give you an idea about the different elements of teacher quality, quantity, and diversity in each of these surveys tell you what is being done to enhance teacher quality, quantity and diversity, who is involved in developing these activities and delivering them, and also who is ultimately benefiting from the teacher quality, quantity and diversity activities. As you'll see in the table above, not all of the projects are doing preservice recruitment and preservice preparation activities, but in 2004-05 there were a higher number of projects conducting these activities than there were in 2003-2004. And as you can see, all of the projects were conducting inservice retention/enhancement activities.

Now we can look a little bit closer at the types of activities that the projects were doing. In terms of preservice recruitment activities conducted by MSPs to recruit undergraduate and graduate students into the teaching profession, the most common activities were actually targeted to STEM students at the IHEs. The most common strategy was to invite the undergraduate or graduate students to help at K-12 special events, followed by creating or providing opportunities for STEM students to tutor K-20 students.

TQQD Activities Conducted: Preservice Recruitment
  • Most common strategies
  • Invite STEM undergraduate/graduate students to help at K-12 special events
  • Create/provide opportunities for STEM undergraduate/graduate students to tutor K-20 students

In preservice preparation, we saw a change across both years in the activity emphasis (see above). With the MIS, we can do further cohort analysis to learn whether the change in emphasis is because of the new Cohort Three projects that were added, or whether, perhaps, Cohort One and Two projects were changing their emphasis in the activities that they were doing.

TQQD Activities Conducted: Preservice Recruitment
  • In 2003-04, the most common preservice preparation activity was providing opportunities for preservice students to gain classroom experience before student teaching.
  • In 2004-05, two of the most common activities designed to enhance preservice preparation involved the development or revision of preservice courses to align with national, state and/or local standards.

Turning to inservice retention and enhancement activities, the three most common strategies are listed above. In both areas, more than eighty percent of the projects were doing these three activities. We found that less than a third of the projects reported providing induction supports for new STEM teachers with MSP support, which is interesting because it is actually one of the strategies listed in the Cohort Two and Three solicitation for enhancing teacher quality, quantity and diversity.

TQQD Activities Conducted: Inservice Retention and Enhancement
  • Most common strategies
  • Conduct activities that develop and utilize teacher leaders
  • Conduct workshops with K-12 teachers that increase general content or pedagogical knowledge
  • Provide administrative supports for K-12 teachers
  • Less than a third of the projects reported providing induction supports for new STEM teachers

Participant Comments: Induction

Comment A:
You said that only 30% of the projects are involved with induction. We are finding in California that there is such a huge induction program at the K through 12 level that teachers are overwhelmed and overworked with just what they have to do to get certification, and it's difficult for us to work with that. So that might be one of the influences affecting that.

Comment B:
I work in a center where there are other subject matters taught also, and all of us are running up against this requirement this year, so I, too, haven't really worked with induction.

That provided an overview of the activities that are being conducted to enhance teacher quality, quantity and diversity. Now we turn to the individuals that were involved with the design and delivery of the activities.

Design and Delivery of TQQD Activities
  • IHE STEM faculty involvement
  • IHE Education faculty involvement
  • K-12 teacher involvement
  • Opportunities for collaboration
  • Are individuals from different partner institutions actually working together in the design and delivery
    of TQQD activities?

Across both years, we see that a majority of MSP activities designed to enhance teacher quality, quantity and diversity were conducted with input from IHE STEM faculty. In fact, in 2004-05, IHE STEM faculty were involved in the design and delivery of 76.3% of all of the teacher quality, quantity and diversity activities that were conducted that year. The solicitation states: "Mathematicians, scientists and engineers, particularly those who are faculty in higher education partner organizations, play a substantial role in Partnership projects. Their substantial intellectual engagement in these projects is one of the attributes that distinguishes the MSP program from other programs seeking to improve K-12 student outcomes in mathematics and in science."

This is very good evidence that the partnerships are, in fact, involving IHE STEM faculty, particularly with the teacher quality, quantity and diversity activities.

IHE Education faculty were also heavily involved in the development and delivery of teacher quality, quantity and diversity activities, but not quite to the same extent as the STEM faculty.

When we look at K-12 teacher involvement, about half of all of the inservice activities (not all of the teacher quality, quantity and diversity activities, but specifically the inservice activities) were conducted with input from the K- 12 teachers. Over a two-year period, there was a slight increase in the number of inservice activities being conducted with K-12 teacher input.

Participant Feedback: Nature of the Collaboration Among Multiple Partners
  • These findings suggest that there are opportunities for collaboration between participants from multiple partners. What the MIS doesn't collect is any quantitative or qualitative information on whether the participants are actually working together, or if the projects are just gaining input from each entity individually. So I'd like to take a minute here to get some of your input. Based on your own experiences, if we were to investigate this issue further, would we find that the individuals from multiple partners are actually working together to design and deliver these activities? Or are you finding that you are getting input from multiple participants, but they're not necessarily working together? - Robyn Bell

  • We're with the Partnership for Students Success in Science in the Bay Area in California, and our model for our summer institute involves higher ed and teachers co-developing the institute and co-delivering it, so we have very much cooperative input. However, during the school year inservices we are more likely to have teachers delivering that, with our higher ed partners participating to give expert input, but not necessarily that involved in development. - Nancy Thomas, President , Board of Trustees, Newark Unified School District; Partnership for Student Success in Science MSP

  • You would find that our partners are working together in co-construction, co-development, and co-delivery year-round in a variety of activities in a variety of districts. It's our model for working and collaborating across the various educational levels. - Sarah Mason, Project Manager and Researcher, University of Wisconsin; System-Wide Change for All Learners and Educators (SCALE) MSP

  • I would say that in our MSP, the level of collaboration is very high. Probably eighty or ninety percent of what we do has all of our partners working together. We hold regular meetings between our partners, between the colleges and the school districts, and with teachers themselves. We've brought back teachers now who are instructing at our summer institutes, and they have a say in how we are organizing everything. At the end of the day, obviously, we make the final choice, but it's very heavily influenced by every one of our partners. - Jeff Commaroto; SCOLLARCITY MSP

  • I think you're missing a category here. In our codevelopment and co-delivery model, we have significant involvement at the K-12 level from what the districts call a variety of things--sometimes they're called science experts, sometimes they're curriculum specialists. Is this something you're considering? I think it's a very important category for identifying the involvement of K-12 districts. We have found that the STEM faculty and the content specialists from the districts have a great deal of reciprocity in their learning. - Anon

  • I should have mentioned before that we are looking here at some of the higher levels of involvement. In the tables in the packets we'll be passing out later, you'll see there are categories for all types of participants and opportunities for collaboration. So we're collecting all of that information on other types of collaborations, but we're just not discussing it here today. - Robyn Bell

Toward the end of this session, if time allows, I'd like to discuss with you, based on your own project experience, whether fifty percent is about where you'd expect the number to be in terms of K-12 teachers being actively involved with the design and delivery of inservice activities, or whether it's higher than expected or lower than expected, based on your own projects.

Next, we looked at the extent to which multiple participant types were involved in the design and delivery of MSP teacher quality, quantity and diversity activities. And over the two-year period, we see that the partnerships were drawing more upon the expertise of multiple partners in the design and delivery of individual activities. We saw that the most opportunities for collaboration were with the inservice retention and enhancement activities between IHE STEM faculty and participants from K-12 partners. So again, we see an increase there over a two-year period in the percent of inservice activities that were designed and delivered with input from both IHE STEM faculty, and K-12 faculty and staff.

Demographics of MSP Teachers and Students

We've looked at activities and at the people who are designing and delivering those activities. I'd now like to look at the people that are being impacted. The graph above displays the demographics of the teachers and students in the MSP partner schools. In 2003-04, and in 2004-05, the teachers in the MSP schools with significant participation were predominantly White. However, when you look at the 2003-04 school year, the highest proportion of teachers in those schools were Hispanic. We also found that when we look at the MSP preservice courses that were being developed with MSP support, we found that the students enrolled in those courses were also predominantly White.

Participating Teachers

One of the things that we did find, in comparing the percent of all White and Hispanic teachers in the participating schools to the percent of teachers in those schools that were actively participating in the MSP (which meant thirty hours or more of participation), is that in both years there was a decrease in the percent of White teachers that were participating and an increase in the percent of Hispanic teachers participating. Since we wouldn't really expect to see these percentages changing very much, this could be an indication that the MSPs are targeting their services to and involving Hispanic teachers, perhaps in response to the demographic makeup of the student populations in the MSP schools.

With the MIS, we have school-level data and can look further into this information, and we can use the MIS to determine if this trend is actually occurring at those schools that have higher proportions of Hispanic students.

Participant Question: Hispanic Demographic Shift

Q:
Why the big shift from the percentage of Hispanic teachers? In the chart Participating Teachers it goes from 23.2% for all teachers in all schools in 2003-04 to 10.3% in 2004-05.

A:
We had a huge jump from 2003-04 to 2004-05 in "not reported," so I think the difference was just in the "not reported." It could also be, with the addition of the new Cohort Three projects, that the demographics changed based on the demographics of the schools that were added. A third explanation is that the data include a greater number of schools for the 2004-05 year.


Participant Question: % of Schools Meeting "Significant Participation" Requirement

Q:
What proportion of the projects meet the "significant participation" requirement in each year? Are we looking at about a third of all MSPs? I'm trying to determine what proportion of MSPs have that "significant participation."

A:
[Collective response from Westat] It varies by cohort by year. Overall, it is thirty-seven or thirty-eight percent. For Cohort One it was thirty-two percent, for Cohort Two it was 46%, and for Cohort Three, which was in its first year, it was 35%. This is the percent of schools partnering with MSP projects that meet the criteria for significant participation. It is the schools, not the projects that need to meet the criteria for significant participation. One big caveat is that this is without two big projects from which we are still collecting data, so these numbers may shift.


Participant Question: Attrition

Q:
Is there a reason why you're not collecting attrition rates?

A:
In the first collection year, we found that a lot of projects were unable to supply that information. Because the projects were having difficulty supplying that information, we felt it was better left to program evaluation.

This was the information we wanted to share with you on the Comprehensive and Targeted projects. I will now turn it over to Holly.





INSTITUTE PARTNERSHIPS

Holly Bozeman
Research Associate, Westat
The MSP Management Information System (MIS)

I'm going to discuss the Institute projects and the information that we have collected to date in the MIS. First, I want you to know that 2004-05 is the first year of data collection, so there are no comparisons that I can give you.

TQQD Elements in MIS Surveys for Institute Partnerships
  • Surveys:
    • Annual Survey for Institute Partnership Projects

    • Annual Survey for IHE Institute Faculty

    • Annual Survey for K-12 Institute Participant

  • Elements:
    • Professional development activities, IHE faculty involvement, and K-12 participant development as intellectual leaders

I will begin by giving you an idea of the types of surveys the MSP MIS administers on an annual basis. Specially, the kind of data each survey collects and the respondents of each survey. There are three surveys total. The first survey is completed by the PI or a person designated by the PI, the second survey is completed by IHE Institute faculty members, and the third by the actual participants. The data and the information about teacher quality, quantity and diversity that is collected relate to specific summer and academic school year professional development activities that the projects offer; the IHE faculty involvement and the areas in which they are involved; and K-12 participant development as intellectual leaders and accomplished practitioners. You all may be familiar with that, which is the purpose of these Institutes - to develop participants as intellectual leaders and accomplished practitioners to increase the quality, quantity, and diversity of the teacher workforce.

Institute Project Overview
  • Surveys:

    • 8 Institute projects
    • 6 Mathematics-focused Institutes
    • 2 Science-focused institutes
    • States:
      • Massachusetts, New Jersey, Pennsylvania, Virginia, Florida, Nebraska, Oregon, and Texas

Of the eight Institute projects, six are math-focused, two are science-focused, and they are operating in the states listed above. There is a lot of information that could be shared with you today, but I pulled out specific numbers to highlight. As was mentioned earlier, at the end of this session we will be handing out packets that offer more detailed tables so you can see the full range of information collected about these projects. Some of the highlights are listed below.

Annual Survey for Partnership Projects
Summer Professional Development Activities
  • 100% Provided courses for K-12 teachers that increased mathematical or science content
  • 87.5% Provided opportunities for participants to earn advanced degrees, certification, or graduate credit upon completion of the institute
  • 75% Conducted courses for K-12 teachers that increased pedagogical knowledge
  • 50% Provided opportunities for participants to get hands-on experiences during the summer institute
  • 12.5% Conduct courses for K-12 teachers that increase their ability to use assessment data to inform their teaching

The percentages for summer professional development activities were high in comparison to academic school year activities outlined below. The highest percentage of response concerning academic school year activities was 62.5%, and that was for conducting workshops or courses for K-12 teachers that increased content and/or pedagogical knowledge.

Annual Survey for Partnership Projects
Academic School Year Professional Development Activities
  • 62.5% Conducted workshops/courses for K-12 teachers that increased content and/or pedagogical knowledge
  • 37.5% Provided curriculum resources to participants after completion of the institute
  • 25% Facilitated online workshops and/or courses during the academic year for institute participants
  • 12% Provided adjunct positions for institute participants at the partner IHEs

We're not sure why these percentages are so low, but a cursory look at the pattern of responses indicates that two projects did not offer academic year activities. We'll be speaking with those projects for further information. Maybe we're not asking the right question; maybe they define their academic year activities in a different format. Since this data has not been validated, we'll have to learn more about this in due time.

Participant Question: Degrees and Certificates

Q:
On the second item under "Summer Professional Development Activities," what is the difference between advanced degrees, certification, and graduate credit? Do you have a breakdown of percentages in these three categories?

A:
Some projects are offering master's degree programs, some are offering academic credit. Regarding certification, we need more information from each project to learn more about what they're offering. We will also be collecting information about the breakdown in these categories. This survey is at the PI level and doesn't include that.

The second survey is to be completed by Institute IHE faculty, and the breakdown below offers an overview of what the faculty sample looks like.

Annual Survey for Partnership Projects
IHE Institute Faculty Characteristics
  • 102 Respondents
  • 52% had tenure status
  • 43% conducted research in an educational field
  • 39% conducted research in a STEM field
  • 63% instructed STEM courses
  • 29% instructed education courses

I find it interesting that there is such a large jump in the types of courses that they have taught, in primarily STEM education, compared to their research focus. We'll talk more about that later on, when we look at some of the activities provided by these faculty members. Faculty identified four primary areas that they are involved in, as outlined below. I've provided the two percentages to represent the high and the low. As you can see, IHE faculty members spent most of their time in the planning and development phase. And then, as demonstrated in the Annual Survey for Institute Partnership Projects, the IHE faculty are showing the same trend in terms of a lower percentage of academic year activities.

Annual Survey for IHE Institute Faculty
Area of Involvement and Intensity between September 2004 and August 2005
  • Four areas of involvement
    • Institute Planning and Development (85.3%)*
    • Summer Institute Activities
    • Academic Year Activities (52%)*
    • Management and/or Other MSP-related Activities
  • 93 faculty members spent more than 40 hours of institute activities
    • * 54 spent more than 200 hours

Annual Survey for IHE Institute Faculty
Summer Institute Activities
  • >50% taught courses that increased participants' content and pedagogical knowledge of mathematics and science
  • 45.2% taught courses for participants that increase abilities to develop new and challenging curriculum materials
  • 31.2% taught courses for participants that improve leadership skills and strategies
  • 21.5% provided mentoring for participants on professional development strategies and other leadership responsibilities
  • 6.5% taught courses for school administrators
Only IHE Institute participants that spent more than 40 hours of their Institute activity between September and August 2004-05 were included in this analysis

For the IHE faculty summer institute activities, I've picked out a few key activities to share with you today. And below are the academic year activities. As you can see, the percentages during the academic year are lower, with the highest overall at 31.2%, which is the percentage of faculty who remained on call for classroom teachers during the school year.

Annual Survey for IHE Institute Faculty
Academic School Year Institute Activities
  • 31.2% remained "on call" for classroom teachers
  • 26.9% engaged their IHE department in activities to improve K-12 instruction and learning
  • 18.3% increased collaborative activities with regional school systems to improve K-12 instruction and learning
  • 9.7% provided instruction during the academic year for teacher leaders related to their leadership responsibilities
Only IHE Institute participants that spent more than 40 hours of their Institute activity between September and August 2004-05 were included in this analysis

It will be interesting to see if, over the duration of this project, the academic year activities increase in the percentage of responses.

I noted earlier the distinction between the research focus of the involved IHE faculty and their instructional focus. We found some interesting information, described below.

Annual Survey for IHE Institute Faculty
Activity Distinction Between Faculty Research and Instruction Focus
  • Summer and academic school year activity percentages were relatively similar for faculty who researched in both STEM and education fields
  • 57.6% STEM faculty instructors vs. 25.9% education faculty instructors taught courses during the summer institute to increase participants' abilities to develop new and challenging curriculum materials
  • 40.7% education faculty instructors vs. 27.1% of STEM faculty instructors remained "on call" for classroom teachers during the academic year

While summer and academic school year activity percentages were similar for faculty who researched in both STEM and education fields, 57.6% of STEM faculty versus 25.9% of education faculty taught courses during the summer institute, while during the academic year the percentages are almost reversed when we look at faculty who remained on call for classroom teachers.

We would like to gain an understanding from the people who work with IHE faculty about the distinctions between STEM and Education faculty instructors' involvement with the Institute partnerships. In terms of professional development activities that the STEM and Education faculty are involved in, are these distinctions something that we should look into further?

Participant Response: STEM and Education Faculty Priorities and Schedules

Comment A:
You said that only 30% of the projects are involved with induction. We are finding in California that there is such a huge induction program at the K through 12 level that teachers are overwhelmed and overworked with just what they have to do to get certification, and it's difficult for us to work with that. So that might be one of the influences affecting that.

Comment B:
I work in a center where there are other subject matters taught also, and all of us are running up against this requirement this year, so I, too, haven't really worked with induction.

I can speak for the University of Wisconsin. If I had to make up a scenario, I would make up something just like that. The reason behind it would be that the STEM faculty have more latitude to choose what courses they're going to teach during the summer than they do during the school year, when they have responsibilities to their department. The Education faculty instructors, at least at the University of Wisconsin, have a great deal of their instructional activity targeted towards people who are already educators who are out in the field. So they teach a lot during the school year and not as much during the summer.

I also think the "on call" is a similar reflection of that, that Education faculty are responsive to their students during the school year and that STEM faculty have to return to their responsibilities for the students who are in their particular field.
- Sarah Mason, Project Manager and Researcher, University of Wisconsin; System-Wide Change for All Learners and Educators (SCALE) MSP

There is one other survey that I did not mention initially, the Initial Survey for K-12 Participants. The survey is administered to all of the participants at the onset of their Institute involvement. It will be a onetime survey for all participants, used to collect baseline information. As a result, we know the following about the K-12 participants in the Institute projects.

Initial Survey for K-12 Participants
Demographic Characteristics and Academic Positions
  • 461 participants total
    • 386 mathematics Institute participants
    • 74 science Institute participants

  • 69.8% female, 88.3% not Hispanic or Latino, and 78.1% White

  • 55.1% have been involved with school reform and enhancement activities previously

I find it interesting that 55.1% have been involved in school reform and enhancement activities previously. It shows that in these Institute projects they are reaching a near equal amount of participants who are both new to and experienced with school reform initiatives. We also know about the primary academic positions of each participant. The majority are teachers, with a few administrators, such as principals, vice-principals, and department heads, and a couple of specialists, meaning curriculum specialists or content specialists. Most of the administrators and specialists are affiliated with math Institute projects.

Initial Survey for K-12 Participants
Primary Academic Positions of Participants
  • 28.6% Elementary school teachers
  • 24.7% Middle school teachers
  • 25.4% High School teachers
  • 11.2% Administrators
  • 5.9% Specialists
  • 3.9% Others

We also surveyed the degrees earned prior to institute enrollment, and can see that for the majority, their degree was in education.

Initial Survey for K-12 Participants
Degrees Earned Prior to Institute Enrollment
  • Degree Types
    • 50% have a master's degree
    • <14% have a Ph.D/Ed.D, professional degree, specialist degree, or other
  • Degree Fields
    • All projects: 73.1% Education, 25.4% Mathematics, and 15.2% Science
    • Science focus: 45.3% earned a degree in a science field
    • Mathematics focus: 29.3%

When we asked in the survey about their leadership roles prior to institute enrollment, the most common response was that they had served on a curriculum committee of some sort. It will be interesting to see, when we administer a follow-up survey, a survey that I will talk about in a few minutes, if the least common responses to this question will increase in numbers. Will they increasingly become representatives or officers for national professional organizations, or serve on business/industry task forces in their local areas?

Initial Survey for K-12 Participants
Leadership Roles Prior to Institute Enrollment
  • Most Common Response
    • 56.8% served on a curriculum committee
  • Least Common Responses
    • 5.6% served as committee representative for national professional organization
    • 3% served as an officer in a local professional organization for science and/or math
    • 2.8% served on business/industry math or science task force in local area

  • Initial Survey for K-12 Participants
    Professional Development Activities
    • Overall, majority of activities received 60 or more percent of responses, except for two
      • 43.4% developed skills in working with parents, school boards, or others outside of the school
      • 47.7% learned how to use data and statistics
    • Learned content knowledge not related to mathematics or science
      • 91.7% elementary, 63.2% middle school, 59.8% high school
    • Used technology in instruction
      • 69.7% elementary, 76.3% middle school, 92.3% high school

    We also surveyed their professional development activities prior to institute enrollment, and asked how the participants had been recruited or designated to attend the institutes.

    Initial Survey for K-12 Participants
    Designation/Recruitment of Institute Participants
    • 40.8% noticed an open invitation and decided to apply
    • 48.9% were recruited by a:
      • School principal
      • Leadership council/curricular committee
      • Other administrator or specialist
    • 3.9% were encouraged by a fellow teacher to participate

    The fact that 3.9% were encouraged by fellow teachers to participate is encouraging. We can be hopeful that this small number may increase over the years as teachers who graduate from the institutes become increasingly involved in leadership roles and reform initiatives.

    Now for the Annual Survey for K-12 Institute Participants. This is the survey that will be administered annually to follow up on the progress of participants, to find out what they are actually doing, their professional status, their continued professional development, and so on. We will want to follow up on the information that we collected in the baseline survey. Are they changing their jobs? Are they teaching different classes? The survey will debut in the spring of 2006.

    Annual Survey for K-12 Institute Participants
    • Professional status
    • Institute participation
    • Continued professional development
    • Professional community building
      • 25% of projects established regular, organized meetings of teacher leaders within K-12 districts during the academic school year
      • 12.5% of projects established STEM in-person or online learning communities or study groups during the academic school year
    • Dissemination of Institute information

    Participant Question: Teacher Sample?

    Q:
    How are you going to do that? Are you going to use samples of teachers from the Institute projects?

    A:
    We will be using the same teachers from the initial baseline survey - 461 teachers involved in Institute projects.

    Generally, we want to monitor their institute participation, particularly any activities that facilitated or hindered their responsibilities as a teacher leader. We want to know how they are continuing to use professional development outside of summer institutes. Are they attending conferences, are they taking classes on their own, are they getting other degrees? We also would like to find out more about what kind of professional community they are building as a result of their participation in the institute. Are they keeping in contact with the other teacher leaders that they met through the institute? Are the partnerships between the IHE and the teacher leaders continuing to evolve? Why are they meeting, how often are they meeting, and how may people are involved?

    Out of all of the projects, 25% said that they will establish regular, organized meetings of teacher leaders within K-12 districts during the academic school year. And 12.5% of projects will establish STEM in-person or online learning communities or study groups during the academic school year. Those are some activities that the projects are doing for teachers to facilitate community building, but we want to know exactly what the teachers are doing on their own.

    The last item deals with dissemination of institute information. Are they giving conference presentations, are they publishing, are they performing demonstration teaching with other teachers in their district, state, or outside their state?

    Overall, all of this information will help us understand the process of becoming an accomplished intellectual teacher leader as the result of participation in a summer institute.

    Participant Question: Comparative Data

    Q:
    These statistics that you collected, did you have a general pool or national pool to compare with?

    A:
    That's a very good question, and we're looking into that right now. We can't make any general or value statements about the type of information that we're getting from this survey at this point because we don't have comparison data, but we are going to look into that.





    GROUP DISCUSSION

    Discussion
    What questions about teacher quality,
    quantity and diversity would you like to
    see answered?

    At this point in the session, the Westat staff handed out two information packets:
    • Teacher Quality, Quantity and Diversity (TQQD) Findings from the MSP Management Information System (MIS), Comprehensive and Targeted Partnerships
    • Teacher Quality, Quantity and Diversity (TQQD) Findings from the MSP Management Information System (MIS), Institute Partnerships

    The tables in these packets summarize selected preliminary findings from the MSP Management Information System (MIS) for the 2003-04 and 2004- 05 school years. Because Westat has not yet validated the 2004-05 school year data included in these packets, these data are subject to change. If you are interested in these preliminary findings, and/ or in the final data, go to the Westat web site or to MSPnet:

    www.westat.com

    www.mspnet.org

    Impact of STEM
    Faculty Involvement on Teachers

    • I don't know if you can answer this question, but at the beginning of this conference there was a slide that shows that there are 900 STEM faculty involved in the MSPs, of which about 300 or so are new. I noticed that you have characteristics of the participants, and you have characteristics of the K-12 participants. Do you have data on the differential effect of the STEM faculty involvement and whether or not it made any difference in the lives of K-12 participants?
      - Anon

    • I wasn't here earlier today to hear that 900 figure you are talking about. But we just collect information about the characteristics of the participants, and you have characteristics of the IHE faculty members who are involved in these projects. We have not made any comparisons or analyzed the impact that they have had.
      - Holly Bozeman, Research Associate, Westat, MSP Management Information System (MIS)

    • Based on the fact that you've got two data sets, I would be interested to know to what extent the teachers, as the result of their interactions with STEM faculty, have a different kind of experience than they would have had that interaction been absent.
      - Anon

    • That is not a question that a system like this is addressing. But there is a RETA that is addressing that, with Xiaodong Zhang as PI (The Effect of STEM Faculty Engagement in MSP: A Longitudinal Study), and the purpose of the RETA is to take a specific look at what is the value-added of having IHE faculty involved with K- 12.
      - Joy Frechtling, Westat; Vertically Integrated Partnerships K-16 MSP

    • But what I am wondering is if there could be a question asked in the survey, or is the survey already set? - Anon

    • There is no teacher segment in the Comprehensive and Targeted surveys, the only teacher input comes in the Institute surveys. I think that's a wonderful question, but I think it's best addressed through a different kind of data collection than what we've been doing on this.
      - Joy Frechtling

    Do the Data Reflect the Original Plan for the Institute MSPs?

    • STEM Faculty Involvement and Entering Teacher Characteristics

      • May I throw out another question? Is anyone here from an Institute Partnership? I'm trying to look at these data and raise two questions about what they mean for the image or the plan for the MSP, and my questions relate mainly to the Institute Partnerships. I think, according to the plan for these, there is supposed to be this rich engagement with STEM faculty; that is part of it. And then, the other part of it is that participants who are picked to come to these institutes are supposed to be sort of at the higher end of the scale of the teaching profession, so that they would have some prerequisites that suggest that they would be equipped to take on teacherleader- type roles. So going to the data that we have here, to the findings that show relatively little reports (I say "reports" because I don't know what's going on) of STEM disciplinary faculty engagement during the academic year, you explained why that might be. And I certainly accept that.

        But is that true to the MSP model if, during the academic year, the responsibility for carrying the water shifts more to the Education faculty? I guess my question there is, is what's being implemented true to what the theory was?
        And the second question, which is even more difficult to answer, is: Looking at the characteristics of the teachers who are participating in these institutes, does that match the image of the teachers on the upper end of the scale who were thought to bring some potential for leadership?
        - Joy Frechtling, Westat; Vertically Integrated Partnerships K-16 MSP
    • Regarding Faculty Involvement

      • With the faculty question, I think you have a tension between faculty members' responsibility to their own institution and their field and their career. And if they are a STEM faculty member, that is what they are going to prioritize. Often, their venture into the MSP model is new for them. They're not necessarily going to have the time to devote to it, and they won't immediately be switching their mission. With the Education faculty, however, their mission is to educate educators and to participate in that more fully. So I think in some ways, the fact that you've gotten the numbers that you have in your institutes of STEM faculty involvement is quite significant. I guess what I would question is, why aren't there more Education faculty involved?
        - Sarah Mason, Project Manager and Researcher, University of Wisconsin; System-Wide Change for All Learners and Educators (SCALE) MSP

      • For some of the STEM faculty in an institution of higher education, that is not part of the ordinary mission, to participate in teacher training. It depends on the higher education institution and how they support the operations of the institute, and whether they communicate that it is part of the faculty's responsibilities, and whether the participation counts for part of the faculty's responsibilities. Then, for example, teaching in the institute amounts to teaching an elective course, which offers very good encouragement.
        - Hai-Lung Dai, University of Pennsylvania; University of Pennsylvania Science Teachers Institute MSP

      • I wonder if the STEM faculty not participating during the school year has the same effect that that type of lull has on students over summer vacation. We know that kids lose ground academically over the summer when they're not in school. I wonder if absence of STEM faculty during the academic year has the same sort of impact on the content knowledge of teachers.
        - Jim Hamos, Program Director, National Science Foundation, Math and Science Partnership
    • Regarding Entry Level of Teachers

      • The other thing I want to comment on is your [Frechtling's] premise regarding the teachers. I think it depends on the mission of the institute. For example, in our case we actually target the needs of teachers with inadequate content preparation to begin with. Certainly, we have teachers who aspire to earn a higher degree, like a master's degree, and they had good content preparation in the past. But we also have a lot of teachers who feel that they don't have sufficient content preparation and they would like to improve that. In fact, many of them went to a remedial teacher training session and then came into the institute.
        - Hai-Lung Dai, University of Pennsylvania; University of Pennsylvania Science Teachers Institute MSP

      • Is your goal, then, to make them adequate teachers or to make them teacher leaders?
        - Anon

      • Both.
        - Hai-Lung Dai

      • Looking back at the data, what about the fact that forty percent of the teachers came to the institutes through an open invitation and decided to apply? They may take the courses and get the credits, but it doesn't have the embodiment of the disciplines or the sanction of the administration.
        - Jim Hamos, Program Director, National Science Foundation, Math and Science Partnership
    • Staying True to the Original Model

      • Jim, could you just comment - is this true to the model that NSF expected?
        - Joy Frechtling, Westat; Vertically Integrated Partnerships K-16 MSP

      • The model left it open. The model didn't say the STEM faculty had to work year-round, or that you should pick the best teachers you can identify because they will make the best teacher leaders. The model says that we do understand that teacher professional development and teacher leadership development shouldn't happen shortterm in a single workshop over the summer. It should be developed in the long term, and you should make sure that there is follow-up afterwards. How there are mixes and matches of those things, I think is interesting. Each of the Institute projects, just like each of the other partnerships, will find their own way within that language. What they deliver will be interesting, and we'll get data regarding the outcomes.
        - Jim Hamos, Program Director, National Science Foundation, Math and Science Partnership

    Teacher Surveys for Other MSPs

    • One thing that is interesting in terms of the Institute projects is that because we hope to develop teacher leaders, we have the teacher filling out individual surveys. That has not happened at all for the other types of MSP projects. These surveys are available. Would it be useful to gather this data from other group subsets - information about who you are and what you do? A lot of Targeted and Comprehensive MSPs are also doing things around teacher leadership. Would they want to use these teacher surveys?
      - Jim Hamos, Program Director, National Science Foundation, Math and Science Partnership

    • The surveys are available, but you might have to customize parts of it if they have questions that they think are specific to what is happening within their teacher leadership efforts.
      - Gary Silverstein, Westat; MSP MIS

    • We are doing a teacher survey, but instead of doing it once a year, because they are so difficult to manage, we are doing it twice during the project. It would have been nice to have gotten one that was already developed.
      - Nancy Thomas, President , Board of Trustees, Newark Unified School District; Partnership for Student Success in Science MSP

    Means of Sharing Strategies and Information

    • Example: Teacher Participation Rates

      • I'd like to know more about the percentage of schools that are participating at the level of thirty hours and thirty percent of the teachers. I wonder how many projects are in a situation like we are, where in the third year of the project we realize that because you can't force teachers to come to the summer institute or after-school activities, we're having trouble getting the number of hours for each teacher that we intended to get. We were able to negotiate with our administrators to set aside staff development days so that we could get that participation up, and it has worked really well. Is there anything on the survey or any way that projects can share information about the difficulties they may be having in teacher participation rates?
        - Nancy Thomas, President , Board of Trustees, Newark Unified School District; Partnership for Student Success in Science MSP

      • We went through the same thing in terms of not having access to teacher time. Right now we're in a transition model where, during the academic year we're having after-school sessions, but what we're finding is that the summer is the time when we're having the most impact because we're having more access to time, so we did see a shift. We also changed our model and had an increase in the number of participating schools. But where we've had the most impact is in one district that has implemented a math curriculum guide. In other districts we've had an increase in the number of school sites through the professional development. Another thing we had was a tremendous amount of district support in terms of funding and institutional support.
        - VanAn Tranchi, Senior Administrative Analyst, University of California, Irvine; Faculty Outreach Collaboration Uniting Scientists, Students and Schools
    • Identifying Targets for Further Information

      • I think what we've found is that word of mouth and quality has caused us to increase, but you want to reach those levels sooner than the third year in the project. So there's nothing like this in your survey?
        - Nancy Thomas

      • No, we don't get specifically to what you are asking about, but it's a monitoring system, and we get the first cut at that. We can look across a project over two years and three years and see where there are jumps in participation rates, both at the project level and at the district level. And if one wanted to, one could go back to those districts and start to collect stories in a targeted way concerning what brought about that increase. Or if the numbers are relatively flat, was that their intention or was there reason they weren't able to bring those numbers up? So it helps you focus where you start asking those questions.
        - Gary Silverstein, Westat; MSP MIS

    Further Breakdown of Teacher Science Degrees

    • I have a question about the degree fields [see "Degrees Earned Prior to Institute Enrollment," page 13] You show that 45.3% of science teachers earned a degree in a science field and 29.3% earned a degree in mathematics. That's an indication of their content preparation. I think in the science category it would be interesting to know what is meant. "Science" could mean chemistry, biology, physics; everything is just lumped together. So does someone who is teaching physics have a physics degree, or do they just have a biology degree? It would be interesting to know the type of degree field in comparison with what they are teaching.
      - Hai-Lung Dai, University of Pennsylvania; University of Pennsylvania Science Teachers Institute MSP

    Racial Demographics

    • On an earlier slide you pointed out the White and Hispanic demographics of participating teachers Participating Teachers. Is there a reason you didn't include African Americans?
      - Anon

    • I was just trying to show the White and Hispanic demographics because that's where we saw a shift . We saw a decrease in White teachers participating and an increase in Hispanic teachers. In all of the other race categories, we didn't see any change. If you look in the information packet for Targeted and Comprehensive MSPs, you'll see the extended demographic information.
      - Robyn Bell, Research Associate, Westat, MSP Management Information System (MIS)