Library

How Valid Are Self-Report Survey Data Obtained from School District Personnel?

"Most quantitative evaluation endeavors undertaken to assess the effectiveness of large scale educational reform efforts must rely on self-report data to some extent. For example, the analytical framework used to assess the impact of the Milwaukee Mathematics Partnership (MMP) on increasing student achievement in mathematics is largely dependent on data obtained from an online survey administered to all Milwaukee Public School District (MPS) employees at the K-8 level who have the potential to positively affect student achievement in mathematics. This survey was designed to measure various aspects of educators’ daily responsibilities that would likely be affected by MMP activities and in turn, affect student achievement in mathematics. However, the validity of self-report data is always suspect due to the possibility of response bias, which occurs if participants respond to items in a more socially appealing manner. In the current survey, because respondents are primarily reporting on the quality of their own work and that of their colleagues it is quite possible that the validity of our data is compromised by response bias. Moreover, if this is true then the results of any statistical tests conducted on these data may be biased because these results are wholly dependent on how well the variables have been measured.

Therefore, this year the evaluation team of the MMP made a concerted effort to assess the validity of our self-report data. Two approaches were taken to attain this goal: (1) A global approach that utilized all schools in the district that participated in the quantitative evaluation efforts; and (2) A case-study approach that only made use of schools that participated in the qualitative evaluation efforts. For the global approach, overall indicators of the quality and quantity of a school’s participation in MMP related activities obtained from the self-report survey data were compared to ratings obtained from the district level Mathematics Teaching Specialists (MTS) assigned to that school. For the case-study approach, an attempt was made to triangulate the self-report survey data obtained with other external criteria in a subset of schools targeted to participate in more in-depth intensive evaluation efforts. The primary purpose of this paper is to describe the methodology utilized and report the findings for both of these approaches to assess the validity of our data."

---

PDFs can be opened with the free Adobe Acrobat Reader.