Opinions Worth Debating

courtesy of the Answer Sheet:
http://www.washingtonpost.com/blogs/answer-sheet/wp/2013/12/03/a-pisa- contradiction/

The current debate regarding interpretation of recently released results of the Program for International Student Assessment (PISA) implies that we need to choose between two interpretations: 1. The educational sky isn't falling. Although US students have never done well on international tests, as a nation we have made remarkable economic progress and remain the strongest most innovative economy; or 2. PISA results should be a wake up call. PISA assesses important 21st century skills. Our students' abilities remain stagnant while other countries are racing ahead.

Could these both be valid claims?

Critique of overreaching evidentiary claims is essential, but the results of the debate may not move us forward toward effective solutions.

I wrote more on this topic for a Huffington Post article. PISA Results: A Chicken Little Moment:
http://www.huffingtonpost.com/arthur-camins/pisa-results-a-chicken-li_ b_4404925.html

Arthur H. Camins

Arthur, thanks for this post and the link to your article. I found it very helpful and thoughtful. I was glad to read your formulation of "equity" and "focus" problems. When I read the rhetoric about the tests and their implications for policy, I can't help remembering studies like "Poverty and education: the way forward" , or papers like David Berliner's "Our Impoverished View of Educational Reform" (both in MSPnet's library). So equity is an indispensable issue, which cannot be addressed through schools (though schools can be one piece of a solution).
But schools can take more of a lead on "focus" as you define it, it seems to me -- when policy does not prevent that.

I like the question, and the first two responses from Brian and Arthur. For other views, I find that Marc Tucker's commentaries on education -- including his thoughts about PISA -- are reliably intelligent and informative.

Because I think that the No Child Left Behind Act institutionalized many of the wrong approaches to improving education, albeit in a bi-partisan manner (gee thanks, Ted Kennedy) -- remember that K-8 science education was pushed aside, for one thing -- I am skeptical of any future strong role for the U.S. Dept. of Education. The federal government is too blunt an instrument to trust in classrooms beyond a certain point. So, yes, PISA points to national needs in education, some of which -- like dramatic inequality in the U.S. among students and their families, or access to early childhood education -- are not really about K-12 schools. Yet we need to be cautious about prescriptive federal laws, especially in this highly partisan environment. How can our STEM education community encourage changes in the structure of U.S. schools, such as providing principals and teachers with more time for PD and work with colleagues? I wish I knew the answer.

Responding to Brian and Andy:

One of the threats on the horizon for science education is a rush at the state level to put consequential science tests in place to assess the NGSS, not so much because science or engineering is valued, but because of requirements of new "value-added" teacher evaluation systems. So, we need push back at the state level with respect to test-based teacher evaluation systems to give teachers time to adjust curriculum and instruction and to revise NGSS as it is tried out in practice. More on that here: http://www.arthurcamins.com/wp-content/uploads/2013/04/NGSS_Wave-or-Ri pple2.pdf

Arthur H. Camins

It is worth noting that that the term the "United States of America" is something of a misnomer when considering education and needs to be kept at the forefront of any consideration of international studies such as PISA. With the word "education" not being present in the Constitution, the states were given plenary power over education through the 10th Amendment. This has given rise to a diverse range of educational practices across the states (for example, just look at how the science and mathematics graduation requirements, teacher certification requirements etc. vary across the states).
To put this in perspective, imagine reporting an average value for a "United States of Europe" (or Asia)- the result would be meaningless given the vastly different education systems that would be somewhat arbitrarily placed together. This is precisely what is being done by producing an average for the USA. As several commentators have noted the average values can result from widely disparate input data or outliers (controlling for poverty, for example, produces a completely different view of the results).

http://www.oecd.org/pisa/keyfindings/PISA2012_US%20report_ebook%28eng% 29.pdf
See page 100 for how well a few countries did on the PISA problems.
Shanghai had only one question where less than 40% of their students got it right, South Korea had eight, the US over 20.

Hi, Richard,
Thanks for this post, which adds some useful details to the discussion. What for you are the most important implications of this data?

The Chinese have two province wide exams, one at the end of ninth grade and one at the end of high school. There was a talk at ICME-12 on the ninth grade exams in a few different areas. Here is a link to it:
http://www.icme12.org/upload/submission/2034_F.pdf
Different exams are given in each area, and are developed locally. Shanghai is one of the areas, and according to the author, a math educator at East China Normal University, Shanghai's program is the most conservative of the areas studied in this paper. Some sample problems are given, and comments on the different areas are mentioned. Here is part of the conclusion: The tradition of mathematics education in China stress solid foundation of
basic knowledge and basic skills. It is believed that creative abilities and individual development is based on a sound mastery of basic knowledge and skills.
PISA does not stress basic knowledge and basic skills at any more than a low level. Shanghai shows that with a solid knowledge of them, and problems which encourage thinking and reasoning, it is possible for a much higher percent of students to do well than we think is possible.

Form the pdf of the cited paper

"It is believed that creative abilities and individual development is based on a sound mastery of basic
knowledge and skills."

The key word here is "believed". There is no causal evidence for such a belief.

Likewise

"a sound mastery of basic mathematical knowledge and skills could be a possible reason for Shanghai students' good performance in PISA mathematics 2009."

The key word here is "could." Such a casual connection is purely speculative in this paper.

These are interesting observations to me. Has any one else seen statements that have no supporting evidence? Are there any statements at all in the PISA which are causal? Are there any statments which are highly correlated?

I have only read the summary, so I apologize if these questions are overly simplistic.

I often hear mathematicians, parents, legislators, and others generalizing from personal experience to far larger populations. This is what I call the autobiographical theory of education. Each one feels her/his experience is "evidence".

I am amazed that state are phasing in Assessment Consortium assessments for CCSSM almost before they have finished phasing in the materials allegedly aligned to the CCSSM. Surely the teachers I've worked with (N about 150) are very much concerned about that aspect of the CCSSM. Folks can hardly believe that they aren't "teaching to the (small number of released items from) the proposed tests."

Here is my favorite exchange on a released PARCC item.

Item shows a figure which is apparently a rectangle with one side labeled 3 and one side labeled 5. It asks for the length of the rectangle.

Mathematician to the math specialist discussing this on behalf of a state Dept of Ed. Would PARCC accept 3 as a correct answer?

Specialist to mathematician: Huh?

Me to specialist. A rectangle can have four sides of equal length. Does PARCC assume that "length of a rectangle" is the length of the longer side if one side is indeed longer?

Specialist: Please direct your concerns directly to PARCC's website.

me: Thanks

for those who read Teachers College Record:

http://www.tcrecord.org/Content.asp?ContentID=17371

OECD's PISA: A tale of flaws and hubris.

Yong Zhao, from the University of Oregon, has a series of posts analyzing some of the more widely-cited PISA results. In the 4th of his posts, he explores how sampling issues have dramatically distorted the way that cross-nation comparisons with Shanghai have been made. His focus in these posts has been on several claims being made, which are not supported by the data, yet are being used to argue for education policy changes which are currently in fashion in some circles. What do you think?
http://nepc.colorado.edu/blog/how-does-pisa-put-world-risk
(this blog post has links to 3 earlier ones)
P.S. He is not afraid to state opinions! But his analyses are often very compelling.

Although I have only skimmed a number of Yong Zhao's posts, I agree that the U.S. would be ill-served by adopting a more authoritarian education system. NCLB has already done enough damage!

PISA is imperfect, yes, but can it be useful nonetheless? When PISA reports that there is a greater correlation in the U.S. between poverty and test scores than in many other nations isn't that information that fits well with research by Linda Darling-Hammond and many others? PISA suggests American teachers are not well paid compared to many nations; is that wrong? When 26% of U.S. students don't reach Level 2 (out of 6) on PISA's math test, should we simply complain about poor sampling in Shanghai and excessive test prep skewing their results, or should we ask what we can do to improve U.S. math education? Yong Zhao may identify some real problems with PISA -- but what does he recommend we do to improve American education?

I think it's worth mulling over Zhao's points, for a couple of reasons. First, he is very well acquainted with the Chinese system, including that of Shanghai. Second, on the basis of this knowledge, as well as broader info about the actual implementation of PISA, he raises important critiques about the validity of the scores-- based on the research of Kreiner and others, and very well summarized in William Stewart's piece in the Times Education Supplement (cited in http://zhaolearning.com/2014/03/23/how-does-pisa-put-the-world-at-risk -part-3-creating-illusory-models-of-excellence/).
I agree that even a very flawed data set can be used to ask important questions, such as those you raise. It is because those questions are not very prominent in the public discourse spurred by the PISA scores that dissenting voices need to be heard.
After all, it is highly unrealistic, given the complexities of the United States economic, justice, and education systems, to seek for a panacea. Such an illusory quest tends to mean that, while policy makers pursue the latest thing, other significant problems are demoted, and removed from the model that is being addressed. This is one contribution to the problems you allude to, which remain problems because schools are embedded in society, and student interventions are therefore limited in their capacity to overcome the consequences of that "situatedness."