MSPnet Academy Discussions

From a practitioner's point of view, ACHIEVE must quickly define itself in terms of 1)what has gone before and 2) what it hope to see happen in K-12 science education. With 50 states operating as 50 fiefdoms funded in many different ways, coherency of standards must translate into a coherency or rigorous outcomes that are respected by both colleges/univeristies and the states' K-12 education departments. I applaud the effort that ACHIEVE represents, especially its possible to international science education. I hope this effort will be a sustained effort (10 years or more) and not just a short-lived "reform" that stirs the pot but produces no fundamental improvements in our science learning. Onward!

I found the presentation today quite interesting -like all such efforts in the past, it is at an intersection of education policy and politics. In understanding both the origins and the possible fate of the Common Core, it might be helpful to know what Achieve is. Who backs it? Who funds it? What does it do to advocate for the adoption of the standards? How is it related to the Federal Government? How is it viewed by the Democratic and Republican parties?

Brian,
I recommend reviewing the information found at http://www.achieve.org/ for the answers to your questions. Achieve has been very transparent regarding its funding, financial support, and its apolitical positions.

Hi, Michael,
I have read the Achieve.org website, so I know the basics available there. The thing is, there are a lot of commissions, nonprofits, think tanks, etc. on the landscape (of education and many other areas, of course). I want to understand how Achieve, as one of these, actually fits into the way policy gets implemented - clearly it's been embraced by the Obama administration, but what role is envisioned for the common core when ESEA is reauthorized (if it is)? What support does it have in Congress? What are the minimum steps a state should take for implementation? How is Achieve working with CCSSO to encourage testing and integration?
In other words: The common core is not just a set of standards, it represents a large-scale policy undertaking, which will require a long investment of attention, as well as money and expertise to be realized. I don't understand yet the theory of action, the logic model.

First Stephen, thank you so much for the webinar presentation. For those of you who missed it, it is now archived on the site. Quite a few people were asking about alignment between the standards and assessments. This seems like a complicated issue as my understanding is that it is not part of Achieve's current mission, and it is not yet clear whether multiple assessments developed, perhaps by each state?

But I suspect that the assessments will influence how, and to what extent, the standards are adopted. The ways that students are asked to demonstrate their knowledge may also end up shaping how much experimentation and inquiry takes place.

We know that inquiry, labs, hands-on experiences, and field take time. Unless the assessments tap into the the skills gained through these experiences, I fear they may be left out.

I would be interested in hearing how you think the path from standards to assessments will take place, and if you suspect that this will vary greatly by state?

--Joni

I agree totally Joni! It seems that each state creates their own assessments which in my opinion muddy the waters that having a consensus on the standards made clear. I think that assessment has to go hand in hand with the standards.

Students need the experiences of inquiry, laboratory, hands-on, field and real world application in order to make sense of what they are learning. These kind of experiences will lead to quality assessments and successful mastery by students. It will be interesting to see how this plays out. If assessment is not apart of the equation, I am afraid that we are in a "short-lived" reform effort. Jeff

Add to the states' propensity for creating divergent standards their lack of continuity with course sequence and you have "curriculum spaghetti" I like the idea, for middle schools, of an 8th grade (not 7th like many states) science assessment that is uniform across the country. HS assessments need to be end of course. This ispossible and practical because most HS science courses are very similar in content, ie HS biology courses are very similar across the states, etc. Can we get higher ed to set some standards as to what science assessments HS students must bring to the admissions process??

Would like to recommend an interesting paper posted to MSPNet which addresses the need for and suggests a format for measuring the skills that inquiry in science can help students develop. "Next Generation Assessments for Measuring Complex Learning in Science" Clarke-Midura,Dede,Norton from the RENNIE CENTER for Education Research and Policy. In my opinion,It will be the nature of the assessment which will drive teaching - maybe more than the Standards.

Donna, I read the article and love the idea of using VPA's (Virtual Performance Assessments) as a way to assess learning. I think it would be a great fit for the new framework.

That articles looks very interesting. I think you are right in that we need to go beyond simple item response assessments both to assess whether or not students can use the content they learn productively and to encourage a greater focus in classrooms on critical thinking skills. The type of reasoning assessed by the program in that paper looks to be the type of reasoning we would want from our future citizens and workers and if the type of critical thinking that I would want them to use across subjects and in there own lives. Technology is certainly the vehicle that can make this type of assessment practical on a large scale.

What we need to avoid is the present use of scale scores. For example, in New York State, the Living Environments' Regents has a possible total of 85 points. To receive a scale score of 65 (which suffices as a passing score) a student needs to receive 40 out of 85. (You do the math.) For each regents, New York State provides a conversion chart to show what scale score a student receives for any raw score.

What shows on the report card of a student who earns 40 out of 85 is the '65' scale score. Although percent correct may not be the best, it is far better than what scale scoring does to determine how much an individual student truly knows.

Frank-
You may be looking too closely at the one test and the its end result. It is my experience that the process in good NY schools is more student focused than your numbers show. At the end of 6th grade a skills practice test is given on physical science knowledge. During 7-8 grade, prior to the 8th grade test, "DBQ's"..Document Based Questions are part of all domain study, aiding students to embed critical thinking in multiple domains...and data evidence. By the 8th grade test , the grounding skills towards the high school regents are tested. This gradual, leaning on knowledge development towards the final Regents courses sets up for a scale score to provide a balance to the TEST design...rather than then student population. This flexibility is overseen by the Regents Committee and one can see the benefit in the Math test from the 2003 or 4 period which was found to be flawed.

The positive point you really are making is how do we provide a valued "learning progression" of testing that assists the science standards to be formative for the students' benefit.

Betsy Stefany

Betsy,

In your response, you mentioned 'good districts. What tests are they using?

In June, 2011 for the Living Environments Regents in New York State, a student needed 40 out of 85 to pass the test.

For the Integrated Algebra Regents in June, 2011 in New York State, a student needed 31 out of 87 to pass.

Can we validate that these scores reflect proficiency?
A better question is, "Can we defend the position that these scores reflect proficiency?"

Will the results of the assessments that are now being developed to reflect the CCSS be reported in the same way as the above mentioned Regents? The results of these assessments need to inform us as to the proficiency of individual students.

Frank Gardella

Frank Gardella

First, I agree that a passing score that represents only 47% of questions answered correctly does raise some questions, which could be directed to and answered by the assessment creators.

However, I disagree with your general statement about scale scores. My understanding is that scale scores allow you to compare "equal" performance across test forms and test levels. A better question to ask is what the "65" does represent. It may very well represent a demonstration of skills that shows grade level proficiency in meeting standards for a student. A percentage score does not do either.

I have used standardized assessments and scale scores for many years with many adult learners, and it is always important - and a struggle - to explain the scores in a way that the student understands what the number "means". But it is a necessary step in any use of standards, in my opinion.

Hi,
I do remember a long time ago beign taught that the best test, that allows the most discrimination, has a mean of 50% and a standard deviation of 25%. I am not sure how this works with multidimensional testing (looking across various items at various levels of critical thinking skills required, as well as different areas of content).

So, I am less concerned with a scaled score than with knowing how the score maps to critical thinking and content. I have reviewed many state tests in the sciences and have been quite concerned. The level of test-development background varies widely, in my experience. No one ever reports out the kind of statistics that would give me the greatest confidence in the items or the instrument.

It is my sincerest wish that with the new standards, we will see a move to national development of assessments that can be tested themselves and demonstrated to represent what we wish to be assessing in our students. As has been said before in this discussion, the assessments are what will drive real change!

I just wanted to make a clarifying remark in reply to Gay's earlier post. I totally agree that a key issue here is how assessments (however constructed) map onto content and skill areas. The states that I am most familiar with (FL and MI) do a good job of this, using clear standards for item writing and test blueprints, and later standard setting procedures to set proficiency cut scores and the like.

Also, most states rely on item response models for their item analysis and psychometric work. Using percent-right scores is quite rare because states need to use historical item pools and extensive item piloting. This is needed because scores must be equated year-to-year and linked across grades. That is very hard to do well using classical test theory. Also when several cutoffs are needed, centering items at the 50% score won't give good reliability for classification.

This is going to be a complex enterprise whether it is done by states, consortia of states, or the nation as a whole.

Betsy has raised an important issue. One of the advantages of national assessment, similar to what NAEP has done over the years, is that states do not arbitrarily set 'cut scores'. There are relatively consistent expectations and that allow (hopefullly) more reliable comparisons for whether the math/science education system is improving. The SAT and ACT have served a similar purpose over many years, but typically have only focused on measurements of college-intending students. Now we are looking at a systemic method for evaluating the links among curriculum, instruction and assessment that can also be compared internationally. Aggregated data on large samples give us a good picture of the 'typical' student and 'typical school'. Those are the trends where we would hope to see steady improvement over time - for all students, not just the top.

So true, and we have an opportunity to create the kind of assessments that measure (and thus promote) the higher order thinking skills, especially analysis and critical evaluation, with the work going on to implement Common Core standards. The reality though is that many states, Alabama is one, are modifying and altering the standards to fit their needs. I hope we have the will to support authentic assessments and not just assessments that use multiple choice questions.

Joni --

I agree that changing the assessment process is critically important. For most of the teachers who I work with, the question of whether some concept or know-how is on the state assessment is among the most important considerations regarding whether something is worth spending time on or not.

The exciting thing about the Framework is its integration of practices, core ideas, and crosscutting concepts. But, unless we have assessments that look at practices and crosscutting concepts, we will end up with instruction that continues to focus on acquiring content knowledge.

It does not seem likely that states will figure out how to do this different, broader kind of assessment on their own. They will need a good national model. Essentially, we need an assessment framework.

-- Bill Zoellick