Stories Sell: The Jasons and Doug Covered their Eyes and Put their Fingers in their Ears, while Telling a Story

The Jasons and Doug strike again in their new edition to proving the validity of Value Added Assessment in  “LA’s Leaders in Learning” published on August 21st. As I read through the piece, I was reminded of a lesson from one of my former bosses when I was in college. I was asked to create an advertising campaign showing the benefits of financial literacy for college students. My boss told me to keep one thing in mind, “Stories sell.”

It seems that the Jasons and Doug continue to cover their ears and close their eyes to the AMPLE evidence that their project is flawed. They used their most recent article to give you more anecdotes to get the reader to ignore the facts. Photo Courtesy: kidsatthought.com

I am not sure if my boss met the Jasons and Doug, but they surely got their information from the same school of thought. The striking thing about there most recent article is that it seemed like they were trying to convince themselves of what they were saying, but I digress.

Let me tell you several stories.

There is ample evidence that the use of any “high-stakes” assessment model to retain students, sanction schools, and punish teachers has lead to the increase of cheating incidents – most recently in Atlanta. This evidence has been well documented going back to the days before value-added became a prominent area of contention among educators:

Teacher’s Caught Cheating – CBS News 2003

Analysis Shows TAKS Cheating Rampant – Dallas News 2007

Under Pressure, Teachers Tamper with Test – New York Times 2010

There is ample evidence that the use of valued-added assessments in regards to assessing teachers is still largely flawed and needs further development:

Study by Kane & Staiger, 2002 for the U.S. Department of Education

Valued-Added Assessments have estimation errors from two sources.
1. Random differences across classrooms in measured factors related to test scores including student abilities, background factors, and other student-level influences.

2. Idiosyncratic unmeasured factor that affects all students affects all students in specific classrooms (e.g. dog barking or a disruptive student), which reflects year-to-year volatility

David Cohen uses satire to drive in the errors of the Valued-Added Assessment

There has been ample evidence that shows that the use of “high-stakes” assessments and analysis models (whether its API or VAA) lead to limiting of the curriculum to the “testable” subjects/skills along with other effects to students:

Analysis of Urban Schools by Diane Chung-University of Michigan

Problems with the goals of NCLB

While these stories only cover a small portion of the mounting evidence used to evaluate high-stake tests and assessments (which what VAA would be used for), the Jasons and Doug are intent on using their stories to prove their point.

The Jasons and Doug in many ways are contradicting themselves through their own rhetoric albeit implicitly. In their most recent article, they dedicated a significant portion to scrutinizing the limitations of API or the Academic Performance Index used by California to “rank” schools in accordance to No Child Left Behind (NCLB).

“The API obscures the fact that students at Wilbur had the potential for further growth that went unrealized. Instead, they tended to slip every year while those at other esteemed schools in well-off neighborhoods made great strides. It also obscures the gains in schools in impoverished areas.”

“In elementary and middle school, the 1,000-point index is based entirely on how high students score on the state’s annual tests, given in Grades 2 through 12. According to state data, 81% of the differences among schools reflect socioeconomic factors such as poverty and parents’ education.”

Scores like API were developed and implemented partly due to NCLB. Diane Ravitch discusses the many prongs of NCLB in her most recent book The Death and Life of the Great American School System. However, one of those prongs were particularly important here:

“All public schools receiving federal funding were required to test all students in grades three through eight annually and once in high school in reading and mathematics and to disaggregate their scores by race, ethnicity, low-income status, disability status, and limited English proficiency. Disaggregation of scores would ensure that every group’s progress was monitored, not hidden in an overall average.

That last line is particularly important here. The development of API was supposed to ensure progress was monitored. The monitoring of the progress was supposed to cut across the various population groups. In other words, this was an attempt to “control” for the outside circumstances that a student may deal with during and after school. To the Jasons and Doug’s own admission that is not what API has done.

“But even those who designed the API more than a decade ago…. recommended measuring student progress as soon as possible.”

“The superiority of looking at student growth was recognized from the very beginning,” said Ed Haertel, a Stanford professor and testing expert who helped develop the API for the state. “It’s much more sensitive and accurate than the current system.”

But California, like most states, isn’t doing it.”

They suggest on their “More on the Value-Added Method” page that “[v]alue-added measures individual students’ year-to-year growth. The API compares the achievement level of one year’s students with those in the same grade the previous year. Experts say both are important and tell parents different things about a school.” The question that comes to my mind is how is a parent supposed to choose which one is more important API or VAA?

This is the issue I am having with the entire LA Times “study” on education. I often get the image of a small child closing their eyes and putting their fingers in their ears. Despite the fact that API was all the rage almost ten-years ago and it is shown (even by their own analysis) that it has problems and limitations, they insist on using a new “trendy” statistical model for evaluating schools and teachers. It is clear that there is NO evidence that VAA will help close the achievement gap for the various populations that were “tackled” in the development of the first statistical model. However the use of stories about Wilbur Elementary School “starting out as high achievers but [losing] ground on state standardized tests” is supposed to convince us otherwise. Maybe they will use the story of Maywood Elementary and the high population of students on free/reduced lunch improving the most. Either way, the LA Times study is a disingenuous attempt to improve education and is offering convoluted and inflammatory rhetoric about teachers and schools under the guise that empiricism will save the day. Unfortunately in this case, the “facts” that the Jasons and Doug are using has done nothing but make the misinformation about the nature of schooling, teaching, leadership, and testing more prevalent.

It was issues like this that I wanted (and I am sure others wanted) to discuss in their “chat” last Thursday. However, they circumvented the concerns of many in order to replace it with tired rhetoric and ambiguous statements.

Unfortunately, they still haven’t changed since the “chat.” In the most recent article they said, “The approach generally doesn’t penalize schools for things beyond their control.” I generally don’t eat fish (because I am a vegetarian) but if there is nothing else available I will. Marriages are generally perceived as a life long commitment, but people get divorced.

I really hope that their true intentions surface as a result of this project.

Advertisements

About mpal219
Educator, Student, Reader, Reformer, & Activist

One Response to Stories Sell: The Jasons and Doug Covered their Eyes and Put their Fingers in their Ears, while Telling a Story

  1. Sabrina says:

    Their true intentions already have surfaced: They want to sell papers, just like you said. In the meantime, more people continue to be confused by what all of this means (because JJ & D don’t understand it well enough to present anything of value, or to do justice to these complicated issues), which will probably cause many people to shut off from education news entirely. Sigh…

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: