Tuesday, September 04, 2007
"Something's Just Not Right"
Just yesterday I wrote a post noting that the higher test scores in NYC public schools that Mayor Moneybags and Chancellor Klein have been boasting about as proof their education reforms are working are a sham. They're a product of in-house grading (with lots of top-down pressure being placed upon teachers to make sure test scores increase), kid glove rubrics (wherein the state and/or the city create tests that look difficult until you see how many correct answers are needed for a passing grade on the grading chart) or easier tests.
Today the NY Daily News publishes similar findings:
The Daily News looked at two different tests - the 2002 and 2005 state math tests and found the 2005 state math test was much easier. They gave the two tests to a sample of kids and the kids also found the 2005 math test was easier. And - lo and behold - the math scores miraculously rose in 2005, enough for Bloomberg to boast about them during his re-election campaign and claim the test score increases were due to his fine leadership abilities and education reforms.
The Daily News, however, says not so much:
And here's what the kids the News gave the 2002 and 2005 tests to had to say about them:
There you have it - straight from a kid's mouth. The short, simple and sweet reason why the test scores went up in 2005?
The test was easier.
It's a shame the Daily News and other media outlets couldn't have looked closely at these scores right around the time Moneybags and Klein were bragging about them. Instead the score increases were reported uncritically by both the newspapers and the electronic media, making it sound as if Bloomberg was the biggest education genius since Socrates.
But he isn't.
He's simply the beneficiary of easier state tests that boost both city and state scores.
If the Daily News could get a look at the city test methodologies, they'd find similar results - in years when the scores go up, the tests are either easier or the rubric is more generous.
But Klein and Moneybags won't let anybody from outside the DOE analyze the tests.
I think we all know why.
Because, as Professor Tobias said in the Daily News article, something's just not right.
UPDATE: The Daily News reports that both the mayor and the governor acknowledged test difficulty levels change from year-to-year:
Okay, so if the tests can only be used in "a relative sense," to "see how well we are doing vis-a-vis other parts of the state," how come Mayor Bloomberg's website, Mikebloomberg.com, boasted so highly about the score increases back in 2005:
Gee, there's no caveat about how the test score increases can only be used in "a relative sense." Instead the press release makes it sound like Great Education Mayor Mike Bloomberg single-handedly brought higher test scores and better educated students to the NYC public school system after decades of dysfunction.
Funny thing though - now we know from the Daily News that the test score increases Bloomberg crowed about back in 2005 are phony and we also know that if Bloomberg would allow independent analysts to look at those city tests from 2005 that he bragged about, they would also found to be suspect.
How do we know that?
Because he won't let anyone look at them.
The test methodologies are a secret.
And there's a reason why they're a secret.
Because he doesn't want anyone just how easy the tests were and/or how generous the rubric was.
SECOND UPDATE: The Daily News reports that Chancellor Klein has been shamed by reports the score increases made on the 2005 fourth grade math test were phony into making the tests and the methodologies of city tests available to an "outside audit bureau" -- but there's a catch:
So what's the catch?
Klein himself will sit on the audit bureau and the rest of the board members will be Klein cronies with vested interests in the outcome of the findings!!! UFT president Randi Weingarten (in many ways also a Klein crony and also with a vested interest in the findings) will sit on the bureau board as well.
In other words, the fix is in. Chancellor Klein's independent outside audit bureau will be neither independent nor filled with outsiders without vested interests in the outcomes of the investigations.
What do real experts without vested interests in Klein's "reforms" think about the announcement of the audit bureau?
Not much:
Indeed.
Today the NY Daily News publishes similar findings:
When test scores rise, politicians crow that schools are getting better, but a Daily News analysis of recent standardized math exams and a News experiment suggest another reason: The questions might be getting easier.
The News obtained technical details on high-stakes math tests given to fourth-graders across the state over the past six years and found that in every year when scores went up, testmakers had identified the questions as easier during pretest trials.
In years when scores were lower, pretest trials showed the questions were harder.
"That's pretty strong evidence that something is just not right with the test," said New York University Prof. Robert Tobias, who ran the Board of Education's testing department for 13 years.
"If this were a single year's data or two years' data, I would say it would be inappropriate to make conclusions," Tobias said. "But with the pattern over time ...that's prima facie evidence that something's not right."
The Daily News looked at two different tests - the 2002 and 2005 state math tests and found the 2005 state math test was much easier. They gave the two tests to a sample of kids and the kids also found the 2005 math test was easier. And - lo and behold - the math scores miraculously rose in 2005, enough for Bloomberg to boast about them during his re-election campaign and claim the test score increases were due to his fine leadership abilities and education reforms.
The Daily News, however, says not so much:
In 2005 when a record-breaking 85% of New York State's fourth-graders passed the test, the questions had the highest average easy score in years. The easy score was .73 - meaning the average question was answered correctly by 73% of the kids who participated in pretest trials.
In contrast, when 68% of kids passed the state test in 2002, the easy score was .61.
State education officials insist their tests are carefully crafted to ensure apples-to-apples comparisons, except in years like 2006 when the test was subjected to an overhaul.
The easy score - called a Probability-value or P-value - is just one measure that testmakers use as part of a sophisticated test-equating process.
...
The News gave the multiple-choice portions of the 2002 and 2005 fourth-grade tests to 34 kids in a Brooklyn College summer program.
Though the two exams were supposed to be equally difficult, the kids did significantly better on the 2005 test, the exam the P-value showed was easier.
And here's what the kids the News gave the 2002 and 2005 tests to had to say about them:
In 2005, the happy news that city fourth-graders had dusted their predecessors and showed record gains on the state exam helped buoy Mayor Bloomberg's reelection drive just weeks before Election Day.
Did Bloomberg's leadership boost the scores?
Not if you ask 9-year-old Kirwin Seger.
When he participated in The News' experiment, he was among several students who said he finished the 2005 test faster than the 2002 version.
"The 2002 questions were more complicated than in 2005," he said. "In 2005, they kept it short, simple and sweet."
There you have it - straight from a kid's mouth. The short, simple and sweet reason why the test scores went up in 2005?
The test was easier.
It's a shame the Daily News and other media outlets couldn't have looked closely at these scores right around the time Moneybags and Klein were bragging about them. Instead the score increases were reported uncritically by both the newspapers and the electronic media, making it sound as if Bloomberg was the biggest education genius since Socrates.
But he isn't.
He's simply the beneficiary of easier state tests that boost both city and state scores.
If the Daily News could get a look at the city test methodologies, they'd find similar results - in years when the scores go up, the tests are either easier or the rubric is more generous.
But Klein and Moneybags won't let anybody from outside the DOE analyze the tests.
I think we all know why.
Because, as Professor Tobias said in the Daily News article, something's just not right.
UPDATE: The Daily News reports that both the mayor and the governor acknowledged test difficulty levels change from year-to-year:
"Tests go up and down in terms of difficulty so all you can really do is use them in a relative sense," Bloomberg said.
...
Although Bloomberg frequently trumpets annual jumps in city scores, yesterday the mayor said the tests are best used for comparing students with one another."What we can use this test for is to see how well we are doing vis-à-vis other parts of the state," Bloomberg said.
Spitzer defended the exams, saying, "We obviously don't think the test is getting easier."
But he acknowledged "you cannot invest enormous weight in a one-year differential because ... there are going to be marginal differences year to year in the nature of the test."
Okay, so if the tests can only be used in "a relative sense," to "see how well we are doing vis-a-vis other parts of the state," how come Mayor Bloomberg's website, Mikebloomberg.com, boasted so highly about the score increases back in 2005:
Mayor Michael R. Bloomberg and Schools Chancellor Joel I. Klein today announced that New York City's public school students in grades 3, 5, 6 and 7 this year achieved the largest one-year gains ever on the City English Language Arts (ELA) and Mathematics (math) tests, as well as the highest overall scores ever on the tests. The number of students meeting or exceeding standards in these grades combined increased by 14.4 percentage points in ELA and by 7.5 percentage points in math from last year. For the first time, 50% or more of all of the City's students in these grades met or exceeded the ELA (54.8%) and math (50.0%) standards for their grades. In addition, Black and Hispanic students achieved their greatest one-year gains and best performance ever on both tests. The tests are developed and scored by two independent testing companies - Harcourt Assessment, Inc. (ELA) and CTB/McGraw-Hill (math).
"One look at today's City ELA and math results and it's clear that the reforms that we put in place are working," said Mayor Bloomberg. "Over the last two years we have begun to bring order and accountability to a system that had been dysfunctional for decades. By providing students with the resources they need and holding them, their teachers, and ourselves accountable for producing results, our schoolchildren are now receiving the education they deserve. Today's results, like the achievement and gains on the State test scores released two weeks ago, are remarkable and a sign that things are moving in the right direction. I congratulate the students, teachers and parents that have worked so hard to achieve these record results."
Gee, there's no caveat about how the test score increases can only be used in "a relative sense." Instead the press release makes it sound like Great Education Mayor Mike Bloomberg single-handedly brought higher test scores and better educated students to the NYC public school system after decades of dysfunction.
Funny thing though - now we know from the Daily News that the test score increases Bloomberg crowed about back in 2005 are phony and we also know that if Bloomberg would allow independent analysts to look at those city tests from 2005 that he bragged about, they would also found to be suspect.
How do we know that?
Because he won't let anyone look at them.
The test methodologies are a secret.
And there's a reason why they're a secret.
Because he doesn't want anyone just how easy the tests were and/or how generous the rubric was.
SECOND UPDATE: The Daily News reports that Chancellor Klein has been shamed by reports the score increases made on the 2005 fourth grade math test were phony into making the tests and the methodologies of city tests available to an "outside audit bureau" -- but there's a catch:
Schools Chancellor Joel Klein will give math and reading test scores and other school documents to a new audit bureau being set up to verify the crucial data, the Daily News has learned.
In an exclusive report this week, The News showed that fourth-grade math exams administered by the state each year may be easier in some years than in others.
The findings led several education advocates to call for audits of school data, including standardized test scores.
An Education Department spokesman said Klein will turn over the test scores and troves of student data to a new organization called the Research Partnership for New York City Schools.
So what's the catch?
Klein himself will sit on the audit bureau and the rest of the board members will be Klein cronies with vested interests in the outcome of the findings!!! UFT president Randi Weingarten (in many ways also a Klein crony and also with a vested interest in the findings) will sit on the bureau board as well.
In other words, the fix is in. Chancellor Klein's independent outside audit bureau will be neither independent nor filled with outsiders without vested interests in the outcomes of the investigations.
What do real experts without vested interests in Klein's "reforms" think about the announcement of the audit bureau?
Not much:
Sol Stern of the Manhattan Institute, a conservative think tank, said, "The whole thing is a fraud.
"Every one of these people has a stake and an interest in what the research will show," he said.
Stern is among those who have called for an independent auditing bureau.
"This is not what I had in mind," he said. "This could turn out worse because it will have the illusion of objectivity."
Indeed.
