School evaluation gets a new tool

First Posted:

MARK GUYDISH

Some public schools that didn’t seem to be doing so well in state standardized tests are actually making substantial improvements when results are viewed through a system called “Value Added Assessment,” according to the state’s first-ever report based on the newer – and more complicated – analysis of test results.

Take Greater Nanticoke Area’s Education Center.
By traditional measures – looking at the percentage of students who score proficient or better in annual math and reading tests – the school has done poorly for years. It failed to meet minimum, state-mandated goals three years in a row before reaching those goals last year. Yet when viewed through the “value-added” system, it’s the top performing school in Luzerne County.
Wyoming Area’s Sara Dymond Elementary is on the other end of the spectrum.
A consistent local star in the state’s annual test result reports, the school has met the minimum goals – which rise periodically – for the percentage scoring proficient every year since 2003.
Yet it is the county’s poorest performing school under the value added microscope.
Same tests, different results; what does it all mean?
Until now, all state annual reports on standardized test scores focused on one big issue:
Achievement.
If students hit a benchmark score, they were deemed “proficient.” It was the same benchmark for all students in all schools, and the federal law known as No Child Left Behind requires that all schools steadily increase the percent of students hitting or exceeding that benchmark, until 100 percent of students are proficient by 2014.
Meet the annual goals and the school is said to have made “Adequate Yearly Progress” toward that 100 percent ideal.
Measuring progress
But from the moment No Child Left Behind took effect, critics pointed to what they contend is a fatal flaw: It might show that a student knows enough to pass the test, but it doesn’t show how much the student learned. A bright student could slide through a year learning very little and ace the test, while a struggling student might learn a great deal but still fail the test.
Enter “value added assessment,” a statistical attempt to measure a student’s progress, or growth, in learning over time.
The difference is important. Statistics have consistently shown that performance on standardized tests is tightly linked to student demographics.
Students from low-income families, or who are members of a minority, tend to do more poorly on those tests.
Special-education students and those who are English Language Learners also do more poorly on standardized tests.
The “achievement” system does not take those factors into account. It requires all students to reach the same test goal.
As a result, a school with a very small percentage of low-income students will almost always outperform one with a high percentage on the state tests under the current system.
For well over a decade of state testing, educators have repeatedly complained about this, pointing out such differences make it unfair to compare schools by test results alone.
Value added is designed to account for demographic differences. It looks at the same test results (no new tests are given) but with an eye on how much a student’s performance improved each year.
Where the student started from doesn’t matter. How far he gets from that starting point to the end of the year is what the system is supposed to gauge.
This being education, the state has a predictable gobbledygook of names and acronyms for it all. The tests are called the Pennsylvania System of School Assessment, or PSSA.
The value added system is called the Pennsylvania Value-Added Assessment System, or PVAAS.
VAAS test results have been provided to school districts on a limited but growing basis since 2008. Thursday marked the first time any of the data has been released to the general public.
The state tests students in grades three through eight and 11 in math and reading. The PVAAS System only looked at test results for grades four through eight, so schools that do not have those grades are not included in the new report.
The state also provided data for 2010 alone, and provided a three-year average. In reviewing the data, The Times Leader looked at the 2010 data, though there were few differences between that and the three year averages.

Seeing entire picture
The state warns against comparing schools based on value-added alone, insisting that a better picture of a school’s success is gained by looking at both the traditional PSSA results and the PVAAS results.
To that end, The Times Leader also looked at the “Adequate Yearly Progress,” or AYP, status of each school in 2010.
There are numerous variables, but AYP is a broad gauge of whether schools had enough students score proficient or better on math and reading scores to meet state goals.
If a school misses AYP one year, it is put in a “warning” status. After that, each consecutive year it misses goals the status changes and the consequences increase.
The ultimate penalty is a state takeover of a school, though the state is far more likely to work with a district creating plans to turn things around.
A school that missed goals one year but makes them the next is said to be “making progress.” If it makes the goals the following year, it is back on the “Made AYP” list.
The state value-added results are presented as a “normal curve equivalents,” a statistical term that means nothing to the average person. Generally speaking, the higher the number, the better a school is doing, though there are variables. The state also presents the results as one of four colors: Green, yellow, rose and red.
The state likens the color scheme to taking a person’s temperature. Green would be an ideal 98.6 degrees. Yellow might be off that mark by a little bit, but not enough to warrant a trip to the doctor.
Rose would be high enough to be concerned, and red is a fever you should take seriously.
Likewise, a school coded green “was effective in supporting students to achieve one year’s worth of academic growth in a year.” Yellow means there’s “minimal evidence” the school was not effective, rose means “moderate evidence” of the same and red means “significant evidence.”
Value-added assessment has been around since the early 1990s, when Tennessee adopted a version in reviewing its state tests.
Gauge for success
Proponents have argued a good value-added system is a better gauge of a school’s success in teaching students than the traditional “achievement” model, and Pennsylvania has tried – without success – to get the federal government to allow the use of value-added assessment in gauging AYP under the rules of No Child Left Behind.

Value-added has also become a rallying cry for those who want to move teachers to a merit-pay system – something strongly opposed by teacher unions. Proponents contend that, since a good value-added assessment should account for demographic differences among students, the results should be a good gauge of how effective individual teachers are, which in turn would give a basis for pay raises. Critics counter that there are far too many variables in a classroom to accurately measure a teacher’s effectiveness through any annual standardized test, regardless of how the results are analyzed.