Anybody who thinks that student achievement test scores and school accountability measures should be easily comprehensible is likely to get a shock this week as California publishes the latest results of its STAR test and reports on its Average Yearly Progress (AYP).

The AYP, required by federal law, is supposed to measure the gains each school makes in getting every major ethnic and economic subgroup to a goal where all students are "proficient" in all major subjects by the year 2013-14. It shouldn't be confused with the state's own API -- Academic Performance Index -- a similar program to drive up each school's student achievement.

For schools, a lot rests on AYP. Those that don't meet their targets two years running for each subgroup -- Latinos, English language learners, poor children -- will, under President Bush's No Child Left Behind education act, face sanctions and be required to pay for transporting students who want to attend a school that's meeting its target.
But because the state sets the academic standards and the rate at which schools have to meet them, neither AYP success nor failure tells very much. California, unlike some other states, has set high standards but put itself on a "balloon payment" schedule that defers the largest required improvements on those standards until after 2007.

But that's only the start of the murkiness.

California has a new nationally standardized test this year, the CAT6, to replace the SAT9, which was at the core of the state's testing program for most of the past five years. Like the SAT9 scores, scores on the CAT6 will be reported in national percentile ranks, meaning where each student -- and each school and district's average -- stands in comparison to all other American students in the same subject and grade.

But because the new test is somewhat different and based on a different national sample of students, and because students and teachers are unfamiliar with it, it could produce either apparent gains or declines in individual scores reflecting nothing but the change in the test. Parents already confused by what test scores mean are not likely to get much help from these new numbers.

State officials say the differences shouldn't be great, but they're also producing a "linking study" that will, in theory, permit accurate comparisons between scores on the SAT9 and the CAT6. They also argue that because the state has substantially de-emphasized the standardized test part in favor of assessments based on the state's own academic standards, the attention should focus on the state standards.

But the results on the California standards-based part of the test allow no national comparisons. Nor are the results given in numbers, only with five descriptors: advanced, proficient, basic, below basic and far below basic.

That change in emphasis alone -- from the nationally standardized test to the test based on the state's own academic criteria -- represents a huge, though largely undiscussed, shift in educational philosophy.

It's consistent with federal requirements but since it's based on California standards, it tends to be squishy. And because each district in every state is under pressure to meet federal progress goals and will suffer penalties if it fails to meet them, there'll be pressure in each state -- indeed there's already pressure -- to lower standards.

Last week, as state education officials, local school representatives and academic experts, among them state Board of Education Chairman Reed Hastings, were trying to explain all that to a roomful of reporters, it became clear that even experienced journalists and educators were baffled by what one called this "bewildering array" of tests and data.

The new federal law, said one of those officials, "has made the situation muddled and confusing." But the changes in the state's own system aren't models of clarity either.

Nor, as UCLA professor Jeannie Oakes pointed out, do test scores amount to a real accountability system. They don't measure dropout rates, which are likely to be driven up by a high-stakes test such as the California high school exit exam. The higher the dropout rate (and the lower the graduation rate), the higher the scores. Nor does the system measure the adequacy of resources -- teachers, books, decent buildings -- California provides for meeting the standards.

The best clue for parents, Hastings said, is proficiency: "If your child is proficient on the state standards," he declared, "he's on his way to college. ... If he's not proficient, he's probably not going to college." No Child Left Behind, he said, "is Congress' first draft." It's the state's own API rankings, which will next be issued in October, that are the best measure of how well schools are doing.

The one certainty is that on the widely respected National Assessment of Educational Progress, California's poor white students are at the very bottom of national scores in fourth-grade reading compared to those in other states; poor Latino kids are eighth; and poor black kids are second from the bottom.