Living (or Dying) With Your NCLB Tests

Schools’ ability to meet expectations will depend on tests’ instructional sensitivity by W. James Popham
In a couple of years, America’s public schools will be inundated by a flood of federally mandated accountability tests. Before the end of the 2005-06 school year, the substantially expanded assessment requirements of No Child Left Behind Act will kick in and, based on students’ scores on those tests, every public school administrator will be regarded as a winner or a loser.

Talk about high stakes!

In some places, the deluge has begun. Several states are continuing to use the tests already in place by expanding that testing to all grade levels required by NCLB.

And while two years may seem like a ways off for those in other states,sensible school leaders recognize they soon will need to deal with the many schools certain to be labeled as “failing” for not meeting the requirement for adequate yearly progress on test performance under NCLB. School administrators who prefer to simply await their fate will have adopted an unsound survival strategy.

The time to dig into this issue is now. Unfortunately, the options for action are neither numerous nor nifty. But first a fundamental proposition needs to be raised—namely, that whether school leaders and their schools will be fairly or unfairly evaluated on the basis of students’ test scores depends completely on the particular tests that have been adopted by their state to determine adequate yearly progress. If a state’s NCLB tests are appropriate, the state’s educators have a chance to succeed. If a state’s NCLB tests are not appropriate, the state’s educators don’t have any chance at all. An appropriate NCLB test is one that has the capacity to detect genuine improvements in instruction.

Unrealistic Expectations
I do not intend here to describe NCLB’s extensive assessment provisions or the demands inherent in the law’s adequate yearly progress requirements. Because NCLB was signed into law by President Bush back in January 2002, most school leaders already have seen more than their share of explanations about the tests called for by the law and the levels of student improvement that must be demonstrated each year.

Regarding the law’s adequate yearly progress requirements, I concur with Robert Linn, a University of Colorado professor of education, who observed during his April 2003 presidential address to the American Educational Research Association that the law’s AYP requirements are altogether “unrealistic.”

So as matters currently stand, every school administrator soon will be saddled with a host of new accountability tests as well as a set of fanciful AYP expectations regarding students’ performances on those tests. Such a daunting reality might understandably encourage school leaders to simply toss in their administrative towels. But it’s definitely not towel-tossing time yet. There still are a few options for action available to school leaders.

Tests Predominate
If a state’s NCLB tests are instructionally sensitive, there’s a chance that (at least until the law’s unrealistic AYP expectations are adjusted) sufficient improvements will be made on those tests so the bulk of that state’s schools and districts will be able to meet their AYP targets. An instructionally sensitive test is one that can, as its name implies, detect instructional improvements when such improvements take place.

On the other hand, if a state’s NCLB tests are instructionally insensitive, there’s almost no possibility that the state’s educators can attain their AYP targets. That’s because an instructionally insensitive test simply isn’t able to help determine instructional quality. Improved teaching, even if it were present, will not be identified. In states where instructionally insensitive NCLB tests are in use, schools and districts deemed by the federal government to be “in need of improvement” will become the rule, not the exception.

Because students’ NCLB test scores will be the dominant determinant of success or failure under NCLB, it should be obvious that the nature of those tests is crucial.

When I suggest that NCLB’s unrealistic AYP expectations are likely to be adjusted, I know all too well what unbridled folly it is to try to predict the actions of our federal policymakers. However, I believe that as soon as more than half of our public schools nationwide are labeled ineffective because of the AYP requirements, those requirements will be modified. In at least a dozen states that have reported test results this fall, more than half of the schools have been labeled as not making adequate yearly progress.

It is absurd to contend so many of the nation’s schools are performing that badly. We will witness a growing clamor from citizens who just don’t believe their local schools are rotten. And ever-responsive to voter distress, elected officials will undoubtedly come up with a more realistic set of expectations for adequate yearly progress.

However, any such AYP alterations are unlikely to take place in the next year or two. Thus for several years public school leaders will need to play the accountability cards they’ve been dealt. And whether those cards are likely to help or harm a state’s students depends almost exclusively on the nature of a state’s NCLB tests. Let’s look, therefore, at what determines the quality of such NCLB tests.

Instructionally Sensitive Tests
An instructionally sensitive test is one that can determine the presence of instructional improvement (or worsened instruction) if it has occurred. Clearly, the underlying accountability strategy inherent in NCLB rests on an assumption that if students are taught more effectively, those students will perform better on accountability tests.

This assumption makes a good deal of sense. If, however, the accountability tests being used are largely impervious to the impact of improved instruction, then NCLB’s overriding accountability strategy clearly collapses. Consider the essential attributes of an instructionally sensitive NCLB test.

• Clear descriptions of what’s assessed.
First, an instructionally sensitive NCLB test must be accompanied by sufficiently clear descriptions of the content standards (curricular aims) being assessed so that a state’s educators can really understand what skills and knowledge are going to be measured. After reading such descriptions, teachers must truly understand what’s to be assessed at a level of detail suitable for their classroom-planning purposes. Those descriptions, however, must be relatively brief and definitely teacher-palatable. We can’t expect busy teachers to wade through lengthy, jargon-laden descriptions of what’s to be tested in order to ferret out an accountability test’s true assessment targets.

• A focus on a modest number of significant curricular aims.
Second, NCLB tests that are instructionally sensitive ought to measure only a small number of content standards. We dare not overwhelm teachers with seemingly endless lists of content standards. But because only a few content standards are to be measured, those curricular aims must be unarguably significant. That is, those content standards must be potent enough to incorporate any subskills or bodies of knowledge students need to achieve such a content standard.

An example of a potent curricular target would be a student’s ability to compose original narrative, explanatory, descriptive or persuasive essays. This composition skill is clearly a significant one, and its mastery surely requires students to learn a host of subskills (for instance, the mechanics of writing) and knowledge (such as the content to be incorporated in the essay). The modest number of content standards we end up selecting for an instructionally sensitive accountability test must be unquestionably potent.

• Instructionally informative results.
Finally, an instructionally sensitive NCLB test, after it has been administered, must report students’ performances so that both teachers and parents can identify which content standards the child has or hasn’t mastered. If test results are not reported according to students’ mastery of each assessed content standard, teachers will never be able to identify the ineffective parts of their instructional program that need to be improved or the parts that are working well.

An instructionally sensitive NCLB test then will satisfy all three of these requirements by providing clear descriptions of what’s assessed, focusing on a few significant curricular aims and reporting its results in an instructionally informative manner. If an NCLB test falls down on one or more of these three requirements, it will be instructionally insensitive.

Gauging Sensitivity
School administrators, therefore, need to discern immediately whether their state’s NCLB tests (either as currently planned or already determined) are instructionally sensitive. What needs to be done, simply, is to appraise your state’s NCLB tests according to the three requirements for such tests. Remember, if your state’s NCLB tests are remiss on any one of these requirements, then school administrators in your state may be more likely to be labeled as ineffective. Even if those labels are grossly inaccurate, the world won’t know it.

To make an accurate appraisal regarding the instructional sensitivity of your state’s NCLB tests, you’ll typically need to secure some information from your state’s department of education. I’m not referring to lengthy technical manuals loaded with sufficient psychometric machinations to ward off all but the ardent. What you need is straightforward information regarding (1) how many and how demanding are the content standards ostensibly measured by your state’s NCLB tests; (2) descriptions of the tests’ assessed content standards; and (3) the reporting procedures used to describe group and individual student results.

Based on your common-sense consideration of such information, you’ll then be able to determine whether your state’s NCLB tests are instructionally sensitive. If you feel uncomfortable about arriving at this crucial judgment alone, join forces with other concerned school leaders in your state. Moreover, a group-based determination of an NCLB test’s quality is likely to have more credibility than will an individual’s judgment. The educational importance of drawing a conclusion about the instructional sensitivity of your state’s NCLB tests is almost impossible to overestimate.

Options at Hand
Based on the judgment made about the instructional sensitivity of your state’s NCLB tests, I believe you can undertake some relevant actions though your choices, not surprisingly, will depend directly on whether your state’s NCLB tests are instructionally sensitive.

If you work in a state whose NCLB tests are instructionally sensitive, count your blessings. You’ll be operating in a setting where, at least for the next several years, you have a genuine opportunity to walk the AYP gangplank without falling off. Whereas school administrators in almost all other states soon will be in for a blasting, you might just snare a bit of bliss.

I suggest that you try to do an efficient job in promoting your students’ mastery of NCLB-tested curricular aims. If you do, this will allow sufficient instructional time so teachers can address other important skills and knowledge that are untested. There’s clearly more stuff that students should learn than what’s assessed on NCLB tests. Efficient promotion of students’ attainment of NCLB-tested curricular targets will leave time to teach students other worthwhile things.

If your state’s NCLB tests are instructionally insensitive, however, your options are neither numerous nor attractive. Let’s face it, you are being required to play a no-win educational game in which there will be some winners and many losers. Faced with that uncomfortable scenario, some educators will be tempted to engage in activities that, unfortunately, end up eroding the quality of a student’s education. For example, teachers may force their students to take part in literally months of practice-item drilling. Such drudgery may boost students’ test scores a few points, but those students usually end up abhorring school. The price is way too high for the payoff.

I can really generate only two legitimate action steps for school administrators who work in a state where NCLB tests are instructionally insensitive.

First, it still may be possible for you and your colleagues to educate state policymakers about the need to install instructionally sensitive NCLB tests. Some seemingly irreversible decisions are, in fact, yet reversible, although many states that are changing their tests have already begun the process of writing requests for proposals.

If you push to get more suitable NCLB tests adopted in your state, you don’t want to be seen as an educator who’s fleeing from accountability. You need to forthrightly endorse the need for a rigorous program of educational accountability. An instructionally sensitive NCLB test will provide citizens and policymakers with accurate evidence of a school’s instructional effectiveness. Any conclusions about a school’s quality based on instructionally insensitive NCLB tests, however, are almost certain to be specious.

As intimidating as this option may appear to be, the replacement of inappropriate NCLB tests with better ones is the only course of action for a school leader who’s concerned about the educational well-being of students. Instructionally insensitive NCLB tests will both yield inaccurate accountability evidence and will diminish educational quality. Such tests should be replaced.

A second option for school leaders is to initiate a powerful public-information campaign about the potential harm to children that will be triggered by the wrong kinds of tests. (See related article, page 12.) This action can be conducted simultaneously with the “change-the-tests” action. Indeed, a solid public information campaign describing the downside of instructionally insensitive NCLB tests might well precede or accompany a policy-level effort to replace existing NCLB tests.

The rationale for this second option is transparent. At present, many educators and most citizens are assessment illiterates. Such individuals can’t be enlisted as allies in a fight to secure better NCLB tests because those individuals won’t understand there’s something wrong with a state’s current tests. However, if parents, policymakers and other concerned citizens truly comprehend how an unsoundly implemented federal law can lower educational quality, then these individuals can be productive participants in any crusade for more suitable testing.

Passivity’s Perils
During recent months I have been following with interest the reactions of both citizens and educators to state-level and district-level reports revealing the percentages of schools that flopped on AYP. Those AYP failure percentages, as we have seen, bounce all over the ballpark. In some states, such as Florida, almost 90 percent of the schools failed their AYP targets. In other states, such as Minnesota, truly small numbers of schools took an AYP tumble.

Citizens’ responses, predictably, depend directly on the proportion of schools that failed. In settings where failure rates are relatively mild, most citizens display only mild displeasure, yet still urge educators to do an even better job. In locales where school failure rates are high, citizens tend to lash out at perceived educator ineptitude.

But what I find astonishing is the reaction of educators to these AYP-based failure reports. Remember, the chief determinant of AYP success is student performance on state NCLB tests. But almost without exception, educators accept the appropriateness of the tests being used in their states. As long as instructionally insensitive NCLB tests rule the day, then educational leaders better start signing up for workshops in “Humble Pie Made Edible.”

I wish I could proffer a series of wonderful choices to school leaders working in states where instructionally insensitive NCLB tests are in place or soon will be. But, at the moment, the barrel of options is about empty. For your own success and, more importantly, for the educational well-being of the students entrusted to you, I urge you to tackle this problem with all the energy you can muster. You can live with appropriate NCLB tests.

James Popham is an emeritus professor of education at UCLA. He can be reached at 1706 Keoniloa Place, Koloa, HI 96756. E-mail: wpopham@ucla.edu. He chaired the Commission on Instructionally Supportive Assessment, on which AASA participated.