In Chapter 5 of her monumental work, Reign of Error, Diane Ravitch offers the results of The National Assessment of Educational Progress (NAEP) assessments as compelling evidence that “students in American public schools today are studying and mastering far more difficult topics in science and mathematics than their peers forty to fifty years ago.” On the basis of this evidence she believes that “Test scores are at their highest point ever recorded” and, therefore, claims that “the educational system is broken and obsolete” are simply not true. It may be true that scores are higher than ever for the students at the elite end of the academic continuum but it ignores the stark reality that is the performance of the majority of American school children.
It is somewhat ironic that Ravitch is critical of the value of standardized testing in assessing the efficacy of public education on the one hand but cites NAEP results as evidence of the health of public education in America on the other. Her argument is that the NAEP assessment process is a different sort of testing and is far more meaningful that they typical standardized competency examinations used in states throughout the U.S. About this she is correct and we will examine the NAEP assessments and their results, shortly.
It is also important that we examine the context in which Ravitch’s arguments are made. Like most educators, Ravitch is frustrated at the savagery with which our schools and teachers are being blamed for the perceived failure of public education in America. Being attacked, even when criticisms are justified, is far more effective at putting educators on the defensive than it is as a catalyst for meaningful educational reform. When the criticisms are unfair and based upon claims that are unfounded or prejudicial, the intensity of one’s defensive posture is magnified.
As we have said throughout this “journaled review of Ravitch’s Reign of Error,” she is right to challenge the basis for such claims and also the solutions proffered by the “evil corporate reformers.” Where she is wrong is to insist that the documented improvements in the performance of our public schools, as measured by NAEP, are acceptable and that they prove that public education in America is not failing. We would suggest that the NAEP results prove rather clearly that public education is, indeed, failing.
Let us digress for a few paragraphs and take a look at the NAEP assessment process. Ravitch is correct that the NAEP assessment process is a meaningful tool and that the NAEP, which is part of the US Department of Education, and its independent governing board, the National Assessment Governing Board (NAGB) are a highly reputable, bipartisan body comprised of educators, elected officials, business people, and members of the general public.
The NAEP measures student performance in reading, math, and other subjects over time and reports results in two ways. The first is by scale scores, ranging from zero to 500 which reflect what students know and can do, without making judgments about whether the performance is good or bad. To use the vernacular, the results “are what they are.”
The second component is that Achievement Levels have been established in an attempt to put the raw results into some sort of meaningful context. It is acknowledged that these achievement levels are somewhat arbitrary and have created opportunities for over-interpretation.
On the NAEP assessments, an “advanced” level of achievement denotes “superior performance at each grade assessed.”
“Proficient” is defined, by the National Assessment Governing Board (NAGB) as “solid academic performance for each grade assessed. This is a very high level of academic achievement. Students reaching this level have demonstrated competency over challenging subject matter, including subject matter knowledge, application of such knowledge to real-world situations, and analytical skills appropriate to the subject matter.”
The NAGB defines “Basic” as “partial mastery of prerequisite knowledge and skills that are fundamental for proficient work at each grade assessed.”
The NAGB “believes, however, that all students should reach the Proficient level; the Basic level is not the desired goal, but rather represents partial mastery that is a step toward Proficient.”
“Below Basic” represents students who are unable to demonstrate even partial mastery of the work at each grade assessed.
We must maintain an awareness that these achievement levels are determined by the judgment of a panel of people on the basis of the performance of actual testers using percentiles. In effect, we are superimposing a distribution curve over the scores in order to make it more meaningful for us. As we shall see, later, the results tell us how much students know but it does not even begin to tell us where they should be performing at a given grade.
We can draw meaningful conclusions from the scores that will help us chart a course for the future, however and we will discuss this in some detail.
An important distinction with respect to NAEP results is that the scale scores of 0-500 change very little over time and actually show how students are moving up the scale, i.e. from the 4th grade assessment to the 8th grade assessment. In essence, the assessment shows how much kids know and can utilize at a given point in time. It might be helpful to think of it as a continuum along which students move as they gain increasing levels of mastery over the subject matter with no ceiling as to how far they can progress other than topping out at a score of 500.
Now, we want to compare the NAEP assessment process with state competency exams and we will use Indiana’s ISTEP+ simply because of the author’s relative level of familiarity with it.
Right out of the gate, be aware that we are going to over-simplify this process but it is the logic of the process that we want to illustrate.
On ISTEP+ for Grade 3 Math, for example, there is an expectation that a given number of areas of content will have been presented to the students by the time the ISTEP+ for that grade level is scheduled to be administered. The primary question the ISTEP+ is designed to measure is whether or not students “have learned what we expect them to know.” Specifically, what they are expected to know is defined within the context of state standards. What is not obvious from the published data are such questions as “How were passing scores determined?” In other words, how high was the bar set? For example is passing 60 percent? Eighty percent?
So, in the case of NAEP we are assessing how much a child of a given age knows and can utilize, without regard for how his or her classmates might be doing. For the ISTEP+ we seem more concerned about how a given student’s performance compares to the performance of his or her classmates.
Returning to our discussion of how NAEP results are to be interpreted, Ravitch disagrees with those who “assume that students who were not “Proficient” on the NAEP were “below grade level”.” “That is wrong” Ravitch insists and she suggests that having “76 percent” or some comparable percentage of the student population at “basic or above” is something to be touted.
I tend to agree with the NAGB that “Basic” is not our desired goal and that all students should reach the “Proficient” level of achievement, which is a point at which they have “demonstrated competency over challenging subject matter. . . .” and can apply “that knowledge to real-world situations. . . .”
Think about the difference, here. Ravitch is stating that the NAEP results in which 71 percent of eighth grade students are at “Basic or above” is validation that our schools are not in a state of crisis.
I would suggest just the opposite, and the NAEP would seem to agree, that it is clearly unacceptable that only 44 percent of eighth grade math students are “Proficient or Advanced.” Why would we ever think our job is done when any student has achieved only “partial mastery of prerequisite knowledge and skills that are fundamental. . .” to achieving proficiency rather than “Proficiency” itself? Why would we ever think it acceptable that 56 percent of eighth grade math students are not at a point where they have mastered the material sufficiently to apply that knowledge to real world problem-solving?
And, let us not forget about the 29 percent of eighth grade math students who have yet to demonstrate even partial mastery of the material. The two realities taken together spell crisis in this author’s mind in bold upper-case letters.
Other points of concern include:
The fact that we have no sense at all about whether or not the bar has been set sufficiently high;
The fact that only 6 to 8 percent are performing at an advanced level;
The fact that the variance between the 10th percentile and the 90th percentile is a cavernous 94 points (for eighth grade math students); and
The fact that the performance gaps between white students and their black and Hispanic classmates remains at an unconscionably high rate and that closing the gap from 32 points to 25 points for African-American students over a twenty plus year period is an accomplishment about which we should feel embarrassed rather than proud.
In American educational thinking, we are caught up in the idea that only 6 to 8 percent of students can be A students. In the business world, no production manager would be content to have such disparity of performance. The expectation would be that 75 percent of more of employees are working at the highest level of productivity and that those employees who are not are receiving aggressive remedial attention.
What we can say with some certainty, when talking about American public education, is that few if any students are performing at the highest level of which they are capable and the vast majority are nowhere close to achieving their potential. Helping individual students reach the highest tiers of their potential should always be the goal of our schools and teachers and we should not be squandering a single second worrying about whether Child “A” is keeping up with his or her classmates.
The NAEP Assessment tools appear to offer a high level of utility in judging the efficacy of our systems of public education. What we need to focus on is closing the gaps and raising the bar.
Let us not forget that the biggest fallacy in over-reliance on standardized testing, of any kind, as the ultimate measure of accountability for schools and teachers is that schools and teachers are only a small part of the equation for academic excellence.
To suggest that our current level of achievements provide evidence that no crisis exists in American public education is nothing short of absurd.