Chapter 5 – Journalled review of Diane Ravitch’s Reign of Error. “The Facts about Test Scores.”

In Chapter 5 of her monumental work, Reign of Error, Diane Ravitch offers the results of The National Assessment of Educational Progress (NAEP) assessments as compelling evidence that “students in American public schools today are studying and mastering far more difficult topics in science and mathematics than their peers forty to fifty years ago.” On the basis of this evidence she believes that “Test scores are at their highest point ever recorded” and, therefore, claims that “the educational system is broken and obsolete” are simply not true. It may be true that scores are higher than ever for the students at the elite end of the academic continuum but it ignores the stark reality that is the performance of the majority of American school children.

It is somewhat ironic that Ravitch is critical of the value of standardized testing in assessing the efficacy of public education on the one hand but cites NAEP results as evidence of the health of public education in America on the other. Her argument is that the NAEP assessment process is a different sort of testing and is far more meaningful that they typical standardized competency examinations used in states throughout the U.S. About this she is correct and we will examine the NAEP assessments and their results, shortly.

It is also important that we examine the context in which Ravitch’s arguments are made. Like most educators, Ravitch is frustrated at the savagery with which our schools and teachers are being blamed for the perceived failure of public education in America. Being attacked, even when criticisms are justified, is far more effective at putting educators on the defensive than it is as a catalyst for meaningful educational reform. When the criticisms are unfair and based upon claims that are unfounded or prejudicial, the intensity of one’s defensive posture is magnified.

As we have said throughout this “journaled review of Ravitch’s Reign of Error,” she is right to challenge the basis for such claims and also the solutions proffered by the “evil corporate reformers.” Where she is wrong is to insist that the documented improvements in the performance of our public schools, as measured by NAEP, are acceptable and that they prove that public education in America is not failing. We would suggest that the NAEP results prove rather clearly that public education is, indeed, failing.

Let us digress for a few paragraphs and take a look at the NAEP assessment process. Ravitch is correct that the NAEP assessment process is a meaningful tool and that the NAEP, which is part of the US Department of Education, and its independent governing board, the National Assessment Governing Board (NAGB) are a highly reputable, bipartisan body comprised of educators, elected officials, business people, and members of the general public.

The NAEP measures student performance in reading, math, and other subjects over time and reports results in two ways. The first is by scale scores, ranging from zero to 500 which reflect what students know and can do, without making judgments about whether the performance is good or bad. To use the vernacular, the results “are what they are.”
The second component is that Achievement Levels have been established in an attempt to put the raw results into some sort of meaningful context. It is acknowledged that these achievement levels are somewhat arbitrary and have created opportunities for over-interpretation.

On the NAEP assessments, an “advanced” level of achievement denotes “superior performance at each grade assessed.”
“Proficient” is defined, by the National Assessment Governing Board (NAGB) as “solid academic performance for each grade assessed. This is a very high level of academic achievement. Students reaching this level have demonstrated competency over challenging subject matter, including subject matter knowledge, application of such knowledge to real-world situations, and analytical skills appropriate to the subject matter.”

The NAGB defines “Basic” as “partial mastery of prerequisite knowledge and skills that are fundamental for proficient work at each grade assessed.”

The NAGB “believes, however, that all students should reach the Proficient level; the Basic level is not the desired goal, but rather represents partial mastery that is a step toward Proficient.”

“Below Basic” represents students who are unable to demonstrate even partial mastery of the work at each grade assessed.

We must maintain an awareness that these achievement levels are determined by the judgment of a panel of people on the basis of the performance of actual testers using percentiles. In effect, we are superimposing a distribution curve over the scores in order to make it more meaningful for us. As we shall see, later, the results tell us how much students know but it does not even begin to tell us where they should be performing at a given grade.

We can draw meaningful conclusions from the scores that will help us chart a course for the future, however and we will discuss this in some detail.

An important distinction with respect to NAEP results is that the scale scores of 0-500 change very little over time and actually show how students are moving up the scale, i.e. from the 4th grade assessment to the 8th grade assessment. In essence, the assessment shows how much kids know and can utilize at a given point in time. It might be helpful to think of it as a continuum along which students move as they gain increasing levels of mastery over the subject matter with no ceiling as to how far they can progress other than topping out at a score of 500.

Now, we want to compare the NAEP assessment process with state competency exams and we will use Indiana’s ISTEP+ simply because of the author’s relative level of familiarity with it.

Right out of the gate, be aware that we are going to over-simplify this process but it is the logic of the process that we want to illustrate.

On ISTEP+ for Grade 3 Math, for example, there is an expectation that a given number of areas of content will have been presented to the students by the time the ISTEP+ for that grade level is scheduled to be administered. The primary question the ISTEP+ is designed to measure is whether or not students “have learned what we expect them to know.” Specifically, what they are expected to know is defined within the context of state standards. What is not obvious from the published data are such questions as “How were passing scores determined?” In other words, how high was the bar set? For example is passing 60 percent? Eighty percent?

So, in the case of NAEP we are assessing how much a child of a given age knows and can utilize, without regard for how his or her classmates might be doing. For the ISTEP+ we seem more concerned about how a given student’s performance compares to the performance of his or her classmates.

Returning to our discussion of how NAEP results are to be interpreted, Ravitch disagrees with those who “assume that students who were not “Proficient” on the NAEP were “below grade level”.” “That is wrong” Ravitch insists and she suggests that having “76 percent” or some comparable percentage of the student population at “basic or above” is something to be touted.

I tend to agree with the NAGB that “Basic” is not our desired goal and that all students should reach the “Proficient” level of achievement, which is a point at which they have “demonstrated competency over challenging subject matter. . . .” and can apply “that knowledge to real-world situations. . . .”

Think about the difference, here. Ravitch is stating that the NAEP results in which 71 percent of eighth grade students are at “Basic or above” is validation that our schools are not in a state of crisis.

I would suggest just the opposite, and the NAEP would seem to agree, that it is clearly unacceptable that only 44 percent of eighth grade math students are “Proficient or Advanced.” Why would we ever think our job is done when any student has achieved only “partial mastery of prerequisite knowledge and skills that are fundamental. . .” to achieving proficiency rather than “Proficiency” itself? Why would we ever think it acceptable that 56 percent of eighth grade math students are not at a point where they have mastered the material sufficiently to apply that knowledge to real world problem-solving?

And, let us not forget about the 29 percent of eighth grade math students who have yet to demonstrate even partial mastery of the material. The two realities taken together spell crisis in this author’s mind in bold upper-case letters.

Other points of concern include:

The fact that we have no sense at all about whether or not the bar has been set sufficiently high;
The fact that only 6 to 8 percent are performing at an advanced level;
The fact that the variance between the 10th percentile and the 90th percentile is a cavernous 94 points (for eighth grade math students); and
The fact that the performance gaps between white students and their black and Hispanic classmates remains at an unconscionably high rate and that closing the gap from 32 points to 25 points for African-American students over a twenty plus year period is an accomplishment about which we should feel embarrassed rather than proud.

In American educational thinking, we are caught up in the idea that only 6 to 8 percent of students can be A students. In the business world, no production manager would be content to have such disparity of performance. The expectation would be that 75 percent of more of employees are working at the highest level of productivity and that those employees who are not are receiving aggressive remedial attention.

What we can say with some certainty, when talking about American public education, is that few if any students are performing at the highest level of which they are capable and the vast majority are nowhere close to achieving their potential. Helping individual students reach the highest tiers of their potential should always be the goal of our schools and teachers and we should not be squandering a single second worrying about whether Child “A” is keeping up with his or her classmates.

The NAEP Assessment tools appear to offer a high level of utility in judging the efficacy of our systems of public education. What we need to focus on is closing the gaps and raising the bar.
Let us not forget that the biggest fallacy in over-reliance on standardized testing, of any kind, as the ultimate measure of accountability for schools and teachers is that schools and teachers are only a small part of the equation for academic excellence.

To suggest that our current level of achievements provide evidence that no crisis exists in American public education is nothing short of absurd.

Charter Schools Are Not the Solution to Public Education in the US

While I would enthusiastically support the concept of creating a charter school to test a new instructional model and would certainly approve the use of incentives to encourage families to place their children in that school, that is not the way charter schools are being utilized here in Fort Wayne, Indiana nor, I suspect, in most other communities in the US.

The Charter schools with which I am familiar are might posture themselves as innovative but, in reality, they are little more than lifeboats floated out into the murky sea of public education to give the few families that are so inclined a place to which they can escape from the public schools.

There are insufficient numbers of these lifeboats to accept more than a miniscule percentage of the total population of children who are in the figurative “damaged ships” that are our urban public schools; thus, such schools can have no more than a marginal impact on the challenges facing public education in America.

The fact that we create these avenues of escape for the most motivated families and their children and still expect the teachers of the abandoned school to improve scores on standardized competency examinations is absurd. Charter schools may be a lifeline for a small number of families but they are a virtual sentence of death, or at least imprisonment, to the abandoned schools and to their students and educators.

The message this sends to the community at large is “we cannot do anything to fix our most challenged public schools so let us, at the very least, help a few families and their children escape.

The practice of using school vouchers is also creating its own series of adverse impacts. Not only do vouchers drain scarce tax dollars out of the accounts of our most challenged schools, those dollars are not creating outcomes that are significantly better for the students as, at least locally, the performance of charter schools on competency exams is disappointing. Worse yet, is that a portion of the tax revenue siphoned out of the budgets of our public schools is being utilized for purposes other than the education of our children.

I know, personally, of two Catholic parishes in Fort Wayne that have found vouchers to be a profitable enterprise and have used the revenue to pay off the parishes debts to the Dioceses or to address non-school related financial concerns while requests for school related uses of the funds have been denied. Now that this practice has come to light, that unfortunate and inappropriate practice will be discouraged, hopefully, if not discontinued.

We say that our purpose is to fix public education and that is expressed in our collective mission statements. Our behavior suggests that we have given up on at least our urban public schools as lost causes.

The fundamental problem with education reform is that it amounts to little more than tinkering with obsolete educational processes that contributes, greatly, to the failure of its students and the overwhelming majority of educational reformers, “corporate reformers” or “traditional,” seem oblivious to this reality. No matter how many times we keep re-shuffling the same deck we will continue to get the same 52 cards. Unfortunately, they are the wrong cards.

If the reader can allow his or her mind to consider, imagine that we have landed on another planet with a couple of million families and we want to set up schools for our children. If we were to take advantage of this unique opportunity and design and educational process from scratch, would it appear anything like public education looks in the United States of America, today? If we were to apply any amount of imaginative, exponential thinking the answer would be that the system we would create would bear little resemblance to what we have here and now.

The saddest fact of all is that reinventing our educational systems and processes would not be all that difficult if only we would open our hearts and minds to a new way of thinking about education. With but a few exceptions, most of the things we would do differently would require nothing more than a majority vote of the local school district.

I am proposing that we apply a systems-thinking approach utilizing present day business principles to reinvent education in America. I am not talking about such business principles of the corporate board room as financial incentives, competition, privatization and entrepreneurialism. In fact, these are the exact wrong business principles.

The business principles to which I refer are the principles learned in an operational setting such has: focus on one’s customer; identifying and focus on one’s mission, structuring the operation to meet its objectives, problem-solving, teamwork, integrating quality assessments into the learning process, and giving the people on the production line (teachers and administrators) the tools and resources they need to do the best job of which they are capable.
In my book, Reinventing Education, Hope, and the American Dream: The Challenge for Twenty-First Century America, I offer a comprehensive blueprint for change.

My request to educators and reformers alike is: let us pause for a moment, clear our minds, and listen to the ideas of someone with a fresh perspective. What do we have to lose other than a few minutes of our time? This is something else smart businesspeople have learned. They often seek out consultants with a broader perspective to challenge their assumptions and paradigms. They even pay for this service, which is the best indicator of its perceived value.