Karen Gentemann hardly looks like a revolutionary. She’s a middle-aged bureaucrat who wears light-blue suits and works out of a second-floor office of a drab brownish building at George Mason University in the bland expanse of suburban Fairfax, Virginia.
Nor does Karen Gentemann act like a revolutionary. She’s friendly, warm, and matter-of-fact. But she has a revolutionary idea: Colleges should publicly tell people how much learning is achieved on campus. They should do their best to measure it; then they should post the information online. She’s got a file cabinet brimming with manila folders from which it only takes her a second to extract the one that shows the best statistical analysis available on how well her school educates students. This may not seem so subversive, but it is. All over the country, administrators like Gentemann have similar manila folders; they just keep them padlocked in basement safes, forever beyond the reach of students, parents, or reporters.
Over the last few years, America’s views on education have hearteningly shifted. Politicians, voters, and many educators have come to a rough agreement that improving schools must involve measuring how much students learn and holding schools accountable for those students who fail. As part of the reform effort, schools need to publicly report data about their performance so parents can make informed decisions about where to live and where to educate their children.
Remarkably, higher education in America remains almost completely insulated from the impact of this movement. Log onto almost any college Web site and you’ll see the same beautiful students talking about their beautiful campuses and how much they learn. But there are no numbers, no studies, no objective measurements. In a Washington Monthly survey of 50 randomly selected research-university Web sites, only 12 percent clearly posted the six-year graduation rate, the most basic statistical measure of effectiveness. Even fewer offered information about student satisfaction with teaching.
Without solid information on what they will learn, students must make choices based on geography, particular programs, or reputation. As Derek Bok, former president of Harvard, has noted: “Competition succeeds only to the extent that customers, judges, or other trusted sources can define success in some legitimate way in order to establish a standard and reward those who best achieve it. In education, at least at the university level, this ability is lacking.”
Colleges don’t put out good information for several reasons. First, it’s hard to measure learning and a college’s contribution to it. You can’t just run simple tests upon a student’s arrival and departure. Grade-school and high-school students also have fairly standard curricula: Everybody studies Thomas Jefferson and equilateral triangles. But college students travel vastly different roads; some pore over Chaucer for four years, others study astronomy.
Second, for at least half a century, academia has eagerly offered precise and quantitative evaluations of everything but itself. College administrators scorn outside accountability and standards and prefer making decisions in the faculty club. In 1997, less than 10 percent of the member schools of the New England Association of Schools and Colleges (the regional accreditation agency for over 200 colleges and universities in New England) reported assessing student learning and using the results to improve teaching.
Third, and most important, it would embarrass many colleges and universities to admit just what happens on campus and how little attention they pay to students. As the Boyer Commission on Educating Undergraduates noted in 1999, undergraduates at research universities, where a large majority of American students undertake their higher education, ” … are the second-class citizens who are allowed to pay taxes but are barred from voting, the guests at the banquet who pay their share of the tab, but are given the leftovers.”
Worse, the problem will take years to change because graduate programs have long produced individuals expert in their discipline but awkward in the classroom. Moreover, promotion and tenure procedures on most campuses provide slim rewards for good teaching; professors are often released from teaching in order to focus on research, but it almost never works the other way around. An academic joke describes a traveler in first-century Judea seeing a figure on a cross, dying in agony with innumerable sobbing worshippers at his feet. “Who’s that?” asks the traveler. “The greatest of teachers. A man who changed so many lives.” “Then why are they crucifying him?” “Didn’t publish.”
Fortunately, there’s a movement underway to change this lack of accountability and the educational neglect it helps foster, led by a few noble administrators like Gentemann. Thank goodness. Just as too many American eighth-graders never learn to read, too many college students learn little more than how to tap a keg.
Ideally, prospective students and their parents would have access to reliable data comparing each institution on the most important collegiate outcomes: learning, graduation rates, and satisfaction. Prospective students could read it and use the data to help choose their college. School administrators could check on how they stack up against their peers.
Such a system already exists to some degree. Unfortunately, the highly influential U.S. News & World Report annual guide to “America’s Best Colleges” pays scant attention to measures of learning or good educational practices, even as it neatly ranks colleges in long lists of the sort that Americans love. It could be a major part of the solution; instead, it’s a problem.
U.S. News’ rankings primarily register a school’s wealth, reputation, and the achievement of the high-school students it admits. At one time, most academics believed in one simple equation: Good students plus good faculty equals good school. The rankings reflect this outlook, tabulating things such as percent of faculty with a doctorate (to measure the quality of the professors) and SAT scores of the freshman class (to get at quality of the students). That’s like measuring the quality of a restaurant by calculating how much it paid for silverware and food: not completely useless, but pretty far from ideal.
Still, such rankings matter deeply on college campuses. Administrators wait with bated breath for them to arrive each fall and brag when their schools do well; trustees fret about them; campuses even hire independent consultants to help figure out how to game them. Last year, Hobart and William Smith College sacked a senior vice president after she failed to submit fresh data to the magazine, an error that caused her college’s rank to tumble.
U.S. News nods in the direction of student learning and claims that it already sets out to measure it, assigning, for example, one percent of a college’s final score to the ratio of students to faculty on campus—a decent proxy for the kind of close interaction between students and professors that enhances learning. But not only does that proxy represent a minuscule contribution to the final score, the magazine also employs some deeply dubious measures, such as the percentage of faculty with doctorates—a factor largely unconnected to the capacity to arouse interest and to teach effectively, according to Alexander Astin, director of UCLA’s Higher Education Research Institute (HERI), who has done the most extensive research on the subject.
Furthermore, in perhaps the greatest indictment of the magazine’s educational worldview, average faculty salary (7 percent of U.S. News’ overall ranking) actually has a negative correlation with student satisfaction and learning. It’s largely a proxy for the emphasis the institution puts on research; and, unfortunately, research and teaching appear to be fairly exclusive, again according to Astin’s data. The more that a school emphasizes publishing papers, or searching out patentable technology, the less it emphasizes access and commitment to students.
And that’s no small matter. Over their four years at college, students at schools with strong research orientations (schools that win kudos from U.S. News for scoring well in the categories just mentioned) display decreased satisfaction with faculty and the overall quality of instruction, decreased leadership skills, and decreased self-reported growth in public speaking skills and other measures of student development. In contrast, in schools with faculties highly focused on students (which gain little in the rankings) undergraduates report increased satisfaction with almost every dimension of college and improvement in every academic outcome studied.
Astin’s research is complicated—measuring everything from what freshmen know when they show up in the fall to student satisfaction to the various programs, practices, and policies in place at the studied school. That’s the only sort of comprehensive data that allows researchers to isolate the quality of an institution from the quality of the undergraduates it admits. U.S. News does recognize the value of this approach and has a category comparing each institution’s graduation rate to its predicted rate, given the standardized test scores of its entering freshmen. Schools that produce more graduates than expected do well. But the measure is crude; it doesn’t distinguish between schools that have lower-than-expected graduation rates because they have challenging curricula and those that simply neglect their freshmen.
U.S. News also suggests that it assesses learning through a survey that the magazine sends to university presidents, provosts, and admissions deans requesting them to rank peer schools’ academic programs on a scale of one to five. This survey makes up 25 percent of a school’s score and, according to U.S. News, “this is how highly knowledgeable college officials rate the educational quality of the schools they feel qualified to rate.” That sounds reasonable. But a closer look suggests, not surprisingly, that the college administrators surveyed share the same bias in favor of research that pervades academia. Analysing U.S. News’ data, we found that a high reputation score in the college guide correlates much more closely with high per-faculty federal research and development expenditures than with high faculty-student ratios or good graduation-rate performance, the magazine’s best measures of undergraduate learning.
U.S. News makes the case that it can’t go further in trying to measure learning because solid, comprehensive, and usable data don’t exist right now. But none of the data that the magazine uses were readily available—at least in a standardized form—when it first devised the college rankings system in 1983. U.S. News built the rankings, and the data came. Universities used to report graduation rates in a bewildering number of ways: four-year, five-year, and six-year rates, with additional variation based on how those rates were calculated. Now, everyone reports in six-year rates in order to comply with the U.S. News standard. The same is true of many of the other categories the magazine uses. As the Hobart and William Smith example demonstrates, schools fall over themselves to comply.
The magazine still claims, though, that it is trying hard to measure learning. According to Peter Cary, editor of special projects who oversees the U.S. News rankings (who refused an interview but responded to written questions from The Washington Monthly), the magazine: “would, indeed, like to have more indicators that measure the actual educational experiences of students and the post-graduate outcomes of their experience. We have been concerned about this for years.”
That may be true. But if it is, the magazine has not done a stellar job integrating existing information. Virtually every college in the country asks its graduating seniors to complete a satisfaction questionnaire. Noel-Levitz, a midwestern research firm, consults with over 1,000 campuses and has established national norms for satisfaction surveys. You don’t find a trace of these in U.S. News. Of course, student satisfaction surveys aren’t perfect. But, then again, U.S. News only includes one vastly less reliable measure of satisfaction—the percentage of alumni who donate—which depends in large part on alumni wealth and how many pesky sophomores man phone banks in exchange for pizza and T-shirts.
U.S. News also claims that making a real effort to measure learning would be prohibitively expensive. But that’s not quite right either. Schools have always paid the cost of gathering the data for U.S. News, and there’s little evidence that further data collection would remotely risk breaking their banks. Setting up a program from scratch might cost too much for many schools. But the needed infrastructure already exists. For example, participating in Astin and HERI’s survey of freshmen costs a school about $1,500. Furthermore, many institutions must measure proxies for learning and provide them to the accreditors who determine if schools are eligible for government grants and aid. So they have to incur the cost or get out of the game—and if they can do it, U.S. News just needs to collect the information.
In truth, there’s usually a way if you are looking for a solution, and the evidence suggests that U.S. News doesn’t really think there’s a problem. After all, it’s been making millions of dollars publishing its rankings of “America’s Best Colleges” based on reputation and inputs for almost two decades. Moreover, although it’s true that, as Cary says, “We don’t hear a clamor from the experts we talk to—including high school guidance counselors—for a completely different methodology” it’s not clear that the magazine is talking to the right people. Assessment of learning is highly specialized and technical and not the expertise of most college presidents, or admissions officials. Furthermore, these “experts” represent their own institutions and often simply try to tweak the methodology in ways that will benefit their particular schools. They don’t represent prospective students or their parents. And few want to publicize the actual rate of learning that goes on at their universities.
There’s a case to be made (though Cary denies it) that the magazine hasn’t done the legwork needed to fairly measure learning because the rankings already accomplish what the top editors want. As was argued in these pages last year , U.S. News originally set out to create a seemingly scientific rankings system that would place well-known universities—particularly Harvard, Yale, and Princeton—on top. Those are the most common places where magazine editors earned their degrees, and the public expects to see them on top. Upsetting that equilibrium upsets the equilibrium in the U.S. News publishing offices.
Cary ultimately argues the following: “In the end, we believe that our rankings pass the very important test that all such efforts should pass—the test of common sense.” But to many people, ranking “America’s Best Colleges” without including legitimate measures of the quality of education and instruction defies common sense, and it was actually this glaring deficiency which sparked the movement towards measuring learning of which Karen Gentemann is a part.
In 1998, a group of educators supported by a grant from the Pew Foundation, motivated in large part by outrage at the U.S. News college rankings, set out to gather and evaluate data on learning. The result was the National Survey on Student Engagement (NSSE).
Cognitive science has convincingly demonstrated that learning comes down to—no surprise—time on task. The more a student puts into her education, the more she learns, provided she’s working on the right tasks. Subsequently, the NSSE survey asks freshmen and seniors to assess time spent—how many hours do they study? talk to a faculty member or students with different political beliefs?—and then shoots back data to schools telling them not only how well they’ve done but how they compare with peer institutions. NSSE’s data are, by far, the most comprehensive effort to quantify how well colleges accomplish their stated principal goal.
Of course, NSSE’s data aren’t perfect. Most problematically, to a certain extent, they measure student, not institutional, quality. You could lock some students in an empty garage for a year and they’d still learn; other students seem genetically programmed to play beer pong. Wrestling with this difficult issue, NSSE’s creators have developed an “Institutional Engagement Index” that shows schools’ scores relative to predicted scores based on characteristics of the facility and students. They present this as only “one way” to look at the results, but, in fact, the index (or something like it) is the only way to measure institutional effectiveness.
Still NSSE’s data are the most comprehensive available and the most likely to help spur academic reform—from education measures gathered by state accreditation agencies (which aren’t comparable across the country) to efforts to measure skill levels five years after graduation (which have a very hard time sorting out how much college contributes to an individual’s success).
Overall, 470 schools have participated in the NSSE survey, nearly a critical mass, though virtually none of the big-name research universities decided to play. Cost, once again, is not the problem. A large university would only have to pay about $5,000 to participate in NSSE. Their administrators argue, instead, that requests for surveys inundate them or that they are above measuring learning. According to Virgil Renzulli, associate vice president for public affairs at Columbia University, “Columbia is one of the most selective institutions in the country. We accept just below 13 percent of applications, so, in a way, asking students here whether they like to learn is like going to the Air Force Academy and asking the students whether they like to fly.”
Of course, this analogy illustrates the problem. Yes, everyone at the Academy likes to fly. But can they? The Air Force tests students in class, in simulators and in real planes with instructors sitting beside them before they get to graduate. They collect and analyze data on the results and look for better ways to train pilots. Does Columbia do the same for its students?
The most prominent schools also dodge the data because large research universities simply don’t devote as much time to students as they should, as they say they do, and as they are reputed to. If they aren’t going to look good, why should they participate? After all, when NSSE chose to publicize the four top-scoring schools in its 2000 survey, the institutions were Beloit, Elon, Sweet Briar, and Centre colleges. Educators recognize all four as good schools, but none is exactly the choice of most status-conscious members of the meritocracy.
Compounding that problem, before beginning the survey, universities balked at making NSSE’s data public, stiffing prospective students who want to know what to expect, and stifling the potential good that could come from prodding universities to pay more attention to their undergraduates. With public data, applicants to large research universities and state schools might see that they’d get more faculty attention at a smaller school. A few top-flight applicants choosing to go elsewhere could shift priorities in the largest schools where it’s often easier to see an orthopedist than a history professor.
Some people in NSSE hope that eventually the data will be made public; some don’t, and rather hope that universities will simply provide them directly to inquiring prospective students. Right now, NSSE relies on schools helping to finance its survey; and as Peter Ewell, one of NSSE’s creators, says, “People won’t pay for the gun that shoots them in the head.”
It’s doubtful however that institutions will be able to keep the data locked up forever. Freedom of Information Act requests or state reporting requirements may well force state universities participating in NSSE to publish results. And, the more administrators like Karen Gentemann who decide to post their data online independently, the more pressure will be put upon the holdouts. When told that her school was one of only a handful to make the results public, Gentemann responded: “I’m shocked. I see no reason why other schools shouldn’t be putting it out there too.”
The best solution, of course, would be for U.S. News to put the information out there, overhauling its rankings and basing them on the best measures of learning, not the best measures of resources and reputation, that it can find. Two years ago, U.S. News did something like this with an issue it published on outstanding high schools. That study had many of the features needed to gauge institutional quality—multiple measures of key outcomes including both measures of learning and academic persistence, controls for students’ family background, and a sophisticated statistical analysis of the data. The study was limited to cities in states that had standardized tests; nonetheless, it was a start.
The first step for U.S. News to accomplish this goal would be to work to incorporate NSSE’s data, or as much of it as U.S. News can get, into the rankings. That may take a while. According to Cary, the top editors of U.S. News met with the NSSE people and decided not to proceed. “The main hurdle was that NSSE had only surveyed students at [a limited number of] schools. Missing were Stanford, Harvard, Yale, Princeton, and many, many others that are of great interest to high school students and which find prominence in the U.S. News listings.” U.S. News has spoken with the American Association of Colleges and Universities (AACU) about developing similar measures. But the AACU is years away from having data equivalent to NSSE’s. It’s a step in the right direction, but we believe that soon NSSE will have a critical mass of colleges and that there will be enough Karen Gentemanns around for all the data to be made public. There’s also reason to keep the pressure on the magazine. There’s something contradictory in a magazine making a mint off of a ranking system called “America’s Best Colleges” that virtually ignores educational quality, spurring on a rival movement, and then claiming that, even if the rivals have the right ideas, their numbers just aren’t good enough.
Right now, U.S. News could be assimilating the available student satisfaction surveys, such as Noel-Levitz’s, and using its considerable sway to standardize data across all institutions. The magazine could also revamp its reputational survey to focus on how much undergraduates learn. Instead of asking admissions deans and university presidents to grade their peers, the magazine could send surveys to institutional researchers and people actively involved in assessing undergraduate education.
Academia needs a push. Five years ago, U.S. News introduced its category crediting schools for high alumni donation rates, a simple change which spawned scores of programs across the country where students call up alumni to ask for money. By making these changes we’ve suggested, the magazine could make a much bigger difference. Instead of encouraging students to talk to alumni for the sake of fund-raising, it could encourage faculty to talk to students for the sake of learning.
|
Amy Graham is the former director of data research for U.S. News & World Report. Nicholas Thompson is an editor of The Washington Monthly. You can read previous Washington Monthly articles on U.S. News by clicking here. You can email the authors by clicking here.
