In the early years of this century, Shanghai Jiao Tong University was a relatively unknown institution outside of China. In 1998, it had been selected by the Chinese government to be included in the 985 Project to build world-class universities. Beginning in 1999, a team at the university, led by Nian Cai Liu, developed the Academic Ranking of World Universities in order to benchmark the position of Chinese universities vis-Ã -vis competitor universities. The ARWU was first released in 2003 and was followed in the same year by Webometrics, developed by the Spanish National Research Council, and in 2004 by the Times Higher Education/QS World Universities Ranking. Today, there are about ten global rankings, and more than sixty countries have national rankings, many of which are sponsored by government or government agencies.
The release of the ARWU marked the emergence of the global phase of rankings. Prior to that, rankings—such as U.S. News and World Report’s college rankings, which were first published in 1983—were primarily about measuring national reputation and performance and providing information to students and parents. By placing higher education quality within a wider comparative and international framework, the ARWU was different. It immediately illustrated that national preeminence was no longer sufficient and that the higher education world was multipolar. In doing so, it set the cat among the pigeons.
Within months of the ARWU’s publication, the rankings were being called “a major wake-up call” for European higher education. To the consternation of European policy makers and academics, their universities made up only about one-third of the world’s top 100 institutions. Over the years, the precise number has ebbed and flowed, but the proportion has remained relatively static, depending on the ranking. European concern was set against the backdrop of the European Union’s (EU) ambitious Lisbon Agenda; launched in 2000, it sought to make Europe “the most dynamic and competitive knowledge-based economy in the world.” Strong, competitive, and modern universities lay at the heart of that goal.
In 2005, the German government launched the Exzellenzinitiative (Initiative for Excellence), replacing its long-standing policy support for equity across all universities with a plan to promote excellence among only a few. (Versions of this have since been introduced in many other European countries, including France, Spain, and Russia, and also around the world.) The French were equally troubled: in 2008, the French Sénat released a report arguing that the country’s researchers were being disadvantaged in favor of English-speaking institutions. A 2008 conference organized under the auspices of the French Presidency of the European Council championed the idea of a new EU ranking.
In contrast to the Shanghai ranking, which concentrates only on research, it was argued that the EU ranking should give due regard to the diversity of institutional missions and the breadth of higher education’s activity across teaching, research, and engagement. (It should be noted that the term “diversity” differs in other countries from its usage in the U.S., where it typically refers to ethnicity, race, and gender; elsewhere it describes a diverse range of college and university missions or purposes.) In 2005, the EU sponsored the first phase of a European classification system, launched as U-Map in 2009. In the same year, a consortium was established to test the feasibility of a multidimensional ranking, released as U-Multirank in 2011 with an operational version due for 2014.
U-Map is Europe’s version of the U.S. Carnegie Classification system. Based on the desire to highlight the diversity of European higher education institutions (HEIs), U-Map was designed as an interactive online tool that allows various stakeholders to choose the multidimensional classification attributes that are most important to them. There are twenty-nine indicators across six dimensions using official and institutional data: teaching and learning, students, research, knowledge exchange, international orientation, and regional engagement. Results are produced as a radar or sunburst diagram[CE1] , with each dimension represented by a different color, in order to provide a visual representation of an institution’s characteristics.
Figure 1
Source: Van Vught et al., U-Map: European Classification of Higher Education (Enschede, Netherlands: University of Twente Centre for Higher Education Policy Studies, 2010), http://www.u-map.org/U-MAP_report.pdf, p. 37.
U-Map is promoted as a profiling tool, to facilitate easy comparison between different institutions and inform student choice or strategic decision making by institutions or governments. So far more than 230 HEIs, primarily in Europe, have signed up and added their profiles to U-Map; the aim is to have 1,000 European HEIs involved by the end of 2013. The U-Map concept has been taken up and developed in different jurisdictions to showcase institutional diversity.
The Nordic countries launched their own U-Map, albeit with lower-than-expected participation, especially from Sweden and Denmark. In the meantime, Norway and the Republic of Ireland have developed their own versions. These are more complex, benefiting from access to a wider range of national and institutional data, but they share some similarities, including the visualisation of the results. In these instances, the maps are used to display institutional differences and to inform strategic dialogues between the government and/or ministry and HEIs about institutional targets and resourcing. An Australian model has recently been developed by the Martin Institute at the University of Melbourne and the Association Centre for Educational Research (ACER).
Borrowing on the experience of U-Map, U-Multirank was conceived to directly challenge the dominance of global rankings at both the conceptual and functional levels. Whereas U-Map profiles what an institution does, U-Multirank aims to assess how well it does these activities. The objective is to overcome complaints that traditional rankings compare apples with oranges rather than apples with apples.
It is being developed by members of the original CHERPA Consortium, which created U-Map and is led by the Centre for Higher Education (CHE) in Germany and the Centre for Higher Education Policy Studies at the University of Twente, plus the Centre for Science and Technology Studies at Leiden University, the academic publisher Elsevier, the Bertelsmann Foundation, and the software firm Folge 3.
U-Multirank is based on four design principles: it is user driven, whereby each individual or stakeholder group can rank the data according to his/her own preferences; it is multidimensional, with information collected according to five different characteristics; there is peer-group comparability, through which HEIs of similar missions can be compared; and it permits multilevel analysis, in which HEIs can be examined at the institutional level but also at the disciplinary or field-based level and at the department level.
U-Multirank also uses interactive online technology to facilitate multi-functionality. The system does not pre-assign a weighting to each indicator, and there are no composite indicators. This will preclude, the promoters say, the results being aggregated into a single-digit ordinal ranking. At the institutional level, the results will be shown in the sunburst format, while the field-based rankings will draw on the experience of the CHE ranking, which bands universities into three different groups (top, middle, and bottom), using traffic-light colors (green, yellow, and red), as illustrated in Figure 2. The intention is to avoid simplistic league tables.
Figure 2
Source: F. A. Van Vught and F. Ziegele, eds., U-Multirank: Design and Testing the Feasibility of a Multidimensional Global University Ranking (Brussels: European Commission Directorate of Education and Culture, 2010), http://ec.europa.eu/education/higher-education/doc/multirank_en.pdf
During the feasibility phase (2009-2011), about 150 HEIs, primarily from Europe, participated. This was a source of disappointment, particularly since only fifty universities from outside Europe and only two in the U.S. signed up. The implementation phase was launched in January 2013, during the Irish Presidency of the European Council. The threshold of 500 universities has been met with HEIs from more than sixty countries, in line with targets of 75 percent from EU countries and 25 percent from non-EU nations. This phase is being funded with 2 million euros ($2.6 million) for two years by the European Commission, with a possibility of another two years of funding in 2015-16. The ultimate intention is for the ranking to be supported by a foundation or similar independent consortium.
Data collection for the current phase is due to begin shortly, with first results expected in early 2014. This phase will focus on institutional and field-based rankings, including mechanical and electrical engineering, business, and physics. The next phase, due at the end of 2014, will cover universities providing degrees in computer science/IT, sociology, psychology, music, and social work. In response to criticism from research-intensive universities, U-Multirank will facilitate comparison by institutional type. A consultation process on the refined indicators and design of the online tool will continue in parallel with implementation.
The major difficulty plaguing any global ranking is the choice of indicators and the availability of meaningful international comparative data. These issues lie at the heart of the innovative aspects of U-Multirank but are also the source of continued criticism and scepticism. Other concerns arise from the purpose and likely use of the ranking’s results, costs, and name.
The most vocal opposition has come from LERU, the League of European Research Universities, which represents twenty-one research-intensive institutions across Europe, including the universities of Cambridge and Oxford, University College London, Imperial College London, and the University of Edinburgh. LERU formally withdrew its support for the project in January 2013, citing concerns about the need for and cost of U-Multirank—a view that also found favor within the UK House of Lords. In March 2013, following a four-month inquiry into the EU’s contribution to the modernization of European higher education, its EU Social Policies and Consumer Protection Sub-Committee released a report saying that U-Multirank was not a priority for the EU at this time and expressing concern about the administrative burden being placed on institutions to provide the requisite data. It also questioned whether the rankings would ultimately be used to allocate resources for various European research or other programs.
It is fair to say that this criticism should be taken with a grain of salt. There is a pernicious Euro-sceptic sentiment that runs through many UK parliamentary and government statements and is especially strong at the moment. The British government, led by the Conservative Prime Minister David Cameron, is struggling to maintain a balanced discussion despite the involvement of the junior coalition partner, the Liberal Democrats, which is a strong supporter of the EU. Cameron has been forced to declare a simple “in-out” referendum on EU membership in reaction to the growing strength of the right-wing UK Independence Party (UKIP), which is eating into his own party support.
LERU criticism is probably best understood in terms of its membership, which is strongly led by UK-based universities. They have arguably benefited the most from the English-language bias that permeates the Times Higher Education and QS rankings—and it could be said that they have the most to lose from an alternative format. LERU apart, other UK universities have applied to be part of U-Multirank.
Nonetheless, genuine issues have arisen, most thoroughly documented in two substantial reports by the European University Association,[*] which is also conducting a study of the impact of rankings on European higher education institutions.
Indicators
The choice of indicators is always a source of contention. This arises from whether the indicators measure something meaningful or simply what’s easy to count (to paraphrase Einstein). There is some distinction between the choice of indicators used for institutional and field-based rankings, but overall they combine traditional indicators with some new innovative ones, such as interdisciplinary programs, art-based research outputs, and regional engagement. They also include student satisfaction data.
However, input and output indicators are used interchangeably. For example, “continuous professional development,” or CPD, activity is used as an indicator of knowledge transfer but is measured simply in terms of the number of such courses offered per full-time academic staff. Similar comments can be made about counting the number of staff employed in the technology transfer office or international students. Arts, humanities, and social science research are still underrepresented because of reliance on traditional bibliometric databases, such as Web of Science and Scopus, and engagement and co-publications are viewed primarily through a techno-science lens.
U-Multirank may have more indicators than other rankings, but it has not cracked the problem of measuring the quality of teaching and learning. It had hoped that the Organisation for Economic Co-operation and Development’s Assessment of Higher Education Learning Outcomes (AHELO) initiative would provide much-needed data, and chose its initial field-based rankings to align with it. But the demise of AHELO, announced in March of this year, after an estimated expenditure of $10 million, put an end to that dream. So it’s back to reliance on expenditure, graduation rates, and academic performance—which, as we know, are poor proxies for teaching and learning outcomes measures.
The U-Multirank team has worked closely with stakeholders to identify new and more useful indicators. But, ultimately, intensity is equated with quality, and this problem is apparent in oral presentations given by team members. In other words, the more there is of a particular activity, the better it is assumed to be. This is then represented in the sunburst graphic as longer (or shorter) legs—resulting in inevitable misinterpretation.
Purpose and Audience
U-Multirank is designed to challenge the methodology and dominance of the big three global rankings—in particular, Shanghai’s ARWU and the Times Higher Education and QS rankings. If it can entice sufficient numbers of U.S. universities—and so far fewer than twenty have signed up—it will also be able to challenge the rankings of U.S. News & World Report.
Use of the term “rankings” to describe what is in effect “banding” has, however, raised some hackles as well as suspicion that it will ultimately produce or facilitate an ordinal ranking. The European Commission has denied that any correlation will be made between the rankings and resource allocation. However, there is already evidence that the EU, as well as other funding agencies, does take rankings into account in the assessment of the “quality” of the research team. Likewise, there is strong evidence that business and employer groups, philanthropic and investment organizations, and other countries—particularly when national scholarships or partnerships are being considered—do factor in rankings.
Another criticism of rankings is that they have a propensity to disproportionately focus on research-intensive universities. This has had the effect of driving up the status and reputation of “world-class” universities who serve privileged students while simultaneously undermining institutional diversity. U-Multirank has embraced diversity as a core principle, but will now be including a series of predefined rankings by institutional type, including research-intensive universities based on about ten research-related, mainly bibliometric, indicators. While this ranking will be made on a multidimensional and more differentiated basis than existing global rankings, public and policy focus may well gravitate toward this particular ranking—thereby undermining the whole purpose of the exercise.
The CHE ranking has wide usage across Germany, and is also used in neighbouring Austria, the Netherlands, and German-speaking cantons of Switzerland. A French version now in development will probably use nine indicators, some of which will take French education peculiarities into account. Nonetheless, it remains to be seen whether the sunburst diagram, as developed by U-Map and U-Multirank, can provide a meaningful comparator framework for students and other stakeholders, especially given concerns about some of the indicators. Despite claims by U-Multirank that it will enhance consumer choice, global rankings today are much more about global and institutional positioning. After all, this was the reason U-Multirank was created.
Data Sources
Critics of rankings often point to an overreliance on institutional self-reporting in the absence of meaningful cross-national comparative data and data definitions. The same problem affects Europe; the definition of who is a student or a faculty member differs from one member state to the next. There has been some discussion about developing a global common data set, but cross-jurisdictional comparisons of educational quality defy simple methods.
Dependence on institutional data has raised questions about the accuracy of reporting and allegations of “gaming,” which have plagued U.S. News and World Report. It has also led to various efforts to boycott rankings in the hope of undermining them—most notably in the U.S. by the Education Conservancy in 2007 and in Canada around the same time. Most of these campaigns have fizzled out, as the boycott has had little effect except to isolate those universities. The main lesson is that being ranked brings visibility, which is necessary oxygen almost regardless of which position the university actually holds in the ranking.
Nevertheless, data accuracy and accessibility remains a potential land mine. Similarly, administrative time and money has come under scrutiny. To get around this, and to ensure that U-Multirank has consistent, accurate, and independent access to data, the EU cleverly commissioned a sister project called EUMIDA, which lays the foundation for regular data collection by national statistical institutes on individual HEIs in EU member states plus Norway and Switzerland, likely to be coordinated through EUROSTAT. The feasibility study was completed in 2010, and the implementation stage is due to start shortly. EUMIDA is a good example of how policy makers can develop solutions to agilely circumvent roadblocks. This effectively nullifies the decision of LERU, and others, to boycott participation.
It also signals an important opportunity. If U-Multirank can pull in data from other national and supranational sources, then it could provide the basis for a worldwide database on higher education. The implications of that would be very significant indeed.
In a globalized world, cross-jurisdictional comparisons are inevitable and only likely to intensify in the future. In addition, demands for greater accountability about higher education performance can no longer be ignored. We have a right to know whether our students’ qualifications are of high quality, are internationally comparable and transferable, and are being achieved at a reasonable and efficient cost. Rankings have arisen because of this information deficit.
Ultimately, new media technologies and formats, such as social media, consumer Web sites and the Internet, and the use of search engines and open-source facilities, will dramatically transform the debate over the coming years by putting more control directly into the hands of users. It is easy to imagine a higher education TripAdvisor, but crowdsourcing carries its own concerns.
In this environment, U-Multirank is a significant improvement on other global rankings. The difficulties encountered by U-Multirank highlight the complexities associated with assessing and comparing quality. Context remains fundamentally important. National or global, public or private, student cohort and learning environment—these dimensions can radically affect the performance of institutions and render simple comparisons meaningless.
However, as an indicator-based system, U-Multirank can only achieve limited transparency, and cannot provide more than a quantitative picture. It cannot pretend to say anything about quality. And if it remains true to its original mission—to be genuinely “multi-rank”—will it struggle to displace other global rankings? Or will decision makers continue to look for simple answers to complex problems?
